Data Files and pandas DataFrames

In the previous section we tried some very basic things with Copilot. Now let’s try a more complex example. In the previous chapter, we worked with the Gapminder dataset, which is a CSV file. We used the pandas library to read the CSV file into a DataFrame, and then used the DataFrame to do some analysis. Let’s do that again, but this time we’ll use Copilot to help us.

First, let’s prompt Copilot to read the CSV file containing the data for Europe. I’ve deliberately started with a very minimal prompt, to help illustrate how the wording of the prompt can affect the results.

That generated a scary looking error message! This is human error, not AI error. But before we sort that out, let’s marvel at what Copilot did there. We gave it a pretty poor prompt (“the gapminder data for europe”), and it guessed that the file was named gapminder_gdp_europe.csv. It also guessed that we wanted to use the pandas library (pd) to read the CSV file into a DataFrame. That’s pretty impressive!

Fortunately, the error is pretty self-explanatory. If we want to work with pandas, we always need to first import the pandas library. By convention we give it the alias pd. So let’s add that to our prompt, and see what happens. Note that Copilot may generate code one line at a time, so you may have to accept the suggestion for the first line of code before it generates the next line of code.

More scary error messages! This time, the last line is the most informative. It’s telling us that it can’t find the file we asked it to read. That’s because we haven’t told it where to look for the file. When we just list the name of a file, pd.read_csv() assumes that the file is in the same directory as the notebook. But in this case, the file is in a subdirectory called data. So let’s add that to our prompt, and see what happens.

Now we’re getting somewhere! Let’s view the first few lines of the DataFrame to make sure it looks right.

Again, we used a pretty minimal prompt to get the output. That is, we didn’t need to use the exact name of the DataFrame (gapminder_europe), nor the exact name of the method (head()).

The code that we got in developing this lesson was print(gapminder_europe.head()). However, the output of passing a DataFrame to print() is not formatted as nicely as if we ask the DataFrame to print itself. So let’s try modifying our prompt to get nicely formatted output:

Still not what we wanted (athough at least each row in the DataFrame is now one row in the output) — and Copilot didn’t have any alternative suggestions. This is where prompt engineering becomes important. Let’s try a different prompt.

That did it! We just needed to be a bit more specific about what we wanted to do, by asking for the output as a table.

It’s worth noting that others working through this lesson got gapminder_europe.head() to the original prompt above, but we kept this series of outputs to illustrate how the wording of the prompt can affect the results.

Split-Apply-Combine¶

In the previous chapter, we learned about the split-apply-combine strategy for data analysis. That’s when we split the data into different groups (e.g., according to a particular variable), apply an operation to each group separately, and then combine the results back into a single table. Let’s try that again, but this time we’ll use Copilot to help us.

Let’s replicate the split-apply-combine analysis we did in the previous chapter, where we calculate the average GDP per capita for each of four regions in Europe: Northern Europe, Southern Europe, Eastern Europe, and Western Europe. We’ll start with a very high-level prompt, and see if we get what we want.

Note that in the code cell below, I only typed the first three-line prompt. Copilot generated the rest of the code, including a series of prompts (comments) to help me fill in the details. In order to get it to generate additional prompts and code, I had to hit enter twice after the first prompt. Apparently Copilot is insistent that your code be nicely formatted, with empty lines between different sections of code that do different steps. This is a good thing, but if you forget to hit Enter twice, you may get nothing an think Copilot isn’t working. Just hit Enter twice, and you should get the prompts you need. Otherwise, try going back to the end of your prompt and hitting Enter again.

Prompt Engineering — Writing Better Prompts¶

Can you figure out why this error happened? Ideally, you should be able to do this by re-reading the code the Copilot generated. But, the error message also provides guidance, because it tells us there is an error on line 7 (this information is near the top of the error message), and the error is KeyError: 'group'. That means that gapminder_europe doesn’t have a column named group. Indeed, if we look at line 7 we see it’s attempting to group the data by two columns, year and group. But gapminder_europe doesn’t have a column named group, which is why we got an error.

There are actually a few problems with this code. Not only does group not exist as a column, but it’s not a great name for a column, because the column should indicate region of Europe. So a better name would be region. Another issue is that the .groupby() method was given another variable, year. But our instructions didn’t say anything about year, so we don’t want to group by year. We just want to group by region.

Let’s try fix those issues through better prompt engineering. One thing you’ll encounter frequently is that you give Copilot too big a job, and doing what you’re asking will require a number of intermediate steps. Copilot isn’t great about long-term planning. Remember that LLMs are just advanced versions of the same kind of language model that suggests words when you’re typing on your phone. It’s often good at predicting the next word you want to type, but if you keep just picking the “best” suggestion, you will get a sentence of gibberish more times than not. The same is true with Copilot. If you give it a big job, it will often generate code that doesn’t work, because it’s trying to do too much at once.

So you need to break the job down into smaller steps. In this case, we’re doing split-apply-combine, so let’s try writing our own prompts for each of those conceptual steps. So let’s start with the first step, ‘split’, and see if we can get Copilot to generate the code to group the data by region.

AI may be seeming less magical now. Firstly, it repeated the entire prompt before generating any code. Secondly, it has labelled every country as Western Europe, which is wrong. In the previous chapter, we had to manually create lists for each region label, containing the names of each country in that region. We could do that here to help Copilot along, but hopefully AI is smart enough to know what geographical region each European country is in. This is a reasonable prediction in this case, especially because the open-source Gapminder dataset is widely used in teaching data science, so there should be many versions of this specific example in Copilot’s training set. We just need to figure out how to get our prompting right.

Break the Problem Down into Smaller Steps¶

Although our prompt may be readily interpretable by a human brain, in terms of communicating what we want, apparently it’s too complex for Copilot’s AI. If we think about the task we’re asking Copilot to do, there are several sub-tasks. That is, in one prompt we’re asking Copilot to create four different lists. What if, instead, we ask for one list at a time?

It turns out that if we start with the prompt:

# create a list of the countries in Northern Europe

Copilot will generate the code to create a list of the countries in Northern Europe.

northern_europe = ['Denmark', 'Finland', 'Iceland', 'Norway', 'Sweden']

That’s a good start. But we need to do this for each region. So if we hit Enter a couple of times to get a blank line, and then type the prompt:

# create a list of the countries in 

Copilot is smart enough to complete this prompt with Southern Europe, and generate the code to create a list of the countries in Southern Europe.

Even better, if we accept that code and hit Enter two more times, Copilot will actually generate the complete next prompt for us:

# create a list of the countries in Eastern Europe

And if we accept that one, it generates the code for us, and generates the next prompt:

# create a list of the countries in Western Europe

Once we accept that last prompt, and the code it generates, and again hit Enter twice, Copilot really shines. It generates the same prompts we saw above, and the code to create a new column called region, and then starts by labelling all countries as Western Europe. It generated this code earlier, but it didn’t work because we hadn’t first used the right prompts to get the four lists of countries in each region. Copilot then goes on to generate code to change the region labels for each country, based on the lists we created above.

However, when we try to run this code, we get an error:

The error message tells us that ['Malta'] not in index. Remember that the index is a set of labels for the rows of the DataFrame. If you look back at the code we used, country was set as the index when we read the CSV file into a DataFrame. So this error is telling us that Malta is not in the index.

If we look at the code that generated the error, we can see that Malta is listed in the countries of Southern Europe. The error we get suggests that data for Malta don’t exist in the DataFrame. Let’s check that. We can get Copilot to help, by writing a prompt asking it to check if Malta is in the index of the DataFrame.

To correct this, we can remove Malta from the Southern Europe list. Hopefully you remember how to remove a particular item from a list, from a previous lesson. But there’s so much to remember! You would probably need to go back and look it up, or Google it. But as you might anticipate, we can prompt Copilot to do this, too!

As I wrote the prompt below, however, Copilot made many incorrect guesses as to what I wanted. Initially rather than southern_europe it suggested gapminder_europe — but that would be completely wrong, because the issue is that Malta isn’t in gapminder_europe! When I started typing s rather than accepting gapminder_europe, Copilot suggested south_europe – but if you look at the code above, Copilot had named that list southern_europe. So, even though Copilot can be sensitive to the context of the file you’re working in, it’s still not perfect, and may suggest erroneous code such as the wrong variable name. If you’re not paying attention, and understanding the code you’re generating with Copilot, it’s highly likely that you will make a lot of errors. But after manually editing the Copilot-generated code (changing south_europe to southern_europe), Copilot did produce a correct suggestion for how to remove Malta from the southern_europe list:

An alternative approach – and probably a better one in practice – would have been to go back to the cell that labelled all the regions, and manually remove Malta from southern_europe. That would ensure that our notebook file had clean, functional, and accurate code, rather than the ugly history of our attempts to generate that code. However, the point here is to illustrate the process of working with Copilot, so we are deliberately not doing that.

Now we want to re-run the code above. Let’s cut and paste it from the cell above into the cell below, and then run it. Make sure not to copy over the code that define the lists of countries in each region, because those are already defined, and we don’t want to add Malta back into southern_europe!

Another error, but it looks like the same problem as before: this time, Belarus is not in the DataFrame.

Why is this happening? Well, we assumed that when Copilot generated those lists of countries in each region, it would use the countries in the gapminder_europe DataFrame. But in reality, we don’t know how it generated those lists. Maybe it was based on data used to train Copilot that had a larger list of countries in Europe, rather than just the countries in the DataFrame. And again, Copilot was not accurately sensitive to the context of our notebook file.

Anyway, we now know how to fix this error, based on what we did above for Malta. But what next? How many more countries did Copilot “hallucinate” were in our DataFrame? We can keep repeating this process until we get no errors, but that seems tedious, and could take a long time (especially if we were working with a larger data set). Let’s see if we can tweak our prompts from above to generate more accurate lists of countries in each region.

More Prompt Engineering¶

My first attempt at a prompt was:

# create a list of the countries in Northern Europe, limited to the countries in the gapminder_europe data frame

However, this generated the code below, which is obviously wrong. First of all, it’s circular, in that it’s defining a variable northern_europe based on selecting rows in the DataFrame that are in a variable with the same name (northern_europe).

northern_europe = gapminder_europe.loc[northern_europe]

The larger problem with this code is that it’s using the northern_europe list, but the issue we’re trying to fix is that that list is incorrect. So we need a better prompt. Again, we need to break the problem down into smaller steps. Let’s start with the first step, and see if we can get Copilot to generate the code to create a list of the countries in Northern Europe,as it did before. Then, in a second step, let’s ask it to remove the countries not in the DataFrame. I tried this:

# create a list of countries in northern europe. Then, remove the names of any 
# countries in the list that are not in the gapminder_europe data frame

But Copilot only generated code for the first step. So, then I cut the second step from the first prompt, and made it a second step. As you see below, this generated code that ran with no errors.

Note that the last line of code above is using list comprehension. This is useful and efficient, but it can be a bit challenging to read and understand if you’re not familiar with it.

The list comprehension above is equivalent to this code:

tmp = []
for country in northern_europe:
    if country in gapminder_europe.index:
        tmp.append(country)

northern_europe = tmp

In this code, we have to create a temporary variable tmp to store the new results as we iterate through the items in the list northern_europe. In contrast, list comprehension works all at once, rather than iteratively over each item in the list. Therefore it’s OK to modify the list while we’re iterating over it, because we’re not actually iterating over it.

Of course, with Copilot if the code works, we don’t necessarily need to understand it. But it’s still good practice to actually understand what your code is doing. If Copilot generates code that you don’t understand, you should take the time to learn something new!

Check Your Results¶

Even if we understand what the code is doing, we should double-check whether the list got changed! We can check by viewing the list. You could ask Copilot to do this, but sometimes for simple things it’s faster just to type the code yourself:

Cross-checking this list against the original one, we can see that Luxembourg is no longer in the list. That suggests that Copilot did indeed remove the countries not in the DataFrame. Let’s try this for the other regions. For completeness – having all the code for one conceptual step in one cell – we’ll copy and paste the Northern Europe code from above, then generate additional prompts for the other regions. Copilot catches on to the repetitive nature of this process quickly, and should generate most of the prompts and code for you. (remember to hit Enter twice after each line of code that you accept, to get the next prompt). In fact, after Copilot finished generating prompts for each region, it again generated the prompts to create a new region column in our DataFrame, and populate it with the region labels.

This time the code ran without errors. We can spot-check our region labels by getting a random sample of rows from the DataFrame:

Since new columns are added to the right side of a DataFrame, you’ll need to scroll to the right to see the new region column, but it is indeed there, and the labels in the sample are correct. If you want more reassurance, you could re-run the cell to get a new random sample, or increase the number of samples from 5 to a larger number.

You could also modify your prompt to show only the region column, and then view ta random sample of rows:

Critically Evaluating Copilot’s Code¶

Although our previous pandas lesson and Copilot both used the gapminder_europe.loc method to label countries, there is one important difference between what we showed in the previous lesson, and Copilot’s code. Copilot first labelled all countries as Western Europe, and then changed the label for non-Western countries. In contrast, in the previous lesson we used the same gapminder_europe.loc method to label Western countries as Copilot did here for the other regions. This illustrates how different code can be used to accomplish the same task.

However, just because two approaches do the same job (or appear to), there is some danger to the approach that Copilot took here. In labelling all the countries first as Western Europe, the assumption is that the other labels we use will cover all of the countries that are not Western Europe. However, what if there was a country that was in the DataFrame, but not in any of Copilot’s lists of regions? Those countries would be labeled as Western Europe, which might be incorrect. In that case, the code Copilot generated would create bugs that would affect the outcome of any analyses applied to the data. This is a good example of why it’s important to read and understand the code you’re using, and not just blindly accept what Copilot generates.

At the very least, a good data scientist will assume the worst of their, or AI-generated, code, and test it to make sure it’s doing what it’s supposed to do. In this case, since the concern is that a country not in Western Europe may get that label, we can check by comparing the countries with that label in the DataFrame, with the list western_europe.

Using Copilot to Check its Own Work¶

Fortunately, Copilot can help us check it’s own work – it even correctly generated a lot of the detailed prompt you see below! The prompt below also shows that you can use complex, multi-step prompts with Copilot, and it will generate code for each step. The difference between the complex prompt here, and some that failed above, is that we clearly end each step with a period, and start the next step as a new sentence.

The output shows that there are no countries labelled as Western Europe that are not in the western_europe list. So, Copilot’s code is correct. But, it is important to always check – whether it’s your own code, or AI-generated.

Finally, note that the code that Copilot did generate there is pretty complex. At this point in your development as a coder, do you think you could have written that? And if so, how long do you think it would have taken? Copilot generated that code in a few seconds. This is the promise of AI-assisted coding. At the same time, as a learner and a critical user of AI-generated code, you should take the time to understand what the code is doing, and why. In this case, you can do internet searches for the components of that code that you may not recognize, such as the set() function and the .index method (``).

Summary¶

• GitHub Copilot is a great coding partner. On its own, it often generates errors — including both fatal Python errors, and more sneaky errors that may not be obvious — cases where the code runs without errors, but there are flaws in the logic of the code.

• In spite of the risk of errors errors, Copilot can save you time by quickly generating code, and often generating things that would take you significant time to look up.

• The best approach to using Copilot is to break down your coding tasks into small steps, and prompt Copilot to generate code for each step. Always be conscious of what you’re trying to do and be clear in the expected outputs of that code.

• Always actually read the code that Copilot generates and be sure you understand it. Based on that understanding, confirm the logic of what it’s doing, and ensure it’s what you expect.

• Always test the results of the code Copilot generates, to confirm that it’s doing what you expect.

• This approach will help you both code more efficiently, and become a better coder yourself. Professional coders have eagerly adopted Copilot due to the efficiencies it creates — but there is a learning curve, and ultimately it is just another tool in your toolbox, not the only tool you need.