Page 7 of 8~113 min topic

NumPy and pandas basics

Worked example: cleaning a small dataset

One small, messy dataset, cleaned start to finish, using every technique from this topic in the order you'd actually reach for them.

~16 min this pageWorked example

1Learn the idea

Read

The starting data

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "class": ["A", "A", "A", "B", "B", "B", "C", "C", "C"],
    "score": [88, 92, np.nan, 79, "95", 61, 70, 65, np.nan],
})

Two problems are already visible if you check df.dtypes right away, matching the habit from the "building a DataFrame" page: the score column contains a mix of numbers, a string ("95"), and missing values, so pandas stores the whole column as object instead of a clean numeric type.

Read

Step 1: fix the type

df["score"] = pd.to_numeric(df["score"], errors="coerce")
print(df.dtypes)   # score is now a proper numeric (float) column

errors="coerce" matters here specifically because it converts anything unexpected into NaN instead of crashing — useful mid-cleanup, though worth double-checking afterward that nothing valid got silently coerced.

Read

Step 2: check how much is missing

print(df.isna().sum())

Two missing scores out of nine rows — small enough, and evenly enough spread across the three classes, that filling with each class's own mean (rather than dropping the rows or using one global mean) is a reasonable choice here. This is a judgment call worth stating explicitly, not a default to apply blindly — a real project might instead decide dropping is safer if missingness isn't random.

Read

Step 3: fill missing values, per group

df["score"] = df.groupby("class")["score"].transform(lambda s: s.fillna(s.mean()))
print(df)

This line combines two techniques from this topic in one place: groupby (split by class) and fillna (handle missing values) — transform applies the fill within each group separately, so class A's missing value gets filled with class A's own mean, not a mix of all three classes.

Read

Step 4: answer the actual question

summary = df.groupby("class")["score"].mean().reset_index()
print(summary.sort_values("score"))

Sorting the summary by score surfaces the lowest-scoring class immediately — the answer to "which class needs extra review time" is now a one-line read off a three-row table, not a manual scan of the raw data.

Read

Step 5: sanity-check before trusting the answer

Before acting on this result, it's worth asking the same verification question that runs through the whole curriculum: does this conclusion survive a second look? Two rows filled with an estimated mean, out of three or four per class, is a meaningful fraction of a very small group — worth mentioning alongside the conclusion ("class C looks lowest, though one of three scores was estimated"), not hidden behind a single clean-looking number.

Read

What this worked example demonstrates

Every technique used here — type coercion, isna(), groupby with fillna, reset_index, sorting — showed up individually on an earlier page. Real data work is exactly this: a small, unglamorous sequence of these same moves, applied to a specific messy dataset, ending in a plain-English answer to a real question.

Go deeper

Before you start

Why this matters

Here's a small, realistic mess: a CSV of quiz scores from three classes, with a few missing values, one row where the score was accidentally entered as text, and a question to answer at the end — "which class should get extra review time?" Walking through this end to end shows how the individual techniques from earlier pages combine into a normal data-cleaning session.

Check your understanding

Page assessment

Answer from memory. Completion is saved from this evidence, not from opening the next page.

1. What is one idea from this page you would apply, and what evidence would you check?

All responses are required.