Page 7 of 8~113 min topic

NumPy and pandas basics

Worked example: cleaning a small dataset

One small, messy dataset, cleaned start to finish, using every technique from this topic in the order you'd actually reach for them.

~16 min this pageWorked example

1Learn the idea

Read

The starting data

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "class": ["A", "A", "A", "B", "B", "B", "C", "C", "C"],
    "score": [88, 92, np.nan, 79, "95", 61, 70, 65, np.nan],
})

Two problems are already visible if you check df.dtypes right away, matching the habit from the "building a DataFrame" page: the score column contains a mix of numbers, a string ("95"), and missing values, so pandas stores the whole column as object instead of a clean numeric type.

Read

Step 1: fix the type

df["score"] = pd.to_numeric(df["score"], errors="coerce")
print(df.dtypes)   # score is now a proper numeric (float) column

errors="coerce" matters here specifically because it converts anything unexpected into NaN instead of crashing — useful mid-cleanup, though worth double-checking afterward that nothing valid got silently coerced.

Read

Step 2: check how much is missing

print(df.isna().sum())

Two missing scores out of nine rows — small enough, and evenly enough spread across the three classes, that filling with each class's own mean (rather than dropping the rows or using one global mean) is a reasonable choice here. This is a judgment call worth stating explicitly, not a default to apply blindly — a real project might instead decide dropping is safer if missingness isn't random.

Read

Step 3: fill missing values, per group

df["score"] = df.groupby("class")["score"].transform(lambda s: s.fillna(s.mean()))
print(df)

This line combines two techniques from this topic in one place: groupby (split by class) and fillna (handle missing values) — transform applies the fill within each group separately, so class A's missing value gets filled with class A's own mean, not a mix of all three classes.

Read

Step 4: answer the actual question

summary = df.groupby("class")["score"].mean().reset_index()
print(summary.sort_values("score"))

Sorting the summary by score surfaces the lowest-scoring class immediately — the answer to "which class needs extra review time" is now a one-line read off a three-row table, not a manual scan of the raw data.

Read

Step 5: sanity-check before trusting the answer

Before acting on this result, it's worth asking the same verification question that runs through the whole curriculum: does this conclusion survive a second look? Two rows filled with an estimated mean, out of three or four per class, is a meaningful fraction of a very small group — worth mentioning alongside the conclusion ("class C looks lowest, though one of three scores was estimated"), not hidden behind a single clean-looking number.

Read

What this worked example demonstrates

Every technique used here — type coercion, isna(), groupby with fillna, reset_index, sorting — showed up individually on an earlier page. Real data work is exactly this: a small, unglamorous sequence of these same moves, applied to a specific messy dataset, ending in a plain-English answer to a real question.

Go deeper

Before you start

Why this matters

Here's a small, realistic mess: a CSV of quiz scores from three classes, with a few missing values, one row where the score was accidentally entered as text, and a question to answer at the end — "which class should get extra review time?" Walking through this end to end shows how the individual techniques from earlier pages combine into a normal data-cleaning session.

In the wild

See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.

Continue learning · glossary & guides