NumPy and pandas basics
Worked example: cleaning a small dataset
One small, messy dataset, cleaned start to finish, using every technique from this topic in the order you'd actually reach for them.
1Learn the idea
Read
The starting data
import pandas as pd
import numpy as np
df = pd.DataFrame({
"class": ["A", "A", "A", "B", "B", "B", "C", "C", "C"],
"score": [88, 92, np.nan, 79, "95", 61, 70, 65, np.nan],
})
Two problems are already visible if you check df.dtypes right away, matching the habit from the "building a DataFrame" page: the score column contains a mix of numbers, a string ("95"), and missing values, so pandas stores the whole column as object instead of a clean numeric type.
Read
Step 1: fix the type
df["score"] = pd.to_numeric(df["score"], errors="coerce")
print(df.dtypes) # score is now a proper numeric (float) column
errors="coerce" matters here specifically because it converts anything unexpected into NaN instead of crashing — useful mid-cleanup, though worth double-checking afterward that nothing valid got silently coerced.
Read
Step 2: check how much is missing
print(df.isna().sum())
Two missing scores out of nine rows — small enough, and evenly enough spread across the three classes, that filling with each class's own mean (rather than dropping the rows or using one global mean) is a reasonable choice here. This is a judgment call worth stating explicitly, not a default to apply blindly — a real project might instead decide dropping is safer if missingness isn't random.
Read
Step 3: fill missing values, per group
df["score"] = df.groupby("class")["score"].transform(lambda s: s.fillna(s.mean()))
print(df)
This line combines two techniques from this topic in one place: groupby (split by class) and fillna (handle missing values) — transform applies the fill within each group separately, so class A's missing value gets filled with class A's own mean, not a mix of all three classes.
Read
Step 4: answer the actual question
summary = df.groupby("class")["score"].mean().reset_index()
print(summary.sort_values("score"))
Sorting the summary by score surfaces the lowest-scoring class immediately — the answer to "which class needs extra review time" is now a one-line read off a three-row table, not a manual scan of the raw data.
Read
Step 5: sanity-check before trusting the answer
Before acting on this result, it's worth asking the same verification question that runs through the whole curriculum: does this conclusion survive a second look? Two rows filled with an estimated mean, out of three or four per class, is a meaningful fraction of a very small group — worth mentioning alongside the conclusion ("class C looks lowest, though one of three scores was estimated"), not hidden behind a single clean-looking number.
Read
What this worked example demonstrates
Every technique used here — type coercion, isna(), groupby with fillna, reset_index, sorting — showed up individually on an earlier page. Real data work is exactly this: a small, unglamorous sequence of these same moves, applied to a specific messy dataset, ending in a plain-English answer to a real question.
Go deeper
Before you start
Why this matters
Here's a small, realistic mess: a CSV of quiz scores from three classes, with a few missing values, one row where the score was accidentally entered as text, and a question to answer at the end — "which class should get extra review time?" Walking through this end to end shows how the individual techniques from earlier pages combine into a normal data-cleaning session.
In the wild
See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.