Reading real CSV data and missing values

Real data almost always has gaps — a sensor that didn't report, a form field a user skipped, a join that didn't find a match. Pandas represents a missing value as NaN ("not a number"), and it's worth checking for these immediately after loading any real dataset:

df.isna().sum()          # count of missing values per column
df.isna().any(axis=1)    # a boolean mask: which rows have any missing value

There's no universally "correct" choice — the right one depends on what the missing value means and what you're about to do with the data:

1. Drop the rows — df.dropna() — reasonable when missing rows are rare and you can afford to lose them, but silently shrinks your dataset, which matters if you're computing statistics or training a model on the result.
2. Fill with a reasonable default — df["score"].fillna(0) or df["score"].fillna(df["score"].mean()) — reasonable when a sensible default exists, but a fabricated value can quietly bias downstream statistics if you forget it's there.
3. Leave it and handle it explicitly downstream — sometimes the right answer is "this needs a person to look at it," not an automatic fill — especially for anything feeding a decision that matters (this is the same judgment as the "when AI shouldn't answer" pattern from earlier lanes, applied to data pipelines).

Whichever you choose, the mistake to avoid is silently dropping or filling without checking how much data that affects — df.isna().sum() before and df.shape before/after any dropna() call is a five-second habit that catches real problems.

A column that displays numbers can still load as text (object dtype) if even one row has an unexpected value — a stray "N/A" string mixed into an otherwise numeric column, for example. df.dtypes catches this immediately; a numeric-looking column showing up as object is a strong signal to inspect it with df["column"].unique() before doing math on it.

df["score"] = pd.to_numeric(df["score"], errors="coerce")  # bad values become NaN instead of crashing

errors="coerce" is the safe default here — it converts anything that can't become a number into NaN (which you can then handle with the strategies above) rather than crashing your whole script on one bad row.

Once a dataset is cleaned, df.to_csv("cleaned.csv", index=False) writes it back out — index=False is worth remembering, since without it pandas writes that auto-generated row-number index as an extra unwanted column in the output file.