Missing data & cleaning

Real datasets have holes: a survey question skipped, a sensor offline, a field that didn't apply. pandas marks these gaps with a special value, NaN ("not a number"), and gives you tools to find them and decide what to do. This is the unglamorous half of every analysis — and often the half that decides whether your results are right.

Let's work on a copy again and punch two holes in it on purpose (a missing rating for Dune, missing votes for Coco):

import numpy as np

m = movies.copy()
m.loc[5, "rating"] = np.nan   # Dune's rating is unknown
m.loc[7, "votes"]  = np.nan   # Coco's votes are unknown

print(m)

title  year      genre  rating   votes
0     Inception  2010     Sci-Fi     8.8  2400.0
1    The Matrix  1999     Sci-Fi     8.7  1900.0
2  Interstellar  2014     Sci-Fi     8.6  1800.0
3      Parasite  2019   Thriller     8.5   780.0
4      Whiplash  2014      Drama     8.5   900.0
5          Dune  2021     Sci-Fi     NaN   650.0
6    La La Land  2016      Drama     8.0   560.0
7          Coco  2017  Animation     8.4     NaN

Notice votes turned from whole numbers into 2400.0, 1900.0… — pandas quietly promoted the column to float, because NaN is a float and a column holds one type.

Find the gaps

isna() returns a boolean mask of where the holes are; sum it to count missing values per column. This is your first move on any real dataset:

m.isna().sum()

title     0
year      0
genre     0
rating    1
votes     1
dtype: int64

Then decide: drop or fill

There's no universal right answer — it's a judgment call, and it changes your results. Two tools:

dropna() removes any row containing a missing value. Safe, but you lose whole rows (here, Dune and Coco vanish entirely — even their good data):

m.dropna()   # rows 5 and 7 are gone

title  year     genre  rating   votes
0     Inception  2010    Sci-Fi     8.8  2400.0
1    The Matrix  1999    Sci-Fi     8.7  1900.0
2  Interstellar  2014    Sci-Fi     8.6  1800.0
3      Parasite  2019  Thriller     8.5   780.0
4      Whiplash  2014     Drama     8.5   900.0
6    La La Land  2016     Drama     8.0   560.0

fillna(value) patches the holes instead, keeping every row. A common choice is to fill a numeric column with its own mean (pandas skips NaN when computing it):

mean_rating = m["rating"].mean().round(2)   # 8.5
m["rating"] = m["rating"].fillna(mean_rating)

m["rating"]

0    8.8
1    8.7
2    8.6
3    8.5
4    8.5
5    8.5
6    8.0
7    8.4
Name: rating, dtype: float64

Row 5 (Dune) is now 8.5 — the filled-in mean. Dropping is honest but throws away data; filling keeps your rows but invents values. Which is right depends entirely on your question — that choice is the analyst's job, not the library's.

Your data's clean. Now the most powerful move in pandas: grouping.