Chapter 6 · Part 3

Capstone: analyze a dataset

Real pandas work isn't isolated calls — it's a pipeline: load a table, then derive, filter, group, aggregate and sort in one readable flow. Let's answer a genuine question about our movies end to end: how did each decade's films score?

One question, one pipeline

pandas methods return DataFrames, so they chain. Wrapping the chain in parentheses lets you put each step on its own line — read it top to bottom like a recipe:

analyze.py

result = (
  movies
  .assign(decade=(movies["year"] // 10) * 10)   # 1. derive a decade column
  .groupby("decade")                            # 2. split by decade
  .agg(                                          # 3. summarize each group
      films=("title", "size"),
      avg_rating=("rating", "mean"),
  )
  .round(2)                                       # 4. tidy the numbers
)

print(result)

output

films  avg_rating
decade
1990        1        8.70
2010        6        8.47
2020        1        8.00

Four steps, one expression, and the answer falls out: the 2010s dominate our little dataset (six films) at a healthy 8.47 average. .assign is the chain-friendly way to add a column — it returns a new DataFrame instead of modifying in place, so it slots into the pipeline. Swap the grouping column, change the aggregation, add a .sort_values — and you can ask almost any question this way.

What you can now do

Six chapters, and you've got the core loop of every data analysis:

Build & inspect a DataFrame — head, shape, info.
Select & filter — columns, .loc/.iloc, boolean masks.
Add & transform columns — vectorized, no loops.
Clean missing data — isna, dropna, fillna.
Group & aggregate — split-apply-combine.
Chain it all into a pipeline.

Where to go next

pandas is deep; these are the natural next stops once the basics click:

merge / join — combine two tables on a shared key, like a database join. The moment your data lives in more than one table, this is how you bring it together.
pivot_table — reshape long data into a grid (rows × columns of aggregates); groupby's spreadsheet-shaped cousin.
Time series — pandas has first-class dates: parse them with pd.to_datetime, then resample by day, month or year.
Plotting — df.plot() turns any DataFrame into a chart via matplotlib, straight from the table.

🎯 Your turn

Find the single highest-rated film in each genre. Hint: groupby('genre')['rating'].idxmax() gives you the index label of the top film per group — feed those labels to .loc.

That's the foundation. pandas sits under nearly every data and machine-learning workflow in Python — you've just learned the tool you'll reach for first, every time. If you came from the NumPy course, you now have both halves of Python's data core; the other courses build on them.