Adding & transforming columns

Analysis is mostly deriving new facts from the columns you already have — a total, a category, a cleaned-up label. In pandas you do this to the whole column at once, no loops, exactly like NumPy's vectorized math.

We'll do this work on a copy, so our clean movies table stays intact for later chapters — a good habit: keep your raw data untouched and derive into a working frame.

m = movies.copy()

A new column from arithmetic

Assign to a column name that doesn't exist yet and pandas creates it. The right-hand side operates on the entire column in one shot:

# which decade is each film from?
m["decade"] = (m["year"] // 10) * 10

m[["title", "year", "decade"]].head(4)

title  year  decade
0     Inception  2010    2010
1    The Matrix  1999    1990
2  Interstellar  2014    2010
3      Parasite  2019    2010

movies["year"] // 10 divides every year by ten at once; there's no for loop in sight. Under the hood this is the NumPy array doing the work.

Transforming text with .str

String columns get a whole toolkit through the .str accessor — .upper(), .lower(), .contains(), .replace(), .len() and more, each applied to every value:

m["genre"].str.upper().head(3)

0    SCI-FI
1    SCI-FI
2    SCI-FI
Name: genre, dtype: str

Custom logic with .apply

When the transformation is too bespoke for built-in operators, .apply runs a function on every value. Here we bucket each rating into a label:

m["tier"] = m["rating"].apply(
  lambda r: "great" if r >= 8.5 else "good"
)

m[["title", "rating", "tier"]]

title  rating   tier
0     Inception     8.8  great
1    The Matrix     8.7  great
2  Interstellar     8.6  great
3      Parasite     8.5  great
4      Whiplash     8.5  great
5          Dune     8.0   good
6    La La Land     8.0   good
7          Coco     8.4   good

Reach for .apply only when vectorized operators can't express what you need — it runs your function row by row, so it's slower than the whole-column arithmetic above. For simple math and string work, stick to the vectorized forms.

You can grow the table. Next, the messier reality: gaps in it.