Pandas GroupBy Explained: Complete Guide with Examples

df.groupby() splits a DataFrame into groups based on the values in one or more columns, applies a function to each group, and combines the results into a new Series or DataFrame. This split-apply-combine pattern is how you calculate totals, averages, counts and other statistics per category in pandas.

How do you use groupby in pandas?

Pick the column to group by, select the column to summarize, and call an aggregation method: df.groupby('category')['value'].sum() returns one total per category, with the category labels as the index. In the sample data below, group A's values (10, 15 and 12) add up to 37.

import pandas as pd

# Sample data
df = pd.DataFrame({
    'category': ['A', 'B', 'A', 'B', 'A', 'C'],
    'value': [10, 20, 15, 25, 12, 30],
    'quantity': [1, 2, 3, 4, 5, 6]
})

# Group by single column and aggregate ✅
df.groupby('category')['value'].sum()
# A    37
# B    45
# C    30

# Multiple aggregations
df.groupby('category')['value'].agg(['sum', 'mean', 'count'])

# Group and get all statistics
df.groupby('category')['value'].describe()

Which aggregation functions work with groupby?

All the common summary statistics are built-in GroupBy methods: sum, mean, count, min, max, std, median, first and last. They run in optimized compiled code, so they are faster than passing an equivalent Python function to apply() or agg().

# Sum
df.groupby('category')['value'].sum()

# Mean (average)
df.groupby('category')['value'].mean()

# Count
df.groupby('category')['value'].count()

# Min and Max
df.groupby('category')['value'].min()
df.groupby('category')['value'].max()

# Standard deviation
df.groupby('category')['value'].std()

# First and Last values
df.groupby('category')['value'].first()
df.groupby('category')['value'].last()

# Median
df.groupby('category')['value'].median()

How do you group by multiple columns in pandas?

Pass a list of column names to groupby: sales.groupby(['region', 'product'])['sales'].sum() returns one row per region and product combination, indexed by a two-level MultiIndex. Call reset_index() or pass as_index=False to turn the keys back into ordinary columns.

sales = pd.DataFrame({
    'region': ['East', 'East', 'West', 'West', 'East'],
    'product': ['A', 'B', 'A', 'B', 'A'],
    'sales': [100, 150, 120, 200, 110]
})

# Group by region and product
result = sales.groupby(['region', 'product'])['sales'].sum()
# region  product
# East    A          210
#         B          150
# West    A          120
#         B          200
# Name: sales, dtype: int64

# Reset index to get normal DataFrame
result = result.reset_index()

# Alternative: group multiple columns
sales.groupby(['region', 'product'], as_index=False)['sales'].sum()

Multiple Aggregations with agg()

Use agg() to compute several statistics in one pass: a list of functions for one column, a dict that maps each column to its own functions, or named aggregations, which also set the output column names and avoid a two-level column header.

# Multiple functions on single column
df.groupby('category')['value'].agg(['sum', 'mean', 'std', 'min', 'max'])

# Different functions on different columns ✅
result = df.groupby('category').agg({
    'value': ['sum', 'mean'],
    'quantity': ['sum', 'count']
})

# Named aggregations (pandas 0.25+)
result = df.groupby('category').agg(
    total_value=('value', 'sum'),
    avg_value=('value', 'mean'),
    total_quantity=('quantity', 'sum'),
    count=('value', 'count')
)

# Custom aggregation function
df.groupby('category')['value'].agg(lambda x: x.max() - x.min())

What does pandas groupby transform do?

transform() calculates a statistic for each group and broadcasts it back to every row of that group, so the result has the same length and index as the original DataFrame. That is why you can assign it straight to a new column, which you can't do with agg(), whose output has one row per group.

df.groupby('category')['value'].agg('sum')        # 3 rows, one per group
# category
# A    37
# B    45
# C    30
# Name: value, dtype: int64

df.groupby('category')['value'].transform('sum')  # 6 rows, same index as df
# 0    37
# 1    45
# 2    37
# 3    45
# 4    37
# 5    30
# Name: value, dtype: int64

Row 0 belongs to group A, so it gets A's total of 37; row 5 is the only C row, so it gets 30. Because the index lines up with df, arithmetic against the original columns works row by row: df['value'] / df.groupby('category')['value'].transform('sum') gives each row's share of its group total, 10 / 37 = 0.27 for row 0.

# Calculate group mean and broadcast to all rows
df['category_mean'] = df.groupby('category')['value'].transform('mean')

# Calculate percentage of group total
df['pct_of_group'] = df['value'] / df.groupby('category')['value'].transform('sum')

# Z-score within groups
df['z_score'] = (df['value'] - df.groupby('category')['value'].transform('mean')) / \
                df.groupby('category')['value'].transform('std')

# Rank within groups
df['rank_in_group'] = df.groupby('category')['value'].rank(ascending=False)

What is the difference between agg, transform and apply?

agg() reduces each group to one row, transform() returns one value per original row, and apply() returns whatever your function returns, from a single number to a whole DataFrame per group. Use agg for summaries, transform to add group statistics as new columns, and apply only when neither fits, because it runs a Python function once per group and is the slowest of the three.

Method	Rows returned (sample df)	Typical use
agg()	One per group (3)	Totals, averages and counts for a report
transform()	One per original row (6)	Group mean, share of group total, z-score within group
apply()	Whatever the function returns	Top N rows per group, custom multi-step logic

g = df.groupby('category')['value']
len(g.agg('sum'))                       # 3  (one row per group)
len(g.transform('sum'))                 # 6  (one row per original row)
len(g.apply(lambda x: x.nlargest(2)))   # 5  (up to 2 rows per group)

Filter Groups

filter() keeps or drops whole groups: your function receives each group as a DataFrame and must return a single True or False, and the result contains the original rows of every group that returned True. With the sample data, len(x) > 2 keeps only group A's three rows.

# Keep only groups with sum > 50
df.groupby('category').filter(lambda x: x['value'].sum() > 50)

# Keep groups with more than 2 members
df.groupby('category').filter(lambda x: len(x) > 2)

# Keep groups where max value > 20
df.groupby('category').filter(lambda x: x['value'].max() > 20)

Apply Custom Functions

apply() passes each group to your function as a DataFrame and combines whatever comes back, so it can return several statistics or several rows per group, such as the top 2 rows of each category. Since pandas 3.0 the grouping columns are no longer included in the DataFrame your function receives.

# Apply custom function to each group
def custom_stats(group):
    return pd.Series({
        'sum': group['value'].sum(),
        'range': group['value'].max() - group['value'].min(),
        'count': len(group)
    })

result = df.groupby('category').apply(custom_stats)

# Apply function that returns DataFrame
def top_2(group):
    return group.nlargest(2, 'value')

df.groupby('category').apply(top_2)

Practical Examples

These patterns cover everyday sales reporting: totals per product, averages across two keys, monthly totals from a date column, the top row in each group, and running totals and averages within each group.

# Example: Sales analysis
sales_data = pd.DataFrame({
    'date': pd.date_range('2024-01-01', periods=100),
    'product': ['A', 'B', 'C'] * 33 + ['A'],
    'sales': range(100, 200),
    'region': ['North', 'South'] * 50
})

# Total sales by product
sales_data.groupby('product')['sales'].sum()

# Average sales by region and product
sales_data.groupby(['region', 'product'])['sales'].mean()

# Sales by month
sales_data['month'] = sales_data['date'].dt.month
monthly_sales = sales_data.groupby('month')['sales'].sum()

# Top selling product per region
sales_data.groupby('region').apply(
    lambda x: x.nlargest(1, 'sales')[['product', 'sales']]
)

# Cumulative sum within groups
sales_data['cumulative_sales'] = sales_data.groupby('product')['sales'].cumsum()

# Running average
sales_data['running_avg'] = sales_data.groupby('product')['sales'].expanding().mean().reset_index(level=0, drop=True)

Performance Tips

The biggest speedups come from using built-in aggregations instead of apply(), grouping on categorical columns, and skipping work you don't need, such as sorting the group keys with sort=False.

# Use categorical data type for groupby columns (faster)
df['category'] = df['category'].astype('category')
df.groupby('category')['value'].sum()  # Faster on large data

# Use sort=False if you don't need sorted results
df.groupby('category', sort=False)['value'].sum()  # Faster

# Use as_index=False to avoid reset_index() later
df.groupby('category', as_index=False)['value'].sum()

# Avoid apply() when possible - use built-in functions
# Slow ❌
df.groupby('category').apply(lambda x: x['value'].sum())

# Fast ✅
df.groupby('category')['value'].sum()

Advanced Patterns

Combine transform, agg and unstack for report-style output: each row's percentage of its group total, flattened column names after a multi-function agg, a pivot-style table, and named aggregations that mix sum, mean, max and size.

# Each row's percent of its group total
df['pct_of_total'] = df.groupby('category')['value'].transform(
    lambda x: x / x.sum() * 100
)

# Group statistics
group_stats = df.groupby('category').agg({
    'value': ['sum', 'mean', 'std', 'count'],
    'quantity': 'sum'
})

# Flatten multi-level columns
group_stats.columns = ['_'.join(col).strip() for col in group_stats.columns]

# Pivot-like with groupby: one row per region, one column per product
sales.groupby(['region', 'product'])['sales'].sum().unstack()
# product    A    B
# region
# East     210  150
# West     120  200

# Multiple groups with different operations
df.groupby('category').agg(
    total_value=('value', 'sum'),
    avg_value=('value', 'mean'),
    max_quantity=('quantity', 'max'),
    count=('value', 'size')  # Use 'size' to count all rows including NaN
)

Why are rows with missing keys left out of groupby results?

groupby drops rows whose group key is NaN by default (dropna=True), so their values silently disappear from the totals. Pass dropna=False, available since pandas 1.1, to keep them as their own NaN group.

scores = pd.DataFrame({'team': ['A', 'B', None, 'A'], 'pts': [10, 20, 5, 15]})

scores.groupby('team')['pts'].sum()
# team
# A    25
# B    20
# Name: pts, dtype: int64

scores.groupby('team', dropna=False)['pts'].sum()
# team
# A      25
# B      20
# NaN     5
# Name: pts, dtype: int64

Best Practices

Keep the keys as columns with as_index=False, name your aggregations, use transform for row-aligned results, and remember that count() skips NaN while size() counts every row.

• ✅ Use as_index=False to keep groupby columns as regular columns
• ✅ Use .agg() with named aggregations for readable code
• ✅ Use .transform() when you need results aligned with original data
• ✅ Convert groupby columns to categorical type for better performance
• ✅ Use size() instead of count() to include NaN values
• ❌ Don't use apply() for simple aggregations (use built-in functions)
• ❌ Don't forget reset_index() if you want a normal DataFrame
• ⚠️ Be aware that count() excludes NaN, but size() includes them