Chi-Square Test Explained

A chi-square (χ²) test checks whether observed category counts differ from expected counts by more than random chance would explain. Use the test of independence to see whether two categorical variables are related, such as marketing channel and conversion, and the goodness-of-fit test to see whether one variable follows an expected distribution.

How do you run a chi-square test of independence in Python?

Build a contingency table of counts with pd.crosstab, then pass it to scipy.stats.chi2_contingency, which returns the chi-square statistic, the p-value, the degrees of freedom, (rows − 1) × (columns − 1), and the expected counts. A p-value below your significance level, usually 0.05, means the two variables are associated.

Question: Are Two Variables Related?

from scipy.stats import chi2_contingency
import pandas as pd

# Example: Is gender related to product preference?
data = pd.DataFrame({
    'Gender': ['M', 'M', 'M', 'F', 'F', 'F'] * 50,
    'Product': ['A', 'B', 'C', 'A', 'B', 'C'] * 50
})

# Create contingency table
contingency_table = pd.crosstab(data['Gender'], data['Product'])
print(contingency_table)

# Product   A   B   C
# Gender
# F        50  50  50
# M        50  50  50

Run the Test

# Chi-square test
chi2, p_value, dof, expected = chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2:.3f}")
print(f"P-value: {p_value:.3f}")
print(f"Degrees of freedom: {dof}")
print("\nExpected frequencies:")
print(expected)

# Interpretation
alpha = 0.05
if p_value < alpha:
    print("✓ Variables are related (dependent)")
else:
    print("✗ No evidence of a relationship")

This toy data has identical counts in every cell, so χ² = 0, p = 1 and there are 2 degrees of freedom. A non-significant result means you found no evidence of a relationship, not proof that the variables are independent.

How do you calculate the chi-square statistic by hand?

Compute each cell's expected count as row total × column total ÷ grand total, then add up (observed − expected)² ÷ expected across all cells. Compare the total with the chi-square distribution for (rows − 1) × (columns − 1) degrees of freedom; at the 5% level the critical values are 3.84 for 1 degree of freedom, 5.99 for 2 and 7.81 for 3.

In the marketing data below, 135 of 300 visitors converted and each channel had 100 visitors, so every channel's expected count is 135 × 100 / 300 = 45 conversions and 55 non-conversions. Email matches exactly and contributes 0. Social contributes (30 − 45)²/45 + (70 − 55)²/55 = 5 + 4.09, and Search contributes the same, so χ² = 18.18 with 2 degrees of freedom. That is far above 5.99, and scipy reports p = 0.0001: conversion rates differ by channel.

import numpy as np

# Observed data
observed = pd.DataFrame({
    'Email': [45, 55],
    'Social': [30, 70],
    'Search': [60, 40]
}, index=['Converted', 'Not Converted'])

print("Observed frequencies:")
print(observed)

#                Email  Social  Search
# Converted         45      30      60
# Not Converted     55      70      40

# Test if conversion rate differs by channel
chi2, p, dof, expected = chi2_contingency(observed)

print(f"\nP-value: {p:.4f}")
if p < 0.05:
    print("Channels have significantly different conversion rates!")
else:
    print("No significant difference between channels")

# P-value: 0.0001
# Channels have significantly different conversion rates!

What is a chi-square goodness-of-fit test?

A goodness-of-fit test checks whether the counts of one categorical variable match a distribution you specify in advance, such as equal counts for each face of a die. Run it with scipy.stats.chisquare(observed, expected); the expected counts must add up to the same total as the observed counts (scipy raises an error otherwise), and the degrees of freedom are the number of categories minus 1.

Question: Does Data Match Expected Distribution?

from scipy.stats import chisquare

# Example: Are die rolls fair?
observed_rolls = [12, 15, 18, 14, 16, 15]  # Counts for faces 1-6
expected_rolls = [15, 15, 15, 15, 15, 15]  # Expected for fair die

chi2, p_value = chisquare(observed_rolls, expected_rolls)

print(f"Chi-square: {chi2:.3f}")
print(f"P-value: {p_value:.3f}")

if p_value < 0.05:
    print("Die is NOT fair")
else:
    print("No evidence the die is unfair")

# Chi-square: 1.333
# P-value: 0.931
# No evidence the die is unfair

Practical Example: Survey Analysis

Larger tables work the same way. With three satisfaction levels across four age groups the test has (3 − 1) × (4 − 1) = 6 degrees of freedom, and here χ² = 14.84 with p = 0.0216, so satisfaction differs across age groups at the 5% level.

# Survey: Is satisfaction related to age group?
survey_data = {
    '18-25': [20, 30, 10],  # [Satisfied, Neutral, Dissatisfied]
    '26-40': [45, 35, 20],
    '41-60': [55, 25, 20],
    '60+':   [30, 15, 5]
}

# Create DataFrame
df = pd.DataFrame(survey_data,
                 index=['Satisfied', 'Neutral', 'Dissatisfied'])

print(df)

# Run test
chi2, p, dof, expected = chi2_contingency(df)

print(f"\nChi-square: {chi2:.2f}")
print(f"P-value: {p:.4f}")

if p < 0.05:
    print("✓ Satisfaction significantly differs across age groups")
else:
    print("✗ No significant difference")

# Chi-square: 14.84
# P-value: 0.0216
# ✓ Satisfaction significantly differs across age groups

Calculating Effect Size (Cramér's V)

Cramér's V measures how strong an association is, from 0 (none) to 1 (perfect), which the p-value cannot tell you: with enough rows, even a trivial difference is significant. It is V = √(χ² / (n × (k − 1))), where n is the total count and k is the smaller of the number of rows and columns.

import numpy as np

def cramers_v(contingency_table):
    chi2, p, dof, expected = chi2_contingency(contingency_table)
    n = contingency_table.sum().sum()
    min_dim = min(contingency_table.shape) - 1
    return np.sqrt(chi2 / (n * min_dim))

# Calculate effect size
v = cramers_v(observed)
print(f"Cramér's V: {v:.3f}")
# Cramér's V: 0.246

# Rough guide for tables with 2 rows or 2 columns
# (Cohen's benchmarks; larger tables use lower cutoffs):
# 0.0 - 0.1: Negligible
# 0.1 - 0.3: Weak
# 0.3 - 0.5: Moderate
# 0.5+: Strong

The marketing table's V of 0.246 sits between Cohen's small (0.1) and medium (0.3) benchmarks, even though its p-value is 0.0001. Report both: the p-value says the association is unlikely to be chance, and V says how strong it is (more in effect size vs significance).

Which cells drive a significant chi-square result?

Compare observed and expected counts cell by cell with Pearson residuals, (O − E) / √E. The squared residuals add up to χ², so the cells with the largest absolute residuals contribute most, and values beyond about ±2 are worth a closer look.

chi2, p, dof, expected = chi2_contingency(observed)
residuals = (observed - expected) / np.sqrt(expected)
print(residuals.round(2))

#                Email  Social  Search
# Converted        0.0   -2.24    2.24
# Not Converted    0.0    2.02   -2.02

Social converts less often than independence predicts and Search more often, while Email sits exactly on its expected count.

Assumptions and Requirements

The test needs raw counts of independent observations, each falling in exactly one category, and expected counts large enough for the chi-square approximation: at least 5 in every cell, or under Cochran's looser rule, at least 5 in 80% of cells and none below 1.

• Independence: Observations must be independent
• Sample size: Expected frequency ≥ 5 in each cell
• Categorical data: Use counts, not percentages
• Mutually exclusive: Each observation in exactly one category

When Expected Counts are Low

Use Fisher's exact test for a 2 × 2 table, or merge sparse categories until every expected count reaches 5. Fisher's test computes the p-value exactly instead of relying on the chi-square approximation, which becomes unreliable when expected counts are small.

# Fisher's Exact Test for small samples
from scipy.stats import fisher_exact

# 2x2 table with small counts
table = [[3, 7],
         [8, 2]]

odds_ratio, p_value = fisher_exact(table)
print(f"P-value: {p_value:.3f}")
# P-value: 0.070

# More reliable than chi-square for small samples

Visualizing Results

A heatmap of the contingency table shows where the counts concentrate, and plotting observed next to expected counts shows which cells pull away from independence.

import seaborn as sns
import matplotlib.pyplot as plt

# Heatmap of contingency table
sns.heatmap(contingency_table, annot=True, fmt='d', cmap='Blues')
plt.title('Gender vs Product Preference')
plt.show()

# Bar plot comparing observed vs expected (marketing data)
chi2, p, dof, expected = chi2_contingency(observed)
expected_df = pd.DataFrame(expected, index=observed.index, columns=observed.columns)
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
observed.plot(kind='bar', ax=ax1, title='Observed')
expected_df.plot(kind='bar', ax=ax2, title='Expected')
plt.tight_layout()

Chi-square test vs t-test: which should you use?

Use a chi-square test when both variables are categorical and you are comparing counts or proportions; use a t-test when you are comparing the mean of a numeric variable between two groups. Converted or not by channel is a chi-square question; average order value for email vs search is a t-test question, and for three or more groups you would use ANOVA.

What are common chi-square mistakes?

The most common mistake is running the test on percentages or proportions instead of raw counts. χ² grows in proportion to the sample size (doubling every count doubles χ²), so rescaled numbers give a meaningless p-value.

• Counting a subject twice: each observation must appear in exactly one cell. For before-and-after yes/no answers from the same people, use McNemar's test instead.
• Missing Yates' correction: for 2 × 2 tables, chi2_contingency applies Yates' continuity correction by default, so [[30, 20], [15, 35]] gives χ² = 7.92 rather than the textbook 9.09. Pass correction=False to match a hand calculation.
• Reading p > 0.05 as proof of independence: it only means the data shows no evidence of a relationship, often because the sample is too small.

Common Use Cases

Reach for a chi-square test whenever both the grouping and the outcome are categories you can count.

• A/B testing with categorical outcomes
• Survey analysis (demographics vs responses)
• Quality control (defect rates by production line)
• Medical studies (treatment vs outcome)
• Market research (customer segment vs preferences)