Hypothesis Testing: A Step-by-Step Guide

Hypothesis testing is a statistical method for deciding whether a difference in your data, such as a higher conversion rate for a new page design, is real or could plausibly be random chance. You assume there is no effect (the null hypothesis), measure how surprising your data would be under that assumption (the p-value), and reject the null only if the data are surprising enough.

What are the 5 steps of hypothesis testing?

The five steps are: state the null and alternative hypotheses, choose a significance level (α), calculate a test statistic, find its p-value, and draw a conclusion by comparing the p-value with α. Do steps 1 and 2 before you look at the data; the example below runs all five on a website redesign.

Step 1: State the Hypotheses

# Null Hypothesis (H0): No effect, no difference
# Alternative Hypothesis (H1): There is an effect/difference

Example: Testing new website design
H0: New design has same conversion rate as old design
H1: New design has different conversion rate

# Two-tailed: Different (higher OR lower)
# One-tailed: Specifically higher (or lower)

Step 2: Choose Significance Level (α)

# Alpha = probability of rejecting H0 when it's true (Type I error)
# Common values: 0.05 (5%), 0.01 (1%)

alpha = 0.05  # 95% confidence level

# Interpretation: Accept 5% chance of false positive

Step 3: Select and Calculate Test Statistic

from scipy import stats
import numpy as np

# T-test for comparing two means
group_a = [23, 25, 27, 24, 26]  # Old design conversions
group_b = [28, 30, 32, 29, 31]  # New design conversions

# Independent samples t-test
t_stat, p_value = stats.ttest_ind(group_a, group_b)

print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.3f}")

# Output:
# t-statistic: -5.000
# p-value: 0.001

Step 4: Determine P-Value

# P-value = probability of a result at least this extreme if H0 is true
# Low p-value = data this extreme would be rare if there were no real difference

if p_value < alpha:
    print(f"p-value ({p_value:.3f}) < alpha ({alpha})")
    print("Result is statistically significant")
else:
    print(f"p-value ({p_value:.3f}) >= alpha ({alpha})")
    print("Result is NOT statistically significant")

# Output:
# p-value (0.001) < alpha (0.05)
# Result is statistically significant

Step 5: Draw Conclusion

if p_value < alpha:
    print("Reject H0: New design has significantly different conversion")
else:
    print("Fail to reject H0: No significant difference detected")

# Output:
# Reject H0: New design has significantly different conversion

# Important: "Fail to reject" ≠ "Accept H0"
# We never prove H0 true, just fail to find evidence against it

Which statistical test should you use?

Pick the test from your data type and design: a t-test compares a mean with a target value or compares two means (independent groups, or paired before/after measurements), a z-test compares means in large samples, and a chi-square test checks whether two categorical variables are related. Conversion and open rates are proportions, so compare them with a two-proportion z-test or a chi-square test, as in the A/B example below.

T-Test (Compare Two Groups)

# One sample t-test (compare to known value)
stats.ttest_1samp(data, popmean=100)

# Independent t-test (two separate groups)
stats.ttest_ind(group1, group2)

# Paired t-test (before/after on same subjects)
stats.ttest_rel(before, after)

Z-Test (Large Samples, Known Variance)

from statsmodels.stats.weightstats import ztest

# For large samples (n > 30)
z_stat, p_value = ztest(group1, group2)

Chi-Square Test (Categorical Data)

# Test relationship between categorical variables
from scipy.stats import chi2_contingency

# Contingency table
observed = [[10, 20], [30, 40]]
chi2, p_value, dof, expected = chi2_contingency(observed)

print(f"Chi-square: {chi2:.3f}, p-value: {p_value:.3f}")

How do you run a hypothesis test on A/B test results?

Treat open or conversion rates as proportions and run a two-proportion z-test on the raw counts. With 120 opens from 1,000 control emails and 145 from 1,000 variant emails, z = 1.649 and p = 0.099, so the 2.5-point lift is not statistically significant at α = 0.05. Before you declare a winner, rerun the test with a larger sample sized in advance (see Power Analysis below).

from math import sqrt
from scipy.stats import norm

# Scenario: testing two email subject lines
opens_a, n_a = 120, 1000   # control
opens_b, n_b = 145, 1000   # variant

p_a, p_b = opens_a / n_a, opens_b / n_b
p_pool = (opens_a + opens_b) / (n_a + n_b)
se = sqrt(p_pool * (1 - p_pool) * (1 / n_a + 1 / n_b))

z = (p_b - p_a) / se
p_value = 2 * norm.sf(abs(z))  # two-tailed

alpha = 0.05
print(f"Control: {p_a:.1%}  Variant: {p_b:.1%}  Lift: {p_b - p_a:.1%}")
print(f"z = {z:.3f}, p-value = {p_value:.3f}")
if p_value < alpha:
    print("Reject H0: the subject lines perform differently")
else:
    print("Fail to reject H0: not significant at alpha = 0.05")

# Output:
# Control: 12.0%  Variant: 14.5%  Lift: 2.5%
# z = 1.649, p-value = 0.099
# Fail to reject H0: not significant at alpha = 0.05

If you use statsmodels, proportions_ztest([145, 120], [1000, 1000]) returns the same z and p-value.

What does the p-value mean in hypothesis testing?

The p-value is the probability of getting a result at least as extreme as yours if the null hypothesis were true. In the email test, p = 0.099 means that if both subject lines had the same true open rate, a gap of 2.5 points or more in either direction would show up in about 10% of tests this size.

It is not the probability that H0 is true, and it says nothing about how large or useful the effect is. With a million emails per group, a lift from 12.0% to 12.2% gives p < 0.001 even though it barely matters. That is why you report the effect with a confidence interval: for the email test, the 95% interval for the lift runs from about −0.5 to +5.5 percentage points, so the data fit both no effect and a sizable one.

Should you use a one-tailed or two-tailed test?

Use a two-tailed test unless you decided before collecting data that only one direction matters and you would treat a result in the other direction the same as no effect. A one-tailed test halves the p-value when the effect goes the predicted way: the email test's two-tailed p = 0.099 becomes 0.0496, just under 0.05. Switching to one tail after seeing the results to cross that line is a form of p-hacking.

Understanding Errors

Every test can be wrong in two ways. A Type I error (false positive) rejects a null hypothesis that is true and happens with probability α; a Type II error (false negative) misses a real effect and happens with probability β, where power = 1 − β.

Type I Error (False Positive)

# Rejecting H0 when it's actually true
# Controlled by alpha (significance level)
# Example: Saying new design works when it doesn't

Type II Error (False Negative)

# Failing to reject H0 when it's actually false
# Probability = Beta (β)
# Power = 1 - β (ability to detect true effect)
# Example: Missing that new design actually works better

Power Analysis

Power analysis tells you how many observations you need before you run the test. To detect a difference of 0.3 standard deviations with a two-sided t-test at α = 0.05 and 80% power, you need 176 per group; halving the effect you want to detect (to 0.15) raises that to 699, roughly four times as many.

import math
from statsmodels.stats.power import TTestIndPower

# Sample size per group for a two-sample t-test
effect_size = 0.3  # Expected difference in standard deviations (Cohen's d)
alpha = 0.05
power = 0.8        # 80% chance to detect the effect if it exists

n = TTestIndPower().solve_power(effect_size=effect_size, alpha=alpha, power=power)
print(f"Need {math.ceil(n)} samples per group")

# Output:
# Need 176 samples per group

Common Pitfalls

Most wrong conclusions come from how a test was run, not from the math: testing until something turns significant, using a sample too small to detect the effect, and treating a significant result as an important one.

• P-hacking: Testing multiple hypotheses until finding significance
• Sample size: Too small = miss real effects
• Confusing practical vs statistical: Significant doesn't always mean important
• Multiple comparisons: Need Bonferroni correction

Best Practices

Fix the hypotheses, α, test and sample size before you see the data, then report the effect size and its confidence interval alongside the p-value.

• Define hypotheses BEFORE collecting data
• Use appropriate test for your data type
• Check assumptions (normality, equal variance)
• Consider both statistical and practical significance
• Report effect size, not just p-value