A/B Testing Explained

A/B testing (split testing) is a randomized experiment that compares two versions of a page, email or feature to find out which performs better. You randomly assign users to version A (the control) or version B (the treatment), measure one outcome such as conversion rate, and use a significance test to check whether the difference is larger than random chance would produce.

What is an example of an A/B test?

A classic A/B test shows half your visitors a blue button and the other half a red one, then checks whether the difference in click rate is real or just luck. If red gets 65 clicks from 1,000 visitors (6.5%) and blue gets 50 from 1,000 (5.0%), red shows a 30% relative lift, but a chi-square test gives p = 0.18, so with this little traffic the gap could easily be random noise.

• Version A (control): Blue button → 50 clicks / 1000 visitors (5.0%)
• Version B (treatment): Red button → 65 clicks / 1000 visitors (6.5%)
• Question: Is red really better, or just random luck?

How do you run an A/B test step by step?

Run an A/B test in five steps: write a hypothesis, choose one primary metric, calculate the sample size you need, randomly split traffic until you reach that size, then analyze the result with a significance test. Fixing the hypothesis, metric and sample size before launch is what keeps the final p-value trustworthy.

1. Define Hypothesis

# Null hypothesis (H0): No difference between A and B
# Alternative hypothesis (H1): A and B differ (two-sided)

# Example:
# H0: Red button has same click rate as blue button
# H1: Red button has a different click rate than blue button

2. Choose Metric

• Conversion rate: % who clicked/bought/signed up
• Revenue per user: Average $ per visitor
• Engagement: Time on site, pages viewed
• Primary metric: Main goal you're optimizing

3. Calculate Sample Size

import math
from scipy.stats import norm

# Parameters
baseline_rate = 0.05   # Current conversion rate (5%)
target_rate = 0.06     # Smallest result worth detecting (5% to 6%)
alpha = 0.05           # Significance level (two-sided)
power = 0.80           # 80% chance to detect a real effect

z_alpha = norm.ppf(1 - alpha / 2)   # 1.96
z_beta = norm.ppf(power)            # 0.84
p_avg = (baseline_rate + target_rate) / 2

# Required sample size per group (two-proportion z-test)
sd_null = math.sqrt(2 * p_avg * (1 - p_avg))
sd_alt = math.sqrt(baseline_rate * (1 - baseline_rate) + target_rate * (1 - target_rate))
n = ((z_alpha * sd_null + z_beta * sd_alt) / (target_rate - baseline_rate)) ** 2

print(f"Need {math.ceil(n):,} visitors per group")
# Need 8,158 visitors per group (16,316 total)

4. Run the Test

import random

def assign_variant(user_id):
    """Randomly assign user to A or B"""
    random.seed(user_id)  # Consistent assignment per user
    return 'A' if random.random() < 0.5 else 'B'

# Track results
results = {
    'A': {'visitors': 0, 'conversions': 0},
    'B': {'visitors': 0, 'conversions': 0}
}

# For each visitor:
variant = assign_variant(user_id)
results[variant]['visitors'] += 1
if user_converted:
    results[variant]['conversions'] += 1

5. Analyze Results

from scipy.stats import chi2_contingency
import numpy as np

# Your data
a_conversions = 50
a_visitors = 1000
b_conversions = 65
b_visitors = 1000

# Create contingency table
observed = [
    [a_conversions, a_visitors - a_conversions],
    [b_conversions, b_visitors - b_conversions]
]

# Chi-square test
chi2, p_value, dof, expected = chi2_contingency(observed)

# Calculate conversion rates
a_rate = a_conversions / a_visitors
b_rate = b_conversions / b_visitors
lift = (b_rate - a_rate) / a_rate * 100

print(f"A conversion rate: {a_rate:.1%}")
print(f"B conversion rate: {b_rate:.1%}")
print(f"Relative lift: {lift:.1f}%")
print(f"P-value: {p_value:.4f}")

if p_value < 0.05:
    print("Statistically significant: B wins.")
else:
    print("Not significant: could be random chance.")

# A conversion rate: 5.0%
# B conversion rate: 6.5%
# Relative lift: 30.0%
# P-value: 0.1787
# Not significant: could be random chance.

What does statistically significant mean in an A/B test?

A result is statistically significant when its p-value is below the significance level (alpha) you chose before the test, usually 0.05. The p-value is the probability of seeing a difference at least as large as yours if A and B actually performed the same, so p = 0.03 means a gap this big would show up only 3% of the time when there is no real difference. It is not the probability that B is better.

# Common significance levels:
# p < 0.05 → 95% confidence (common standard)
# p < 0.01 → 99% confidence (more stringent)
# p < 0.10 → 90% confidence (more lenient)

# Example interpretation:
if p_value < 0.05:
    print("If A and B performed the same, a gap this large would occur less than 5% of the time")
    print("Reject null hypothesis")
    print("B is significantly different from A")

Calculating Confidence Intervals

A confidence interval gives the range of plausible values for each version's conversion rate, so you see how precise the estimate is and not just whether it passed a threshold. The function below uses the normal approximation, which is reasonable when each group has plenty of conversions (a common rule of thumb is at least 10 conversions and 10 non-conversions).

import numpy as np
import scipy.stats as stats

def proportion_ci(successes, trials, confidence=0.95):
    """Calculate confidence interval for proportion"""
    rate = successes / trials
    z = stats.norm.ppf((1 + confidence) / 2)
    se = np.sqrt(rate * (1 - rate) / trials)
    margin = z * se
    return (rate - margin, rate + margin)

# Example
a_ci = proportion_ci(50, 1000)
b_ci = proportion_ci(65, 1000)

print(f"A: {a_ci[0]:.1%} to {a_ci[1]:.1%}")
print(f"B: {b_ci[0]:.1%} to {b_ci[1]:.1%}")
# A: 3.6% to 6.4%
# B: 5.0% to 8.0%

# If confidence intervals don't overlap → significant difference
# If they overlap (like these), run a proper test before concluding anything

Overlapping intervals do not prove there is no difference; see overlapping confidence intervals for why, and how to compute the interval for the difference itself.

Common Pitfalls

Most broken A/B tests fail on process rather than math: stopping as soon as the p-value dips below 0.05, testing many variants or metrics without adjusting alpha, unequal traffic splits, and changing the test while it runs. Each one inflates the false positive rate or biases the comparison.

1. Stopping Too Early (Peeking)

# BAD: Check results every day, stop when p < 0.05
# This inflates false positive rate!

# GOOD: Calculate sample size upfront, wait until reached
target_sample_size = 10000
if current_sample_size >= target_sample_size:
    analyze_results()  # Only check when planned sample size reached

2. Not Accounting for Multiple Comparisons

# Testing 10 variants? Need Bonferroni correction
num_tests = 10
adjusted_alpha = 0.05 / num_tests  # 0.005
# Use adjusted_alpha as significance threshold

Bonferroni is the simplest correction but also the strictest; the multiple comparisons guide compares it with Benjamini-Hochberg.

3. Ignoring Sample Ratio Mismatch

# Check if A and B got equal traffic
expected_ratio = 0.5
actual_ratio = a_visitors / (a_visitors + b_visitors)

if abs(actual_ratio - expected_ratio) > 0.02:
    print("Sample ratio mismatch! Check randomization.")

A fixed tolerance like this is only a rough first check. For a formal test, run a chi-square goodness-of-fit test on the visitor counts, as shown in the sample ratio mismatch tip.

4. Changing Test Mid-Flight

• Don't change variant design during test
• Don't change success metric mid-test
• Don't extend test duration to achieve significance

Advanced: Sequential Testing

Sequential testing lets you look at results during the test and stop early without inflating the false positive rate, because each planned look uses a stricter significance threshold than a single fixed-sample test would. It only works if you choose the sequential design before launch; checking an ordinary fixed-sample test every day is still peeking.

# Alternative to fixed-sample testing
# Each planned interim look uses an adjusted threshold,
# so the overall false positive rate stays at alpha

# Common approaches:
# - Group sequential designs (O'Brien-Fleming or Pocock boundaries)
# - Mixture SPRT (mSPRT) / always-valid p-values

# Benefit: Can stop test early if clear winner emerges

Real-World Example

In this checkout test the treatment lifts conversion from 5.00% to 5.70% (a 14% relative lift) and revenue per visitor from $2.50 to $2.85, yet the chi-square p-value is 0.13, so the result is not statistically significant. Detecting a change from 5.0% to 5.7% at 80% power needs about 16,200 visitors per group, more than three times the 5,000 this test collected.

# E-commerce checkout test
test_data = {
    'control': {
        'visitors': 5000,
        'purchases': 250,
        'revenue': 12500
    },
    'treatment': {
        'visitors': 5000,
        'purchases': 285,
        'revenue': 14250
    }
}

# Calculate metrics
for variant in ['control', 'treatment']:
    data = test_data[variant]
    conv_rate = data['purchases'] / data['visitors']
    revenue_per_visitor = data['revenue'] / data['visitors']

    print(f"{variant.title()}:")
    print(f"  Conversion rate: {conv_rate:.2%}")
    print(f"  Revenue per visitor: ${revenue_per_visitor:.2f}")

# Statistical test
observed = [
    [250, 4750],  # control: conversions, non-conversions
    [285, 4715]   # treatment: conversions, non-conversions
]
chi2, p_value, dof, expected = chi2_contingency(observed)

print(f"\nP-value: {p_value:.4f}")
if p_value < 0.05:
    print("Winner: Treatment!")
    lift = (285/5000 - 250/5000) / (250/5000) * 100
    print(f"Lift: {lift:.1f}%")
else:
    print("No significant winner yet")

# Control:
#   Conversion rate: 5.00%
#   Revenue per visitor: $2.50
# Treatment:
#   Conversion rate: 5.70%
#   Revenue per visitor: $2.85
#
# P-value: 0.1308
# No significant winner yet

When to Use A/B Testing

Use A/B testing when you can randomly assign users to versions of something you control and have enough traffic to reach the required sample size within a few weeks. It is a poor fit for changes you cannot randomize, and for low-traffic pages where reaching the sample size would take months.

• ✅ Website/app changes (buttons, copy, layouts)
• ✅ Email campaigns (subject lines, send times)
• ✅ Pricing strategies
• ✅ Product features
• ✅ Marketing campaigns

Best Practices

Decide the hypothesis, primary metric, sample size and duration before launch, then leave the test alone until it reaches that sample size.

• One change at a time: Test single variable
• Calculate sample size upfront: Don't guess
• Run for full weeks: Account for day-of-week effects
• Randomize properly: Truly random assignment
• Wait for significance: Don't peek early
• Consider practical significance: Is 0.1% lift worth it?
• Document everything: Hypothesis, metrics, results

Tools for A/B Testing

Most teams use an experimentation platform to handle random assignment and tracking, then check the statistics in Python or R. Google Optimize, once the common free option, was shut down by Google in September 2023.

• Optimizely: Enterprise platform
• VWO: Visual editor, easy setup
• Python libraries: scipy, statsmodels
• R: pwr package for power analysis