Calculating A/B Test Sample Size

To calculate A/B test sample size, you need four inputs: the baseline conversion rate, the minimum detectable effect (MDE), the significance level (usually 5%) and the statistical power (usually 80%). The smaller the effect you want to detect, the more users you need: detecting a lift from a 10% to a 12% conversion rate at those settings takes 3,839 users per variant.

What inputs do you need to calculate A/B test sample size?

You need four numbers, all chosen before the test starts: the baseline conversion rate, the minimum detectable effect, the significance level and the statistical power. The first two describe your metric and the change worth detecting; the last two set how much risk you accept of a false positive and of missing a real effect (see statistical power explained).

1. Baseline conversion rate (current performance, from recent data)
2. Minimum detectable effect (smallest change you care about; say whether it is absolute or relative)
3. Statistical power (usually 80% = 0.80, the chance of detecting the effect if it is real)
4. Significance level (usually 5% = 0.05, the false positive rate you accept)

How many users do you need for an A/B test?

For a 10% baseline click-through rate, detecting a 2-percentage-point lift to 12% at 5% significance and 80% power takes 3,839 users per variant, or 7,678 in total. Plug your own baseline and target rate into the formula below to get your number.

Current: 10% click-through rate
Want to detect: 2% absolute increase (to 12%)
Power: 80%
Significance: 5%

n per variant = (z_α/2 + z_β)² × [p1(1 − p1) + p2(1 − p2)] / (p2 − p1)²

z_α/2 = 1.960   (two-sided test at 5% significance)
z_β   = 0.8416  (80% power)
p1 = 0.10, p2 = 0.12

n = (1.960 + 0.8416)² × (0.10 × 0.90 + 0.12 × 0.88) / 0.02²
  = 7.849 × 0.1956 / 0.0004
  = 3,838.1  → round up to 3,839 per variant

The same calculation in Python, using only scipy:

import math
from scipy.stats import norm

def sample_size_per_group(p1, p2, alpha=0.05, power=0.80):
    z_alpha = norm.ppf(1 - alpha / 2)   # 1.96 for a two-sided 5% test
    z_beta = norm.ppf(power)            # 0.84 for 80% power
    variance = p1 * (1 - p1) + p2 * (1 - p2)
    return math.ceil((z_alpha + z_beta) ** 2 * variance / (p2 - p1) ** 2)

print(sample_size_per_group(0.10, 0.12))  # 3839

Calculators use slightly different variance approximations (pooled variance, an arcsine transformation), so expect answers that differ by up to a few hundred users at this scale. Any of them is fine for planning; what matters is fixing the number before the test starts.

What are the most common A/B test sample size mistakes?

The most common mistake is not fixing the sample size in advance and instead checking results repeatedly, stopping as soon as p < 0.05. Checking at 10 evenly spaced points and stopping at the first significant result raises the false positive rate from 5% to about 19%, even when the variants are identical.

• Running the test "until significant": every extra look is another chance for noise to cross the threshold. If you must monitor continuously, use a sequential testing method designed for repeated looks.
• Stopping early when winning: you are cherry-picking lucky noise, and early leads tend to shrink as more data arrives (regression to the mean).
• Not deciding the minimum effect size upfront: this leads to underpowered tests. With half the required sample (1,920 per variant in the example), power drops from 80% to about 51%, a coin flip's chance of detecting a real 2-point lift.
• Testing several variants at the full 5% level: each extra comparison adds false positive risk. With a Bonferroni correction for three variants against one control (α = 0.05 / 3), the example needs 5,120 users per variant instead of 3,839; see the multiple comparisons problem.

Rule of Thumb

Smaller effects need bigger samples. Sample size grows with the inverse square of the effect, so halving the minimum detectable effect roughly quadruples the users you need, and detecting a 1% relative lift instead of a 10% one takes roughly 100x more data. For a quick estimate at 5% significance and 80% power, use n ≈ 16 × p(1 − p) / δ² per variant, where δ is the absolute difference: 16 × 0.10 × 0.90 / 0.02² = 3,600, slightly below the exact 3,839.

Should the minimum detectable effect be absolute or relative?

Either works, but always say which one you mean, because the same "2%" describes very different tests. On a 10% baseline, a 2% absolute lift means 10% → 12% and needs 3,839 users per variant, while a 2% relative lift means 10% → 10.2% and needs 356,332 users per variant, about 93 times as many.

How long should an A/B test run?

Divide the total sample by the number of eligible users entering the test each day, then round up to whole weeks. With 7,678 users needed and 1,000 eligible visitors a day, the math says 7.7 days, so run for two full weeks: whole weeks balance weekday and weekend behavior, and a longer run gives any novelty effect time to fade.

How do you calculate sample size for revenue or other continuous metrics?

Replace p(1 − p) with the metric's variance: n per variant = 2 × (z_α/2 + z_β)² × σ² / δ², where σ is the standard deviation and δ the difference you want to detect. If average order value has a standard deviation of $40 and you want to detect a $2 change, n = 2 × 7.849 × 40² / 2² ≈ 6,280 orders per variant. Revenue is usually heavily skewed, so estimate σ from recent data rather than guessing; a few very large orders can inflate it substantially.