Confidence Intervals Explained

A confidence interval is a range around an estimate that shows how precisely your data pins down the true value. A click-through rate reported as 10% (95% CI: 8% to 12%) means the data is consistent with a true rate anywhere from about 8% to 12%, and the 95% is how often the method that built the interval captures the true value.

What does a 95% confidence interval actually mean?

A 95% confidence interval means the method used to build it captures the true value 95% of the time across repeated samples. It does not mean there is a 95% probability that the true value sits inside the one interval you calculated: the true value is fixed, and your interval either contains it or does not.

Your click-through rate is 10% (95% CI: 8% - 12%)

Wrong interpretation: "There's a 95% chance the true value is between 8-12%"

Right interpretation: "If I repeated this measurement 100 times, about 95 of those intervals would contain the true value"

If you want a direct probability statement about where the true value lies, you need a Bayesian credible interval instead; see Bayesian vs frequentist A/B testing.

How do you calculate a 95% confidence interval?

Add and subtract 1.96 standard errors from your estimate: CI = estimate ± 1.96 × SE. For a proportion, SE = √(p(1 − p)/n); for a mean, SE = s/√n, where s is the sample standard deviation, and with small samples you replace 1.96 with the critical value from the t distribution (2.36 for 8 values).

Proportion: 90 clicks from 900 impressions
p      = 90 / 900               = 0.10
SE     = √(0.10 × 0.90 / 900)   = 0.01
margin = 1.96 × 0.01            = 0.0196
95% CI = 0.10 ± 0.0196          = 8.0% to 12.0%

The same calculation in Python, plus a t-based interval for the mean of a small sample:

import math
from scipy import stats

# Proportion: 90 clicks from 900 impressions
clicks, impressions = 90, 900
p = clicks / impressions                   # 0.10
se = math.sqrt(p * (1 - p) / impressions)  # 0.01
z = stats.norm.ppf(0.975)                  # 1.96
print(f"CTR: {p:.1%} (95% CI: {p - z*se:.1%} to {p + z*se:.1%})")

# Mean of a small sample: use the t distribution
order_values = [42, 55, 38, 61, 47, 52, 44, 59]
mean = sum(order_values) / len(order_values)
se_mean = stats.sem(order_values)          # sample std / sqrt(n)
low, high = stats.t.interval(0.95, len(order_values) - 1, loc=mean, scale=se_mean)
print(f"Mean: {mean:.2f} (95% CI: {low:.2f} to {high:.2f})")

# CTR: 10.0% (95% CI: 8.0% to 12.0%)
# Mean: 49.75 (95% CI: 42.80 to 56.70)

What does a wide confidence interval tell you?

A wide interval means high uncertainty: the data is consistent with a broad range of true values, so you need more data before acting on the estimate. A narrow interval means high precision and an estimate you can act on with more confidence.

Width shrinks with the square root of the sample size, so halving the margin of error takes about four times as much data. At a 10% click-through rate, 900 impressions give a margin of about ±2 percentage points (8.0% to 12.0%), while 3,600 impressions give about ±1 point (9.0% to 11.0%). More variable data and a higher confidence level also widen the interval.

What is the difference between a 95% and a 99% confidence interval?

A 99% interval is wider because it uses 2.576 standard errors instead of 1.96. For 90 clicks from 900 impressions, the 99% interval runs from 7.4% to 12.6% instead of 8.0% to 12.0%: you gain a higher capture rate and give up precision. A 90% interval uses 1.645 standard errors and is narrower still; 95% is the usual default in analytics and A/B testing.

Overlapping CIs

If two confidence intervals overlap, the difference might not be statistically significant. But you need a proper test to be sure: two 95% intervals can overlap a little and the difference can still be significant at p < 0.05. If the intervals of two independent groups do not overlap at all, the difference is significant. The overlapping confidence intervals tip shows how to test the difference directly.

What are common mistakes with confidence intervals?

The most common mistake is reading a 95% interval as a 95% probability that the true value is inside it. Also watch for confusing the standard deviation with the standard error, applying ±1.96 × SE to very small counts, and judging significance from overlapping intervals.

• Using the standard deviation instead of the standard error: mean ± 1.96 × SD describes where about 95% of individual values fall in normal data, not how precisely you know the mean.
• Using ±1.96 × SE with very few successes: 2 conversions from 40 visitors gives −1.8% to 11.8%, an impossible negative lower bound. Use the Wilson interval instead (below).
• Comparing overlap instead of testing the difference: build an interval for the difference between groups rather than eyeballing two separate intervals.

from scipy import stats

result = stats.binomtest(2, 40)   # 2 conversions from 40 visitors
ci = result.proportion_ci(confidence_level=0.95, method='wilson')
print(f"{ci.low:.1%} to {ci.high:.1%}")
# 1.4% to 16.5%

Why They're Better Than P-Values

A confidence interval carries the same significance information as a p-value plus the size and precision of the effect. When the interval and the test use the same standard error, a 95% interval for a difference that excludes zero corresponds to p < 0.05 in a two-sided test, and the interval also shows how large or small the true effect could plausibly be.

• ✅ Show effect size (not just significance)
• ✅ Display uncertainty visually
• ✅ Help with practical decisions
• ✅ Don't rely on arbitrary thresholds