P-Value Explained

A p-value is the probability of getting results at least as extreme as the ones you observed if the null hypothesis (no real effect or difference) were true. A small p-value, conventionally below 0.05, means your data would be unusual if there were no effect, so you reject the null hypothesis; it does not tell you the probability that the null hypothesis is true or how large the effect is.

How do you interpret a p-value in plain English?

Read it as "how surprising would this data be if nothing were going on?" If you test a new website design and see 10% more sales, the p-value is how often a lift at least that large would show up by chance alone if the design made no difference at all.

• P-value = 0.03 (3%): with no real effect, a lift this large would appear in only about 3% of experiments, so the data would be surprising under "no effect" and the result is significant at 0.05
• P-value = 0.40 (40%): a lift this large would appear in about 40% of experiments even with no real effect, so the data are entirely consistent with luck

What does a p-value below 0.05 mean?

It means the result is statistically significant at the conventional 5% significance level (alpha = 0.05): data this extreme would occur less than 5% of the time if the null hypothesis were true, so you reject the null hypothesis.

• p < 0.05: reject the null hypothesis (statistically significant)
• p ≥ 0.05: fail to reject the null hypothesis; this is not proof that there is no effect, only that the data are not strong enough to rule out chance

Note: 0.05 is a convention, not a law. Some fields use 0.01 or 0.10, and you should choose your threshold before you look at the data.

What is a good p-value?

There is no universally good p-value: the smaller it is, the stronger the evidence against the null hypothesis, and 0.05 is only the most common cut-off. Use this table as a rough guide to the strength of evidence, with decisions based on a 0.05 significance level.

P-Value	Meaning	Decision
< 0.001	Very strong evidence	Reject null hypothesis
0.001 - 0.01	Strong evidence	Reject null hypothesis
0.01 - 0.05	Moderate evidence	Reject null hypothesis
0.05 - 0.10	Weak evidence	Cannot reject null at 0.05 (sometimes called marginal)
> 0.10	Little to no evidence	Cannot reject null

Real Example: A/B Test

A button test that lifts the click rate from 5.0% to 6.5% looks like a win, but a chi-square test on 1,000 visitors per variant gives p = 0.18, well above 0.05, so the data cannot rule out chance. A difference this size needs a larger sample to reach significance; the A/B test sample size guide shows how to plan for it.

Scenario: Testing two website buttons

Control (A):  1000 visitors, 50 clicks (5.0% click rate)
Treatment (B): 1000 visitors, 65 clicks (6.5% click rate)

Question: Is B really better, or just luck?

# Python: Chi-square test
from scipy.stats import chi2_contingency

observed = [[50, 950], [65, 935]]  # clicks, no-clicks
chi2, p_value, dof, expected = chi2_contingency(observed)

print(f"P-value: {p_value:.4f}")
# P-value: 0.1787

# Interpretation: p = 0.18 > 0.05
# Cannot reject null hypothesis
# A difference this large is common under pure chance

Python Examples

SciPy returns the p-value directly from each test: ttest_ind compares the means of two groups, pearsonr tests whether a correlation differs from zero, and ttest_1samp compares one sample's mean with a fixed value.

from scipy import stats
import numpy as np

# Example 1: T-test (comparing two groups)
group_a = [23, 25, 28, 22, 24]  # Control
group_b = [30, 32, 35, 29, 33]  # Treatment

t_stat, p_value = stats.ttest_ind(group_a, group_b)
print(f"P-value: {p_value:.4f}")  # 0.0011

if p_value < 0.05:
    print("Statistically significant difference")
else:
    print("No significant difference")

# Example 2: Correlation test
x = [1, 2, 3, 4, 5]
y = [2, 4, 5, 4, 5]

correlation, p_value = stats.pearsonr(x, y)
print(f"Correlation: {correlation:.3f}")  # 0.775
print(f"P-value: {p_value:.4f}")  # 0.1240

# Example 3: One-sample t-test
# Is mean different from hypothesized value?
data = [10.2, 9.8, 10.5, 10.1, 9.9]
t_stat, p_value = stats.ttest_1samp(data, 10)  # Test if mean = 10
print(f"P-value: {p_value:.4f}")  # 0.4601

The correlation of 0.775 looks strong but is not significant (p = 0.12) because it rests on only five points: with a small sample, even a large effect can produce a high p-value.

Common Misconceptions

Most p-value mistakes come from reading it as the probability that a hypothesis is true or as a measure of how big an effect is; it is neither.

• Wrong: "P-value is the probability the null hypothesis is true"
• Right: "P-value is the probability of seeing data at least this extreme IF the null hypothesis is true"

• Wrong: "P < 0.05 proves the effect is real"
• Right: "P < 0.05 means data this extreme would be unusual if there were no effect; it is evidence, not proof"

• Wrong: "Lower p-value = bigger effect"
• Right: "Lower p-value = stronger evidence against the null, which can come from a large sample rather than a large effect"

P-Value vs Effect Size

A p-value tells you how strong the evidence is against the null hypothesis. Effect size tells you how big the effect is.

# Small effect, large sample → low p-value (significant)
# Large effect, small sample → high p-value (not significant)

# Always report both!
print(f"P-value: {p_value:.4f}")
print(f"Effect size (Cohen's d): {effect_size:.2f}")

SQL Implementation

Most SQL databases have no built-in p-value function, but SQL can produce the per-group mean, standard deviation and sample size that a t-test needs, which you then pass to Python or a calculator. A t-test needs the sample standard deviation: STDDEV returns it in PostgreSQL, but in MySQL STDDEV is the population version, so use STDDEV_SAMP there.

-- While SQL doesn't calculate p-values directly,
-- you can prepare data for statistical tests

-- Example: Prepare data for t-test
SELECT
    experiment_group,
    AVG(conversion_rate) as mean_conversion,
    STDDEV(conversion_rate) as std_dev,
    COUNT(*) as sample_size
FROM ab_test_results
GROUP BY experiment_group;

Important Limitations

A p-value only measures how compatible your data are with the null hypothesis, so sample size, the number of tests you run and selective reporting can all make it misleading.

• Sample size matters: Tiny samples rarely give p < 0.05, even with real effects
• Multiple testing: Testing many hypotheses inflates false positives
• Not everything: P-value doesn't tell you about practical importance
• Publication bias: Studies with p < 0.05 more likely to be published

Best Practices

Fix your significance level before you collect data, and report the p-value together with the effect size and a confidence interval so readers can judge both the strength of the evidence and whether the effect matters.

• Pre-register: Decide significance level before testing
• Report effect size: Don't just report p-value
• Report confidence intervals: More informative than p-value alone
• Consider context: 0.05 threshold isn't sacred
• Replicate: One low p-value doesn't prove anything