Standard Deviation Explained

Standard deviation measures how spread out values are around the mean. It is the square root of the variance (the average squared distance from the mean), so it is in the same units as your data: a low standard deviation means values cluster near the mean, and a high one means they are spread out.

What does standard deviation tell you?

Standard deviation tells you how consistent values are around the average, which the mean alone hides. Two classes can both average 75 on a test while one has a sample standard deviation of 1.58 (scores from 73 to 77) and the other 20.62 (scores from 50 to 100).

• Class A scores: [73, 74, 75, 76, 77] → Standard deviation: 1.58 (tight cluster)
• Class B scores: [50, 60, 75, 90, 100] → Standard deviation: 20.62 (wide spread)

Same mean, very different distributions. For roughly normal data, standard deviation also tells you where most values fall: about 68% lie within 1 standard deviation of the mean, 95% within 2 and 99.7% within 3. With a mean score of 75 and a standard deviation of 10, about 68% of scores would land between 65 and 85 and about 95% between 55 and 95 (more on this in the normal distribution tip).

The 68-95-99.7 rule does not hold for skewed data such as revenue per user. For any distribution, Chebyshev's inequality only guarantees that at least 75% of values fall within 2 standard deviations and at least 8/9 (about 88.9%) within 3.

What is the formula for standard deviation?

The population standard deviation is σ = √(Σ(x − μ)² / N): subtract the mean from each value, square the differences, average them, and take the square root. The sample standard deviation s follows the same steps but divides by n − 1 instead of N.

Population:  σ = √(Σ(x - μ)² / N)
Sample:      s = √(Σ(x - x̄)² / (n - 1))

Where:
- x = each value
- μ = population mean, x̄ = sample mean
- N = number of values in the population, n = number in the sample
- Σ = sum of

How do you calculate standard deviation by hand?

Find the mean, subtract it from every value, square each difference, average the squared differences to get the variance, then take the square root. For the values 4, 8, 6, 5 and 3, the population standard deviation is 1.72 (divide by 5) and the sample standard deviation is 1.92 (divide by 4).

Data: [4, 8, 6, 5, 3]

Step 1: Calculate mean
mean = (4 + 8 + 6 + 5 + 3) / 5 = 5.2

Step 2: Find differences from mean
4 - 5.2 = -1.2
8 - 5.2 = 2.8
6 - 5.2 = 0.8
5 - 5.2 = -0.2
3 - 5.2 = -2.2

Step 3: Square the differences
(-1.2)² = 1.44
(2.8)²  = 7.84
(0.8)²  = 0.64
(-0.2)² = 0.04
(-2.2)² = 4.84

Step 4: Calculate variance (average of squared differences)
variance = (1.44 + 7.84 + 0.64 + 0.04 + 4.84) / 5 = 2.96

Step 5: Take square root
standard deviation = √2.96 = 1.72

Sample version (divide by n - 1 = 4 in step 4):
variance = 14.8 / 4 = 3.7
standard deviation = √3.7 = 1.92

Python Implementation

Use NumPy's np.std(), but note that it defaults to the population formula (ddof=0); pass ddof=1 for the sample standard deviation. Pandas .std() and Python's built-in statistics.stdev() default to the sample formula, while statistics.pstdev() gives the population value.

import numpy as np

data = [4, 8, 6, 5, 3]

# Method 1: NumPy (easiest)
std_dev = np.std(data)  # Population std dev
print(f"Standard deviation: {std_dev:.2f}")  # 1.72

# Method 2: Sample standard deviation (use ddof=1)
std_dev_sample = np.std(data, ddof=1)  # 1.92

# Method 3: Manual calculation
mean = np.mean(data)
squared_diffs = [(x - mean)**2 for x in data]
variance = sum(squared_diffs) / len(data)
std_dev_manual = variance ** 0.5

# Also get variance
variance = np.var(data)  # 2.96

Population vs Sample Standard Deviation

Use the population formula (divide by N) only when your data includes every member of the group you care about; use the sample formula (divide by n − 1) when the data is a sample used to estimate a larger population, which is the usual case in analysis. Dividing by n − 1, called Bessel's correction, removes the bias that makes dividing a sample's squared deviations by n underestimate the population variance.

• Population (σ): Divide by N (when you have all data)
• Sample (s): Divide by N-1 (when you have a sample)

# Python
np.std(data)           # Population (N)
np.std(data, ddof=1)   # Sample (N-1)

# Pandas defaults to sample (N-1)
df['column'].std()     # Sample standard deviation

This is why the same five values give 1.72 in NumPy or Excel's STDEV.P and 1.92 in pandas or STDEV.S. Check which formula each tool uses before comparing results.

Interpreting Standard Deviation

There is no universally good or bad value: judge a standard deviation against the mean and the units of the data. The coefficient of variation (standard deviation ÷ mean) puts different scales on equal footing, since a standard deviation of 5 is 5% of a mean of 100 but 25% of a mean of 20.

• Low std dev relative to the mean: Data is consistent and predictable
• Medium std dev: Some variability, but not extreme
• High std dev relative to the mean: Data is scattered and unpredictable

Real-World Examples

In quality control a small standard deviation means a consistent process, in finance the standard deviation of returns is the usual measure of volatility, and in data cleaning values more than 2 or 3 standard deviations from the mean are candidates for outliers.

# Example 1: Quality control
widget_weights = [100.1, 99.9, 100.0, 100.2, 99.8]
std = np.std(widget_weights)  # 0.14
# Low std dev = consistent manufacturing

# Example 2: Stock volatility
stock_returns = [0.05, -0.03, 0.08, -0.12, 0.15, -0.08]
volatility = np.std(stock_returns)  # 0.094
# High std dev = risky stock

# Example 3: Outlier detection
data = [10, 12, 11, 13, 12, 85]  # 85 is outlier
mean = np.mean(data)
std = np.std(data)
# Values beyond mean ± 2*std are often outliers
outliers = [x for x in data if abs(x - mean) > 2*std]
print(outliers)  # [85]

Be careful with small samples: the single value of 85 inflates the standard deviation from about 1.0 (for the other five values) to 27.4, which can hide outliers. The IQR method in outlier detection is more robust, and z-scores express each value's distance from the mean in standard deviations.

SQL Implementation

Most SQL databases have built-in functions: STDDEV_SAMP for a sample and STDDEV_POP for a population in PostgreSQL, MySQL, BigQuery, Snowflake and Oracle. Plain STDDEV is the sample version in PostgreSQL, BigQuery and Snowflake but the population version in MySQL, and SQL Server uses STDEV (sample) and STDEVP (population) instead.

-- Calculate standard deviation and variance
SELECT
    STDDEV_POP(salary) as pop_std_dev,    -- Population
    STDDEV_SAMP(salary) as sample_std_dev, -- Sample
    VAR_POP(salary) as variance
FROM employees;

Excel Formulas

Use STDEV.S when your data is a sample and STDEV.P when it covers the whole population. These functions were added in Excel 2010; the older STDEV (sample) and STDEVP (population) still work for compatibility.

=STDEV.P(A1:A100)  -- Population standard deviation
=STDEV.S(A1:A100)  -- Sample standard deviation (most common)
=VAR.P(A1:A100)    -- Population variance
=VAR.S(A1:A100)    -- Sample variance

Relationship: Variance vs Standard Deviation

Variance is the average squared deviation from the mean and standard deviation is its square root, so both measure spread. Standard deviation is easier to interpret because it is in the original units: for the five-value example, the variance is 2.96 in squared units and the standard deviation is 1.72 in the data's own units.

• Variance: Average of squared differences (σ²)
• Standard Deviation: Square root of variance (σ)
• Standard deviation is in the same units as your data (easier to interpret)

# Both measure spread, but std dev is more interpretable
data = [4, 8, 6, 5, 3]
variance = np.var(data)       # 2.96 (squared units)
std_dev = np.std(data)        # 1.72 (original units)
std_dev = np.sqrt(variance)   # Same thing

What is the difference between standard deviation and standard error?

Standard deviation describes how spread out individual values are; standard error describes how precisely you have estimated the mean, and equals the standard deviation divided by √n. For the five values 4, 8, 6, 5, 3, the sample standard deviation is 1.92 and the standard error of the mean is 1.92 / √5 = 0.86. Use standard deviation to describe variability in the data, and standard error or a confidence interval to describe uncertainty in an average.

Using Standard Deviation

Standard deviation underpins most everyday statistics, from flagging unusual values to building confidence intervals and sizing A/B tests.

• Comparing variability: Which dataset is more consistent?
• Outlier detection: Flag values more than 2 or 3 standard deviations from the mean
• Confidence intervals: Mean ± 1.96 × (std / √n) is an approximate 95% CI for the mean of a large sample
• Quality control: Monitor if process stays within acceptable range
• Risk assessment: Higher std dev = higher uncertainty