A/B testing

Statistical Significance Calculator

Two numbers look different — this decides whether the difference is more than sampling noise, and how large it plausibly is.

Updated August 2026Runs in your browser — nothing is uploadedVerified against R

How to read this result

Significant means unlikely under the null hypothesis — nothing more. It does not mean large, important, or reliable. With enough data, a difference of one hundredth of a percentage point is significant; with too little, a difference worth acting on is not.

The confidence interval is the number to act on. It gives the range of true differences consistent with the data. Decide in advance what improvement would justify the change, then check whether the interval excludes that, not merely zero.

Confidence and p-value are two views of one number. 95% confidence corresponds to p = 0.05. Reporting "97% confident" is the same statement as p = 0.03, and neither is the probability that the result is real.

A non-significant result is not evidence of no difference. It usually means the sample was too small. The power figure reported above says how likely this test was to detect the difference it observed — if that is 30%, "no difference" is not the conclusion the data support.

Left curve: what the statistic does when nothing is going on. Right curve: what it does when the effect is real. α is the sliver of the left curve past the threshold — a false positive. β is the part of the right curve that falls short — a miss. Power is everything to the right of the line under the second curve.

The formula

Two rates

z = \frac{{\hat{p}}_{B} - {\hat{p}}_{A}}{\sqrt{\bar{p} (1 - \bar{p}) (\frac{1}{n_{A}} + \frac{1}{n_{B}})}}

$\bar{p}$: the pooled rate across both groups

The interval on the difference uses the unpooled standard error, which is the standard convention.

Two means (Welch)

t = \frac{{\bar{x}}_{B} - {\bar{x}}_{A}}{\sqrt{\frac{s_{A^{2}}}{n_{A}} + \frac{s_{B^{2}}}{n_{B}}}}

With the Welch-Satterthwaite degrees of freedom, so unequal variances and unequal group sizes are both handled.

Worked example

Two landing pages, five thousand visitors each

Page A converts 200 of 5,000 visitors (4.0%). Page B converts 250 of 5,000 (5.0%). One percentage point — is that real?

Compute both rates.
p̂_A = 0.040, p̂_B = 0.050
Pool them under the null hypothesis.
p̄ = 450 / 10,000 = 0.045
The pooled standard error of the difference.
√(0.045 × 0.955 × (1/5000 + 1/5000)) = 0.0041465
Divide the difference by it.
z = 0.010 / 0.0041465 = 2.412
Two-tailed p-value.
p = 0.0159
The 95% interval on the difference, using the unpooled standard error.
0.19 to 1.81 percentage points

p-value 0.0159confidence 98.4%difference +1.00 pprelative lift +25%95% CI 0.19 to 1.81 pp

Significant, and the interval runs from 0.19 to 1.81 percentage points — anywhere from a 5% to a 45% relative lift. Ten thousand visitors establishes the direction and leaves the magnitude wide open.

Checked against R's prop.test(c(250, 200), c(5000, 5000), correct = FALSE).

The calculator above is loaded with these numbers by the Load the worked example button.

Assumptions, and when to use something else

The sample size was fixed in advance and you looked once. Stopping as soon as the result turns significant inflates the false-positive rate from 5% to well over 20%. This is the assumption that is broken most often and matters most.
Independent observations, and independent groups. One visitor, one outcome, assigned once.
A single comparison. Testing four variants against a control quadruples the chances of a false positive unless α is adjusted.
Enough events. For rates, at least about 10 conversions and 10 non-conversions in each group.

You are running a conversion experimentA/B test calculatorThe same test with lift, minimum detectable effect and the sequential-testing caveat spelled out.
You have raw data rather than summariesT-testPaste the two columns and get the same answer plus the diagnostics.
The counts are smallFisher's exact testExact, with no minimum-count requirement.
The result was not significantPower calculatorDistinguishes “no effect” from “not enough data”.

Questions people ask

What does 95% statistical significance mean?

That a difference at least this large would occur less than 5% of the time if the two groups were truly identical. It is not a 95% probability that the difference is real, and it says nothing about how large the difference is.

Is a p-value of 0.06 meaningless?

No. It is slightly weaker evidence than 0.04 and nearly identical evidence in practical terms. The 0.05 line is a convention; treating 0.049 and 0.051 as different kinds of result is the most common misuse of the whole framework.

How many visitors do I need for significance?

It depends on the baseline rate and the improvement you want to detect. Detecting a move from 4% to 5% with 80% power takes about 6,900 per group; detecting 4% to 6% takes about 1,900. Work it out before you start with the sample size calculator.

Can I stop the test as soon as it becomes significant?

No — and this is the single most expensive mistake in experimentation. Every look is another chance to cross the threshold by luck, so peeking daily can push the real false-positive rate above 20%. Fix the sample size in advance, or use a sequential design built for continuous monitoring.