A/B testing

A/B Test Calculator

Paste the visitors and conversions from both variants and get the verdict — plus the range the true lift plausibly falls in.

Also called conversion rate significance calculator, A/B test significance calculator, split test calculator.

Updated August 2026Runs in your browser — nothing is uploadedVerified against R

How to read this result

Read the interval, not the verdict. "Significant" tells you the direction is probably real. The confidence interval on the lift tells you how big it plausibly is, and that is what a shipping decision depends on. A significant lift whose interval runs from +0.2% to +45% is a result that justifies more data, not a launch.

Absolute and relative lift are different claims. Four per cent to five per cent is one percentage point and a 25% relative improvement. Both are reported; quoting only the relative one without the baseline is how tiny changes get sold as breakthroughs.

"Not significant" often means "not enough traffic". The minimum detectable effect above says what this test could have found at 80% power. If it is 20% and your realistic gain is 3%, the test was never capable of answering the question.

Peeking invalidates all of it. These p-values assume you fixed the sample size in advance and looked once. Checking daily and stopping when it goes green raises the false-positive rate from 5% to above 20% — the single most expensive habit in conversion testing.

The formula

z = \frac{{\hat{p}}_{B} - {\hat{p}}_{A}}{\sqrt{\bar{p} (1 - \bar{p}) (\frac{1}{n_{A}} + \frac{1}{n_{B}})}} lift = \frac{{\hat{p}}_{B} - {\hat{p}}_{A}}{{\hat{p}}_{A}}

A two-proportion z-test. The interval on the difference uses the unpooled standard error and is then divided by the baseline to give the relative-lift interval.

Worked example

A 4% control against a 5% variant

Control: 200 conversions from 5,000 visitors. Variant: 250 from 5,000. A one-point improvement, a 25% relative lift, on ten thousand visitors in total.

The two rates.
4.00% and 5.00%
Pooled rate under the null hypothesis.
450 / 10,000 = 4.50%
Pooled standard error.
√(0.045 × 0.955 × 2/5000) = 0.0041465
z, and the two-tailed p-value.
z = 2.412, p = 0.0159
The 95% interval on the absolute difference, unpooled.
0.19 to 1.81 percentage points
Which as a relative lift is:
roughly +4.7% to +45%

p-value 0.0159confidence 98.4%absolute lift +1.00 pprelative lift +25.0%95% CI (absolute) 0.19 to 1.81 pp

The headline is “+25% lift, significant”. The honest summary is “somewhere between +5% and +45%, probably positive”. Both come from the same arithmetic, and only the second supports a forecast.

Checked against R's prop.test(c(250, 200), c(5000, 5000), correct = FALSE).

The calculator above is loaded with these numbers by the Load the worked example button.

Assumptions, and when to use something else

Random assignment, and each visitor counted once. Assigning by session rather than by visitor breaks independence and shrinks the true standard error.
A fixed sample size, analysed once. See the peeking warning above.
One variant against one control. With several variants the false-positive rate multiplies; divide α by the number of comparisons or run them sequentially.
A stable environment. Run for whole weeks so that day-of-week effects average out, and avoid overlapping the test with a marketing campaign that changes the traffic mix.

You have not started the testSample size calculatorDecide how much traffic you need before you begin — this is the step that makes the rest valid.
The result was not significantMinimum detectable effectFind out whether the test could ever have detected the lift you hoped for.
The metric is revenue per visitor, not a rateT-testContinuous metrics need a t-test — and revenue is usually skewed enough to warrant checking the assumptions.
The counts are smallFisher's exact testExact rather than approximate, with no minimum count.

Questions people ask

How long should I run an A/B test?

Until you reach the sample size you calculated in advance, and no fewer than one or two whole weeks so that day-of-week effects average out. Stopping early because it looks good is the mistake that makes most reported wins unreproducible.

What does 95% confidence mean here?

That if the two variants were truly identical, a difference this large would appear less than 5% of the time. Run twenty tests on identical pages and you should expect about one “winner”.

Can I test three variants at once?

Yes, but each additional comparison adds a chance of a false positive. With three variants against a control at α = 0.05, the chance of at least one spurious winner is about 14%. Divide α by the number of comparisons, or accept the higher risk knowingly.

My test is significant but the lift is tiny. Should I ship?

That is a business question, not a statistical one — which is the point. Check whether the bottom of the confidence interval is still an improvement worth the engineering cost. Significance only rules out zero.