Category
A/B Testing and Experiment Design Calculators
Decide the sample size before you start, read the result after you stop, and know which differences your test was ever capable of detecting.
The order these tools should be used in
Before the test: decide what improvement would be worth shipping, then use the sample size calculator to find how many visitors each variant needs to detect it with 80% power. If that number is out of reach, the minimum detectable effect tells you what you can detect with the traffic you have — and if that is 20% when your realistic gain is 3%, the test is not worth running.
After the test: the A/B test calculator or the statistical significance calculator gives you the p-value, the confidence interval on the difference and the observed lift. Read the interval, not just the p-value: it tells you how large the improvement plausibly is, which is the number the decision actually depends on.
The mistake that costs the most
Peeking. Checking the dashboard every morning and stopping the moment the result turns significant sounds harmless and is not: it inflates the false-positive rate from 5% to somewhere above 20%, because every look is another chance to cross the line by luck. A fixed-horizon test — which is what these calculators assume — is valid only if you set the sample size in advance and look once at the end. If you must monitor continuously, you need a sequential design with alpha spending, or a Bayesian stopping rule, not this arithmetic.
Significance is not importance
A test with enough traffic will find a difference of 0.01 percentage points and label it significant. Whether that difference is worth a deployment is a business question, and the effect size and the confidence interval are how you answer it. The useful habit is to write down, before the test, the smallest lift you would act on — then check whether the interval excludes it, not merely whether it excludes zero.
Ratios of risk
For a two-by-two outcome — exposed or not, converted or not — the odds ratio and relative risk are the two standard summaries, and they are not interchangeable. When the outcome is common, the odds ratio sits further from 1 than the risk ratio does, and reporting one as the other systematically overstates the effect.
The 8 calculators in this category
- Sample Size CalculatorWork out how many observations you need: for a survey margin of error, for estimating a mean, or for detecting a difference between two rates with a given power.
- Statistical Significance CalculatorTest two conversion rates or two means for statistical significance: p-value, confidence level, the interval on the difference, effect size and the power the test actually had.
- Statistical Power CalculatorCompute statistical power for a t-test or a two-proportion test using the non-central t distribution, with the power curve, the sample size needed for 80% and the two distributions drawn.
- A/B Test CalculatorRead a conversion experiment: p-value, confidence level, absolute and relative lift with confidence intervals, the power the test had and the smallest lift it could have detected.
- Effect Size CalculatorCompute Cohen's d, Hedges' g and Glass's delta from raw data, summary statistics or a t value, with the confidence interval, the common-language effect size and the overlap.
- Odds Ratio CalculatorCompute the odds ratio from a 2×2 table with its confidence interval, p-value, the relative risk and the risk difference, plus the Haldane correction for zero cells.
- Relative Risk CalculatorCompute relative risk from a 2×2 table with its confidence interval, the absolute risk difference, the number needed to treat and the odds ratio for comparison.
- Minimum Detectable Effect CalculatorFind the smallest effect your sample size can reliably detect, for a conversion rate or a mean, with the curve showing how the MDE falls as the sample grows.
Questions people ask
How long should I run an A/B test?
Until you reach the sample size you calculated in advance, and no less than a full business cycle — usually one or two whole weeks — so that day-of-week effects average out. Stopping early because the result looks good is the single most expensive habit in conversion testing.
What does 95% confidence mean in an A/B test?
That the procedure produces an interval covering the true difference 95% of the time, and that a result labelled significant at α = 0.05 would occur by chance in 5% of tests where nothing is really different. Run twenty tests on identical pages and you should expect roughly one 'winner'.
Can I test more than two variants at once?
Yes, but the false-positive rate multiplies with the number of comparisons. Either divide α by the number of variants (the Bonferroni correction) or accept that with four variants at α = 0.05 the chance of at least one spurious winner is about 14%.