A/B testing
Minimum Detectable Effect Calculator
The smallest improvement your test could actually find — the number that says whether it is worth running.
How to read this result
The MDE is the power calculation run backwards. Instead of asking how many observations you need for a given effect, it asks what effect you could detect with the observations you have.
Compute it before the test, and quote it after. Before: it tells you whether the experiment is worth running at all. After: it converts "not significant" into the far more useful "this test could not have detected anything smaller than 12%".
It is not a threshold for the result. A test can detect an effect smaller than its MDE — just less than 80% of the time. And an observed lift below the MDE is not evidence of no effect.
Traffic buys sensitivity slowly. Four times the sample halves the detectable effect. If your MDE is 20% and you need 5%, you need sixteen times the traffic, which is usually the point at which the experiment is redesigned rather than extended.
The formula
- 1.96 at the 5% level, two-tailed
- 0.84 for 80% power
- the sample size per group
An approximation for intuition; the calculator solves the exact power equation numerically, which is why its answer differs slightly from this formula.
Worked example
What 1,774 visitors per variant can see
Your baseline conversion rate is 10% and you can get 1,774 visitors to each variant. What is the smallest improvement you could reliably detect at 80% power?
- Set the target power and significance.80% power, α = 0.05 two-tailed
- Solve the power equation for the second rate.the rate detectable at 80% power is 13.0%
- Express it as an absolute difference.3.0 percentage points
- And as a relative lift.30% relative
Note the symmetry with the sample size example: 1,774 per group is exactly what detecting 10% → 13% requires. The two calculators solve one equation from opposite ends, and they agree by construction.
Checked against the inverse of R's power.prop.test, which gives n = 1,774 for exactly this detectable difference.
The calculator above is loaded with these numbers by the Load the worked example button.
Assumptions, and when to use something else
The MDE assumes the same design as the power calculation it inverts: fixed sample size, one analysis, independent observations, equal groups. It also assumes a baseline rate — if the true baseline drifts during the test, the achieved sensitivity drifts with it.
For a mean, the calculation needs the standard deviation, and the MDE scales directly with it. An optimistic SD produces an optimistic MDE.
- You can choose the sample sizeSample size calculatorSolve for n instead, from the effect you want to detect.
- You want the power of a specific effectPower calculatorThe probability of detecting an effect you name.
- The test has finishedA/B test calculatorRead the result, with the MDE reported alongside it.
Questions people ask
What is a minimum detectable effect?
The smallest true effect that a study with your sample size would detect at your chosen power — usually 80%. Smaller effects can still turn up significant, just less than 80% of the time.
Should I compute the MDE before or after the test?
Both. Before, it tells you whether the experiment can answer your question. After, it explains a non-significant result: “we could only have detected a 30% lift, and we hoped for 5%” is a very different statement from “there is no effect”.
Why is my MDE so large?
Because the sample is small, the baseline rate is low, or both. Conversion tests on low-traffic pages routinely have MDEs above 50%, which means they can only detect changes nobody expects.
Can I lower the MDE without more traffic?
A little. Accepting lower power or a higher α lowers it at the cost of more errors. Reducing variance helps more: a lower-variance metric, a paired or within-subject design, or removing an obvious source of noise all buy sensitivity that traffic would otherwise have to pay for.