How to check A/B test significance
- Enter the number of visitors and conversions for the control (A) and the variant (B), from your testing tool or analytics.
- Choose the confidence level, 95% being standard, and whether you are testing for any difference (two-sided) or only for B beating A (one-sided).
- Read the verdict, the conversion rates, the relative uplift, the p-value and the confidence interval for the difference.
How the test works
This calculator uses the standard two-proportion z-test. It pools both groups to estimate the conversion rate if there were no real difference, works out how much the rates would vary by chance with your sample sizes, and measures how many standard errors apart A and B are (the z-score). The p-value is the probability of seeing a difference at least this large if the variants were really identical. Below 0.05, the result is called significant at 95% confidence. For 200 conversions from 10,000 visitors against 250 from 10,000, z ≈ 2.38 and p ≈ 0.017, so B's 25% uplift is significant.
The confidence interval for the difference is often more useful than the yes-or-no verdict: it shows the plausible range of the true effect, which may be much smaller than the observed uplift. Common mistakes are stopping a test the moment it turns significant (peeking inflates false positives), running tests too short to cover weekly cycles, and testing many variants or metrics at once without adjustment. Decide your sample size in advance with our sample size calculator, then run the test to the end. Everything is calculated in your browser.
What does statistically significant mean?
That a difference this large would be unlikely (for example less than 5% likely) if A and B truly performed the same.
Should I use a one-sided or two-sided test?
Two-sided is the safer default. Use one-sided only if you decided in advance that you only care whether B is better.
How long should I run an A/B test?
Until you reach the sample size you planned, and for at least one or two full weeks to cover weekday and weekend behaviour.