Free A/B Test Significance Calculator

Statistical significance tells you how likely it is that the difference between your control and variant results is due to the change you made rather than random chance. An A/B test result is considered statistically significant when the probability of getting the observed difference by chance alone falls below a threshold — typically 5% (95% confidence) or 1% (99% confidence).

Without statistical significance, you cannot reliably distinguish a real improvement from natural variation in your traffic. For example, if your variant converts at 5.8% and your control at 5.0%, that could be a genuine 16% lift — or it could be noise. The significance calculator tells you which it is, based on the volume of data you have collected.

Running underpowered tests (stopping too early or with too few visitors) is one of the most common A/B testing mistakes. The more visitors and conversions you collect, the smaller the difference you can detect reliably. This calculator uses the two-proportion z-test, which is the standard method for comparing conversion rates between two groups.

Enter the number of visitors and conversions for your control (A) and your variant (B). A conversion can be any event you are measuring — a purchase, a sign-up, a click, a form submission. The calculator works for any binary outcome.

The results update automatically as you type. You will see the conversion rate for each variant, the percentage uplift (how much better or worse the variant is relative to the control), the z-score, the p-value, and whether the result is significant at 95% or 99% confidence.

If the result is not yet significant, keep collecting data. The p-value will decrease as your sample size grows, assuming the true difference remains consistent.

The p-value is the probability of observing a difference at least as large as the one you measured, assuming there is actually no difference between the control and variant. A p-value of 0.03 means there is a 3% chance the observed difference is due to chance alone. Most A/B tests use a significance threshold of 0.05 (5%).

The z-score measures how many standard deviations the observed difference is from zero. For a two-tailed test at 95% confidence, a z-score with an absolute value above 1.96 indicates significance. At 99% confidence, the threshold is 2.576. A higher absolute z-score means the result is more extreme relative to what you would expect by chance.

Confidence level is the inverse of the significance threshold. A 95% confidence level means you are willing to accept a 5% chance of a false positive (calling a winner when there is none). 99% confidence means you accept only a 1% chance. Higher confidence requires more data to reach but gives you greater certainty in the result.

The most common mistake in A/B testing is stopping too early. If you check results daily and stop as soon as you see significance, you inflate your false positive rate significantly. This is called peeking. The correct approach is to set a minimum sample size before the test starts and run until you reach it, regardless of interim results.

To calculate the required sample size, use a power analysis based on your current baseline conversion rate, the minimum effect size you care about (the smallest improvement worth shipping), and your desired confidence level. As a rough guide, detecting a 10% relative lift on a 5% baseline conversion rate requires approximately 10,000 visitors per variant at 80% statistical power.

Once your test has run for at least one full business cycle (typically one to two weeks to capture day-of-week variation) and you have reached your minimum sample size, you can declare a winner if the result is significant, or a null result if it is not. Null results are valuable — they tell you the variant did not make a meaningful difference and you should try a bigger change.