A/B Test Statistical Significance Calculator

This calculator uses a two-proportion z-test, which is the standard for comparing conversion rates in A/B tests. It computes a z-score from the pooled proportion standard error, then derives a two-tailed p-value using the normal distribution. This is equivalent to Pearson's chi-square test for 2×2 contingency tables.

The p-value is the probability of observing a difference this large (or larger) purely by chance, assuming the null hypothesis is true (i.e., both variants have the same conversion rate). A p-value below 0.05 at 95% confidence means there is less than a 5% probability this result is random.

A result is statistically significant when the p-value is below your chosen alpha threshold (0.05 for 95% confidence). It means the observed difference is unlikely to be due to random sampling variation. Without statistical significance, you cannot reliably conclude that one variant is actually better.

Stopping a test as soon as significance is reached inflates your false-positive rate dramatically — a phenomenon called 'peeking'. Even if your results look significant after 20% of the planned sample, the actual false-positive rate can be 20-30% instead of 5%. Always predetermine your sample size and run to completion.

MDE is the smallest relative improvement you care about detecting. A 20% MDE on a 5% baseline means you want to reliably detect a lift from 5% to 6%. Smaller MDEs require larger sample sizes. Setting too ambitious an MDE (e.g. 5%) requires enormous traffic and long test durations.

The confidence interval shows the range of plausible values for the true difference (B minus A) in conversion rates. A 95% CI of [+0.5%, +2.1%] means you are 95% confident the true improvement is between 0.5% and 2.1% percentage points. If the interval contains 0, the test is inconclusive.