A/B Test Sample Size Calculator

Without a pre-determined sample size, you risk stopping the test too early (when results look significant by chance) or too late (wasting traffic). Pre-calculating sample size ensures your test has enough statistical power to reliably detect your target improvement and control your false-positive rate.

Statistical power (1-β) is the probability of detecting a real effect when one truly exists. An 80% power means you have an 80% chance of seeing a significant result if your variant really is better by the minimum detectable effect (MDE). Higher power requires larger samples. Most A/B tests use 80% or 90% power.

MDE is the smallest relative improvement you care about detecting. If your baseline conversion rate is 5% and you want to detect a 20% relative improvement (to 6%), that's a 20% MDE. Smaller MDEs require much larger sample sizes — a 5% MDE might need 10x more traffic than a 20% MDE.

The standard is α = 0.05 (95% confidence), meaning a 5% false-positive rate. For high-stakes decisions (pricing, checkout flow), use α = 0.01. For exploratory tests where you'll validate winners anyway, α = 0.10 is acceptable. Never use α > 0.10 for production decisions.

Consider increasing MDE (accept only larger improvements), reducing variants (stick to A/B not A/B/C), or running the test longer. If your monthly traffic can't reach the sample within a reasonable timeframe (under 4-8 weeks), the test may not be feasible. Focus on higher-traffic pages or use Bayesian testing methods.

Test duration = required sample size per variant / (daily traffic × allocation percentage). If you need 5,000 visitors per variant with 1,000 daily visitors split 50/50, it takes 5,000 / 500 = 10 days. Add 10-20% buffer for weekday/weekend traffic variation.