Is the difference between two variants real, or just noise? A two-proportion z-test with p-value and confidence.
A two-proportion z-test compares the conversion rates of two variants. Here p₁ and p₂ are each variant's conversion rate, n₁ and n₂ are the visitor counts, and p is the pooled conversion rate across both groups. The z-score measures how many standard errors apart the two rates sit; the p-value is two-tailed, so it captures a difference in either direction. A p-value below 0.05 clears the 95% confidence bar.
Variant A converts 100 of 2,000 visitors (5.0%); variant B converts 130 of 2,000 (6.5%). The pooled rate is 5.75%, the standard error is about 0.0074, and z ≈ 2.03. That gives a two-tailed p-value of about 0.042 — roughly 95.8% confidence — so the 1.5-point uplift is statistically significant at the 95% level.
A significant result only means the difference is probably not random — it says nothing about whether the difference matters. With huge samples a trivial uplift can be significant, so always read the effect size alongside the p-value. Just as important, run each test to a sample size you set in advance rather than stopping the moment it crosses the line: peeking repeatedly inflates the false-positive rate. By convention 95% confidence means p < 0.05, but that threshold is a shared habit, not a hard rule — decide what evidence a real decision needs.
What does statistical significance mean in an A/B test? That the observed gap between variants is unlikely under the assumption they perform identically — at 95% confidence, less than a 5% chance of a gap this large by fluke.
What is a good p-value? Below 0.05 for 95% confidence. Lower means stronger evidence, but 0.05 is a convention you can tighten for higher-stakes calls.
Why shouldn't I stop a test early? Peeking and stopping on the first significant reading inflates the false-positive rate above 5%. Set a sample size up front and run to it.
