Significância A/B Test (Qui-quadrado)

A/B test significance with chi-square

Lay out your data as a 2×2 table (variant A/B against converted/not converted) and Pearson's chi-square asks one question: do the conversion rates actually differ? The statistic is χ² = Σ (O − E)² / E with 1 degree of freedom, and the expected counts come from E = (row total × column total) / grand total. You reject H₀ (no difference) once p < 0.05, which lines up with χ² > 3.84. Take A = 80/1000 (8%) versus B = 110/1000 (11%): that works out to χ² ≈ 5.0 and p ≈ 0.025, so the lift is real. Work out your sample size ahead of time with Casagrande-Pike so the test isn't underpowered. And watch the temptation to peek at running totals — that inflates type I error. Lock in your n beforehand, or switch to a Bayesian A/B setup with credible intervals.

Applications

Conversion-rate optimization (CRO), UI tweaks like button color or layout, tests on email copy and subject lines, pricing-page experiments, paid-ads creative splits, and product calls made on data in platforms such as Optimizely and VWO (Google Optimize shut down in 2023).

FAQ

Chi-square or two-proportion z-test? On a 2×2 table they're the same thing, mathematically: z² = χ². Go with whatever your stack already reports.

What if expected counts are below 5? Reach for Fisher's exact test. The chi-square approximation gets unreliable on small samples.

Can I stop the test early once it hits significance? Don't. Naïve peeking pushes your false-positive rate well past 5%. If you need to look as data arrives, use sequential methods (mSPRT, group-sequential bounds) or Bayesian decision rules.