A/B Test Significance Calculator (Chi-Square)
Visitors and conversions of variants A and B in a 2x2 table return the statistic; past 3.84 with 1 degree of freedom the gap is significant at p 0.05.
χ² + p<0.05
—
A/B test significance with chi-square
Lay out your data as a 2×2 table (variant A/B against converted/not converted) and Pearson's chi-square asks one question: do the conversion rates actually differ? The statistic is χ² = Σ (O − E)² / E with 1 degree of freedom, and the expected counts come from E = (row total × column total) / grand total. You reject H₀ (no difference) once p < 0.05, which lines up with χ² > 3.84. Take A = 80/1000 (8%) versus B = 110/1000 (11%): that works out to χ² ≈ 5.0 and p ≈ 0.025, so the lift is real. Work out your sample size ahead of time with Casagrande-Pike so the test isn't underpowered. And watch the temptation to peek at running totals — that inflates type I error. Lock in your n beforehand, or switch to a Bayesian A/B setup with credible intervals.
Applications
Conversion-rate optimization (CRO), UI tweaks like button color or layout, tests on email copy and subject lines, pricing-page experiments, paid-ads creative splits, and product calls made on data in platforms such as Optimizely and VWO (Google Optimize shut down in 2023).
FAQ
Chi-square or two-proportion z-test? On a 2×2 table they're the same thing, mathematically: z² = χ². Go with whatever your stack already reports.
What if expected counts are below 5? Reach for Fisher's exact test. The chi-square approximation gets unreliable on small samples.
Can I stop the test early once it hits significance? Don't. Naïve peeking pushes your false-positive rate well past 5%. If you need to look as data arrives, use sequential methods (mSPRT, group-sequential bounds) or Bayesian decision rules.
Related Tools
CI for Variance and Standard Deviation
Computes the confidence interval for the variance and standard deviation of a normal population, from a sample. While most tools estimate the mean, this one estimates the spread — useful in quality control, where consistency matters as much as the central value. The calculation uses the chi-square distribution, which is asymmetric, so the interval is not centered on the sample variance. Enter the sample and the confidence level.
One-Sample Variance Test (Chi-Square)
Computes the chi-square test for the variance of a single sample, which checks whether the population variance equals a reference value. It's the dispersion counterpart of the one-sample t-test for the mean. It shows up often in quality control: is a process's variability within the specified limit, or has it increased? The statistic compares the sample variance with the hypothesized one and follows a chi-square distribution. Enter the sample and the hypothesized variance (σ₀²).
Chi Square Frequency Test Calculator
Computes chi square goodness of fit statistic from observed and expected frequencies of each category for a hypothesis test.
McNemar's Test (Paired)
Computes McNemar's test, used to compare two paired proportions — when the same subjects are measured twice (before and after, or by two methods). Unlike the two-proportion test, which treats the groups as independent, McNemar looks only at the cases that switched classification (the discordant cells b and c of the 2×2 table), ignoring those that stayed the same. The continuity-corrected version follows a chi-square with 1 degree of freedom. Enter the two discordant counts.
Chi-Square Goodness-of-Fit Calculator
Calculate the chi-square (χ²) goodness-of-fit statistic from observed and expected frequencies. Test whether your data follows the expected distribution.
Mood's Median Test
Computes Mood's median test, a non-parametric test that checks whether several groups share the same median. The idea is simple and robust: you compute the grand median of all the data together and count, in each group, how many values fall above it. If the groups had the same median, those counts would be proportional to the sizes; an imbalance produces a large chi-square statistic. It's quite resistant to outliers, working only with counts. Enter the groups separated by semicolons.
The results provided by this tool are for general informational and educational purposes only and do not constitute professional, financial, medical, legal, tax or accounting advice. Always confirm important decisions with a qualified professional and official sources.