Statistical Significance Calculator

Test whether your A/B test results are statistically significant. Get p-values, confidence intervals, relative lift, and power analysis using a two-proportion z-test.

> Last verified: March 2026 - All steps tested on Chrome 134 (latest stable). Extension data verified against Chrome Web Store.

What Is Statistical Significance

Statistical significance answers a specific question. If there were truly no difference between my control and variant, how likely would I be to see a result at least as extreme as what I observed? If that probability is very low (typically below 5%), we call the result statistically significant.

The concept comes from hypothesis testing, developed by Ronald Fisher and later formalized by Jerzy Neyman and Egon Pearson in the early 20th century. In an A/B testing context, the null hypothesis states that the conversion rates of the control and variant are identical. The alternative hypothesis states they are different.

When you run an A/B test, you are collecting sample data to make an inference about the true underlying conversion rates. Because samples involve randomness, you will almost always see some difference between groups even when no real difference exists. Statistical significance quantifies whether the observed difference is large enough relative to the sample size to rule out random chance as a plausible explanation.

I want to emphasize that "significant" in statistics does not mean "large" or "important." A tiny 0.1 percentage point difference can be statistically significant with a large enough sample, while a 5 percentage point difference might not be significant with a small sample. The word refers purely to the reliability of the finding, not its magnitude.

P-Values Explained

The p-value is the probability of observing a test result at least as extreme as the one calculated, assuming the null hypothesis is true. A p-value of 0.03 means there is a 3% chance of seeing a difference this large (or larger) if the control and variant truly had identical conversion rates.

Common p-value thresholds and what they mean in practice:

P-Value	Confidence	Interpretation
p < 0.01	99%+	Very strong evidence against the null hypothesis
p < 0.05	95%+	Standard threshold for declaring significance
p < 0.10	90%+	Weak evidence, sometimes used in exploratory testing
p > 0.10	<90%	Insufficient evidence to reject the null hypothesis

A critical misconception is that the p-value tells you the probability that the variant is better. It does not. The p-value is the probability of the data given no difference, not the probability of no difference given the data. These are fundamentally different questions, and confusing them leads to flawed decision-making.

Another misconception is that a p-value of 0.05 means there is a 95% chance the variant is better. Again, incorrect. The p-value is a statement about the data under the null hypothesis, not a statement about which version is truly superior. Bayesian methods provide the kind of probability statements most people actually want, but they require specifying a prior belief about the expected effect size.

How This A/B Test Calculator Works

This calculator performs a two-proportion z-test, which is the standard method for comparing two conversion rates. Here is the math behind the results.

First, the calculator computes the conversion rate for each group. Control rate = control conversions divided by control visitors. Variant rate = variant conversions divided by variant visitors.

Next, it calculates the pooled proportion: the total conversions across both groups divided by the total visitors across both groups. This pooled proportion is used under the null hypothesis assumption that both groups have the same true rate.

The standard error of the difference is computed as the square root of the pooled proportion times (1 minus pooled proportion) times the sum of (1 divided by control visitors) plus (1 divided by variant visitors).

The z-score equals the absolute difference in conversion rates divided by the standard error. This z-score is then converted to a p-value using the standard normal distribution. For a two-tailed test, the p-value is 2 times the probability of observing a z-score this extreme or more extreme.

The confidence interval for the difference is calculated using the unpooled standard error (since the confidence interval does not assume equal proportions). The interval is the observed difference plus or minus the critical z-value times the unpooled standard error.

Confidence Intervals and What They Mean

A confidence interval provides a range of plausible values for the true difference between conversion rates. A 95% confidence interval means that if you repeated the experiment many times, about 95% of the intervals would contain the true difference.

For A/B testing, the confidence interval is more informative than the p-value alone because it tells you both the direction and the likely magnitude of the effect. A confidence interval of [0.5%, 2.3%] says that the variant likely improves conversion by somewhere between 0.5 and 2.3 percentage points. A p-value alone tells you only that the difference is significant, not how big it is.

When the confidence interval for the difference does not include zero, the result is statistically significant at the corresponding confidence level. When it does include zero, the data is consistent with no real difference.

The width of the confidence interval depends on sample size. Larger samples produce narrower intervals, giving you a more precise estimate of the true effect. If your confidence interval is very wide (for example, -1% to +4%), your sample is probably too small to draw dependable conclusions, even if the p-value happens to be below 0.05.

Type I and Type II Errors

Every statistical test involves a tradeoff between two types of errors.

A Type I error (false positive) occurs when you conclude the variant is different when it is actually not. You ship a change that does not really improve anything, possibly wasting development resources and confusing future analyses. The probability of a Type I error equals your significance level (alpha). At alpha = 0.05, you will declare a false positive about 5% of the time when the null hypothesis is true.

A Type II error (false negative) occurs when you fail to detect a real difference. The variant actually is better, but your test did not have enough data to show it. You miss an improvement opportunity. The probability of a Type II error is called beta, and it equals 1 minus your statistical power. At 80% power, you have a 20% chance of missing a real effect.

The relationship between these errors is important. You cannot reduce both simultaneously without increasing your sample size. Making your significance threshold more strict (say, moving from alpha = 0.05 to alpha = 0.01) reduces false positives but increases false negatives for the same sample size. The only way to reduce both is to collect more data.

In business contexts, consider the cost of each error type. If implementing the variant is cheap and reversible, you might accept a higher false positive rate (alpha = 0.10) because the cost of wrongly adopting it is low. If the variant requires significant engineering investment, you want a lower alpha to avoid wasted effort.

Sample Size Planning

Running an A/B test without a sample size calculation is like leaving on a road trip without knowing the distance. You might arrive on time, or you might stop far too early or drive much longer than necessary.

Sample size depends on four inputs. First, the baseline conversion rate of your control. Higher baseline rates require smaller samples because the signal-to-noise ratio is better. Second, the minimum detectable effect (MDE), the smallest improvement you want to be able to detect. Smaller effects require much larger samples. Third, statistical power, typically set at 80%. Higher power means larger samples. Fourth, significance level, typically 0.05.

Here are some sample size benchmarks for common scenarios at 80% power and 95% confidence:

Baseline Rate	10% Relative Lift	20% Relative Lift	50% Relative Lift
2%	195,000 per group	49,000 per group	8,000 per group
5%	73,000 per group	18,500 per group	3,100 per group
10%	33,000 per group	8,500 per group	1,500 per group
20%	14,000 per group	3,700 per group	680 per group

These numbers explain why testing small improvements on low-traffic pages is extremely difficult. To detect a 10% relative lift on a 2% baseline conversion rate, you need nearly 400,000 total visitors. At 500 visitors per day, that test would take over two years. This is not feasible for most websites, which is why focusing on testing larger changes (higher MDE) is often more practical.

Common A/B Testing Mistakes

Peeking at Results Too Early

Checking your test results repeatedly and stopping when they look significant is called the "peeking problem." Every time you check, you increase the chance of a false positive. If you check after every 100 visitors at alpha = 0.05, your actual false positive rate can climb above 30%. The fix is to pre-determine your sample size and commit to running the test to completion, or use sequential testing methods (like SPRT or always-valid confidence intervals) that are designed for continuous monitoring.

Under-Powered Tests

Running a test with too few visitors and then concluding "no significant difference, so the variant does not work" is a Type II error waiting to happen. With low power (say 30%), you have a 70% chance of missing a real effect. Before declaring a negative result, check whether your test had adequate power. If power was below 60%, the test was too small to draw conclusions.

Multiple Comparisons

Testing many variants or metrics simultaneously without adjusting for multiple comparisons inflates your false positive rate. If you test 20 metrics at alpha = 0.05, you expect to find one "significant" result by chance alone. The Bonferroni correction (divide alpha by the number of comparisons) or the Benjamini-Hochberg procedure can address this.

Ignoring Segments

A test might show no overall significant effect but have a strong effect within a specific segment (mobile users, returning visitors, a particular geography). Post-hoc segment analysis is valid for generating hypotheses but should not be treated as confirmatory. If you find a segment-level effect, run a follow-up test targeting that segment to confirm it.

Seasonal and Day-of-Week Effects

Running a test from Monday to Thursday and comparing against a weekend period introduces confounding variables. Always run both groups simultaneously for at least one full business cycle (typically one full week minimum). Longer tests should cover multiple weeks to account for paycheck cycles, holidays, and other periodic patterns.

When to Stop an A/B Test

The simplest rule is to calculate your required sample size before starting the test and stop when you reach it. Period. Do not stop early because results look good. Do not extend because results look borderline.

If you need the flexibility to stop early, use sequential testing methods. Group sequential designs allow you to check results at pre-defined intervals (say, after 25%, 50%, 75%, and 100% of planned sample) with adjusted significance thresholds at each checkpoint. The O'Brien-Fleming spending function is the most popular approach, and it barely inflates the overall alpha while allowing early stopping for clearly significant results.

Bayesian methods offer another approach. Instead of p-values, Bayesian A/B testing computes the probability that the variant is better than the control (the posterior probability). You can monitor this probability continuously without a peeking penalty, and stop when the probability exceeds your threshold (say, 95% probability of improvement and expected loss below a business-meaningful threshold).

As a practical minimum, I recommend running any A/B test for at least 7 days to capture day-of-week effects, even if you reach statistical significance earlier. Tests shorter than one full week are vulnerable to systematic biases from traffic pattern differences across days.

Statistical vs Practical Significance

A result can be statistically significant but practically meaningless. If a test with 500,000 visitors per group shows a conversion rate change from 5.00% to 5.02%, the p-value might be 0.03 (significant), but the actual business impact of a 0.02 percentage point lift is negligible.

Practical significance considers whether the observed effect is large enough to matter. This depends on your business context. For a site doing $10 million in annual revenue, even a 0.1% conversion rate improvement might translate to $10,000 in additional revenue. For a smaller operation, the same 0.1% lift might not justify the effort of implementation.

Before running any test, define your minimum detectable effect not just in statistical terms but in business terms. Ask yourself: what is the smallest improvement that would make implementing this change worthwhile? If the answer is a 10% relative lift, design your test to detect that effect size and do not get excited about smaller lifts that happen to be statistically significant.

The confidence interval is your best tool for evaluating practical significance. If the entire confidence interval falls above your minimum business-relevant threshold, the result is both statistically and practically significant. If the interval includes values below your threshold, you have statistical significance but uncertain practical value.

Bayesian vs Frequentist A/B Testing

The calculator on this page uses the frequentist approach (p-values and confidence intervals), which is the traditional and most widely used method. But Bayesian A/B testing has gained popularity in recent years, and understanding the difference helps you choose the right approach for your situation.

Frequentist testing asks: "If there were no difference, what is the probability of seeing data this extreme?" The answer is the p-value. You commit to a sample size before the test, run until completion, and then evaluate the result against your significance threshold. The main advantage is simplicity and well-established methodology. The main drawback is the inability to monitor results continuously without inflating error rates.

Bayesian testing asks: "Given the data I observed, what is the probability that Variant B is better than Control A?" This is the question most people actually want answered. Bayesian methods compute a posterior probability of improvement, which is easy to use and can be monitored continuously without a peeking penalty.

Here is a concrete comparison. A frequentist result might say "p = 0.03, reject the null hypothesis at 95% confidence." A Bayesian result for the same data might say "There is a 97.2% probability that Variant B has a higher conversion rate than Control A, and the expected improvement is 0.8 percentage points." The Bayesian result directly answers the decision-maker's question, while the frequentist result requires more interpretation.

The drawback of Bayesian methods is that they require specifying a prior distribution, which represents your belief about the effect size before seeing data. A non-informative prior (assuming equal probability for all possible effects) minimizes the influence of the prior, but still introduces a subjective element that frequentist methods avoid. In practice, for most A/B tests with reasonable sample sizes, both approaches reach the same conclusion.

Multi-Variant Testing (A/B/C/n Testing)

When testing more than two variants simultaneously, the multiple comparisons problem becomes a serious concern. With three variants (A, B, C), you have three pairwise comparisons (A vs B, A vs C, B vs C). With five variants, you have ten comparisons. Each comparison at alpha = 0.05 has a 5% chance of a false positive, and those probabilities accumulate.

The simplest correction is the Bonferroni method: divide your significance threshold by the number of comparisons. With three comparisons and a desired overall alpha of 0.05, use 0.05/3 = 0.0167 as your per-comparison threshold. This is conservative (it reduces false positives at the cost of higher false negatives) but straightforward to apply.

The Benjamini-Hochberg procedure is less conservative. It controls the false discovery rate (FDR) rather than the family-wise error rate. Rank your p-values from smallest to largest, then compare each p-value to (rank / number of tests) times alpha. This approach allows more discoveries while still controlling the expected proportion of false positives among declared significant results.

For multi-arm bandit approaches (used by Google improve and some other platforms), the concept of statistical significance is replaced by regret minimization. The algorithm dynamically allocates more traffic to better-performing variants, reducing the opportunity cost of testing. This approach is useful when the test duration is long and you want to reduce lost conversions during the testing period.

Understanding Effect Size

Effect size measures the magnitude of the difference between groups, independent of sample size. While p-values tell you whether an effect exists, effect size tells you how large it is. Two common effect size measures for proportion tests are the absolute difference and Cohen's h.

The absolute difference is simply the variant rate minus the control rate. If your control converts at 5.0% and the variant at 6.0%, the absolute difference is 1.0 percentage point. This is the most easy to use measure and directly translates to business impact.

The relative difference (or relative lift) expresses the change as a percentage of the baseline. Using the same example, the relative lift is 1.0 / 5.0 = 20%. A 20% relative lift sounds more impressive than a 1 percentage point absolute change, which is why marketers often report relative lift. Both numbers describe the same reality, but they frame it differently.

Cohen's h is a standardized effect size for comparing proportions. It is calculated as 2 times arcsin(sqrt(p1)) minus 2 times arcsin(sqrt(p2)). Cohen classified h = 0.2 as small, h = 0.5 as medium, and h = 0.8 as large. These classifications are somewhat arbitrary but provide useful benchmarks. Most A/B tests in digital marketing involve small effect sizes (h < 0.2), which is why large sample sizes are usually necessary.

Reporting effect size alongside p-values is considered best practice in statistics. A statistically significant result with a tiny effect size (say, 0.02 percentage points absolute improvement) may not justify the development cost of implementing the change. Conversely, a large effect size that narrowly misses significance (p = 0.06) may still be worth investigating further or implementing on a trial basis.

Real-World A/B Testing Case Studies

Understanding statistical significance is easier with concrete examples from actual testing scenarios. These examples illustrate common patterns and lessons.

E-Commerce Button Color Test

A medium-sized e-commerce site tested changing their "Add to Cart" button from blue to green. After 15,000 visitors per group, the green button showed a conversion rate of 4.2% versus 3.9% for blue (p = 0.18). The result was not statistically significant. The team correctly decided not to implement the change, noting that the sample might need to be 60,000+ per group to detect a 0.3 percentage point difference reliably.

SaaS Pricing Page Redesign

A SaaS company redesigned their pricing page with clearer feature comparisons. After 8,000 visitors per variant, the new design achieved 12.3% sign-up rate versus 9.8% for the original (p < 0.001, 25.5% relative lift). The confidence interval for the difference was [1.5%, 3.5%], entirely above zero. This was both statistically and practically significant, and the team implemented the change immediately.

Email Subject Line Test

An email marketing team tested two subject lines across 25,000 recipients per variant. Subject A achieved 22.1% open rate, Subject B achieved 21.4% (p = 0.07). At 95% confidence, this was not significant. However, the team used 90% confidence as their threshold for email tests (since the cost of a wrong decision is low and tests cannot be rerun easily). At 90% confidence, the result was significant, and they chose Subject A for the full send.

These examples highlight a important point. The significance threshold should match the stakes of the decision. High-cost, hard-to-reverse changes warrant 99% confidence. Low-cost, easily reversible changes might use 90%.

Frequently Asked Questions

A/B Testing Tools and Platforms

While this calculator handles the statistical analysis, running A/B tests requires a testing platform to split traffic and serve different experiences. Here is an overview of the major platforms and their statistical approaches.

Google improve (Sunset) and Alternatives

Google improve was the most widely used free A/B testing tool before Google discontinued it in September 2023. The platform used Bayesian methods for its reporting, showing the probability that each variant beats the baseline. Popular replacements include VWO (Visual Website Optimizer), which offers a free tier, and Google's recommendation to use third-party tools with Google Analytics 4 integration.

Optimizely

Optimizely uses a sequential testing framework called Stats Engine, which allows continuous monitoring without the peeking problem. It uses the false discovery rate (FDR) approach rather than traditional p-values, controlling the expected proportion of false positives among all declared significant results. This is more appropriate for companies running many concurrent tests.

VWO

VWO offers both Bayesian and frequentist reporting. Its SmartStats engine uses Bayesian methods by default, showing the probability of a variant being the winner and the expected loss from choosing each variant. The expected loss metric is particularly useful for business decisions because it quantifies the downside risk in concrete terms (e.g., "If you choose Variant B and it is actually worse, you expect to lose 0.1% in conversion rate").

AB Tasty

AB Tasty uses Bayesian inference with a sequential testing approach. It reports results in terms of probability to beat the original and improvement range (confidence interval), making results accessible to non-statisticians. The platform automatically handles the multiple testing correction when you compare more than two variants.

Custom Solutions

Many large companies build custom A/B testing infrastructure. Netflix, Uber, Microsoft, and Booking.com have all published papers describing their testing platforms. These custom systems typically use modern statistical methods (like CUPED for variance reduction) that can detect smaller effects with fewer samples than standard approaches. If you are building a custom system, this calculator provides a useful validation tool to verify your implementation against a known-correct calculation.

Choosing the Right Metrics for Your Test

Selecting the correct metric to test is as important as the statistical methodology. The wrong metric can lead you to improve for the wrong outcome.

Primary metrics (also called success metrics or decision metrics) are the metrics that determine whether you ship the change. You should have exactly one primary metric per test. Common primary metrics include conversion rate, revenue per visitor, sign-up rate, and engagement rate. Having a single primary metric avoids the multiple testing problem and provides a clear decision framework.

Secondary metrics (also called guardrail metrics) are metrics you monitor to ensure the change does not cause unintended harm. If your primary metric is purchase conversion rate, your guardrail metrics might include page load time, bounce rate, customer service contact rate, and return rate. If the variant improves conversion rate but significantly degrades a guardrail metric, you should investigate before shipping.

Leading versus lagging metrics matter for test duration. Conversion rate (a leading metric) can be measured immediately after each visit. Customer lifetime value (a lagging metric) takes months to materialize. Using leading metrics keeps test duration manageable, but ensure that improving the leading metric actually drives the lagging business outcome you care about.

Ratio metrics (like conversion rate, which is conversions divided by visitors) are generally more suitable for A/B testing than count metrics (like total revenue or total page views) because they naturally normalize for traffic volume differences between variants. If one variant randomly receives 2% more traffic due to implementation quirks, a ratio metric handles this gracefully while a count metric would be biased.

Related Tools