What this calculator does
You ran a split test. Variant B beat variant A. This tool answers the only question that matters next: is that gap real, or did you just flip coins for two weeks?
It runs a two-proportion z-test, the standard method for comparing two conversion rates. It is two-tailed, meaning it checks for a difference in either direction, not just the one you were hoping for.
SE = √( p × (1 − p) × (1 ÷ visitors A + 1 ÷ visitors B) )
z = (rate B − rate A) ÷ SE
p-value = 2 × (1 − Φ(|z|))
The pooled rate treats both variants as one bucket. That is the null hypothesis: assume no difference, then measure how surprising your data looks under that assumption. Φ is the standard normal CDF; the calculator evaluates it with the Abramowitz–Stegun approximation, accurate to about seven decimal places.
Worked example
Take the calculator's defaults. Variant A: 4,800 visitors and 144 conversions, a 3.00% rate. Variant B: 4,800 visitors and 180 conversions, so 3.75%. That is a +25.0% relative lift.
Pool them: 324 conversions ÷ 9,600 visitors = 3.375%. The standard error works out to √(0.03375 × 0.96625 × (1/4,800 + 1/4,800)) = 0.00369. The observed gap is 0.75 percentage points.
p-value = 2 × (1 − Φ(2.03)) ≈ 0.042
A p-value of 0.042 means: if A and B truly converted at the same rate, a gap this large would appear in roughly 4 of every 100 tests. That clears the 0.05 bar: significant at 95.8% confidence, and the verdict card goes green. Shrink B to 170 conversions and the same lift direction drops out of significance. That is how close most “winners” live to the line.
The peeking problem
Here is how most tests get called wrong. You launch Monday, check the dashboard every morning, and stop the moment the tool says significant. That habit inflates your false-positive rate far past the advertised 5%.
A test bouncing around zero will cross the significance line at some point by luck alone. Stopping at the first green light means you sampled the noise at its most flattering moment. The p-value you report no longer means what it claims.
The fix is boring and it works: decide the sample size before you launch, run to that number, then read the result once. The A/B Test Sample Size Calculator gives you that number from your baseline rate and the smallest lift you care about. Until you hit it, the p-value here is a progress report, not a verdict.
Significant is not the same as meaningful
Significance says the gap is probably real. It says nothing about whether the gap is worth anything. Push enough traffic through a test and a 1% relative lift will eventually clear the bar. Real, yes, and still too small to pay for the build time to ship it.
The reverse bites harder. With small samples, a genuinely large lift can read as not significant, and teams kill promising variants because the test never had the power to catch them. Not significant means “not proven”, never “proven equal”.
So before you ship or kill anything, translate the lift into money. The Conversion Rate Calculator shows what a 0.75-point gain is worth in orders and revenue at your traffic. A verdict plus a dollar figure is a decision; a verdict alone is trivia.
Frequently asked questions
What does p < 0.05 actually mean?
It means: if there were truly no difference between the variants, data this lopsided would appear less than 5% of the time. It is a statement about your data under a no-difference assumption, not a 95% probability that B is better. The distinction matters once you run tests in volume: at the 0.05 bar, roughly 1 in 20 no-difference tests will still flash significant.
One-tailed or two-tailed: which is this?
Two-tailed. It asks whether the variants differ in either direction, the honest default when you would act on a loss as well as a win. A one-tailed test halves the p-value but assumes you knew the direction in advance and would ignore a result the other way. If one-tailed is the only way your result clears the bar, you do not have a result. You have a rounding trick.
Can I stop the test early once it hits significance?
Not with this kind of test. The 5% false-positive guarantee holds only if you fix the sample size first and read the answer once, at the end. Repeated peeks give a fluke many chances to get caught mid-wobble. If you genuinely need early stopping, use a sequential testing framework built for it. Otherwise run to your planned sample.
How big a sample do I need?
It depends on your baseline rate and the smallest lift worth detecting. Small lifts on low baseline rates need tens of thousands of visitors per variant; large lifts on high rates need far fewer. Work the number out before launch with the sample size calculator linked above. Guessing after the fact is how peeking starts.
My test is significant but the lift is tiny. Ship it?
Usually yes, if shipping is nearly free: a real 2% lift compounds across every visitor after it. The real question is cost: build time, added complexity, a variant that is harder to maintain. Turn the lift into monthly orders and dollars, then compare against what shipping costs. Significance clears the “is it real” bar; it never answers “is it worth it”.
Does this work for revenue per visitor or AOV?
No. This test compares proportions, binary outcomes like converted or did not. Revenue per visitor is continuous and skewed hard by a few big orders, so it needs a t-test or a bootstrap, plus larger samples than intuition suggests. A practical pattern: call conversion rate with this tool, then check that AOV did not quietly drop on the winner.
Where StefanBrain fits
Calling winners is the cheap half of testing. The expensive half is feeding the machine: enough genuinely different concepts that one has a chance to win big. StefanBrain generates the hooks, static ads, video scripts, and landing pages that keep a testing pipeline full, trained on 7+ years of DTC marketing IP. Then size the spend side with the Creative Testing Budget Calculator so every concept gets enough data for this page to judge it fairly.