A/B Testing: How to Run a Test and Read the Results Right

In June 2026, ConversionTeam published an audit of 2,288 A/B tests across 71 clients. Counting anything that simply finished ahead, 50.5% of new variations “won.” Apply statistical significance to those same tests, and 19.1% were left.

That gap is what this article is about. Running an A/B test is the easy part: the tools split the traffic for you. The hard part is not celebrating a difference that was only noise, then spending a whole quarter shipping nothing.

To run an A/B test, write a hypothesis, pick one primary metric, calculate the required sample before you start, split traffic randomly between two versions and wait until the planned end date. Only then compare: a version wins when the confidence interval for the difference sits entirely above zero.

What an A/B test is, and what it isn’t

An A/B test compares two versions of one thing (a page, an ad, an email subject line, a price) by showing each to a randomly assigned group at the same time. Randomization is where the value comes from: because the groups are alike in every other way, a difference in results most likely comes from your change.

So swapping a page on Monday and comparing it with last week is not an A/B test. It is a before-and-after, contaminated by holidays, campaigns, weather and payday. It gives you a clue, not proof.

How to run an A/B test in 6 steps

1. Write the hypothesis before you open the tool

A simple format: “if we show shipping costs on the product page, conversion will rise, because the checkout surprise is what kills the sale.” Without the “because,” you only learn which version won, not why. The why is what carries over to your next test.

2. Pick one primary metric

Just one, chosen in advance. Track 12 metrics and one will rise by chance, and your report will call that a win. Everything else is a guardrail: conversion can go up, but average order value and returns cannot drop. If you are unsure about the denominator, start with our guide to conversion rate.

3. Calculate the sample before you start

This is the step nearly everyone skips. You need three numbers: current conversion, the smallest lift that would justify the change (the minimum detectable effect) and the statistical standard, usually 95% confidence with 80% power. The table below shows the sessions required per version under that standard.

Current conversion	10% lift	20% lift	30% lift	50% lift
1%	163,100	42,700	19,800	7,800
2%	80,700	21,100	9,800	3,800
3%	53,200	13,900	6,500	2,500
5%	31,200	8,200	3,800	1,500

Notice the price of precision: detecting a 10% lift takes almost four times the sample of a 20% lift. In practice, a new button color rarely moves enough to show up. Test big changes instead: the offer, the guarantee, the page order, the value proposition.

4. Randomize by person and run both versions together

Someone who saw version B needs to keep seeing B when they come back. With both live at once, campaigns and seasonality hit the two groups equally.

5. Run full weeks

Even after reaching your sample, close out 7-day cycles, because Saturday shoppers do not behave like Tuesday shoppers. Avoid ending a test in the middle of a period like Black Friday: that week’s audience is different, and the winner may not hold in December.

6. Read it only at the end

Looking at the dashboard does no harm. Deciding from it does. The next section shows why.

Why stopping when it “hits significance” creates false winners

That 95% confidence applies to a single reading at the planned sample size. Every extra look with decision power is one more chance for luck to cross the line.

We ran a simulation to measure it: 20,000 tests in which A and B were identical, with 3% conversion, 500 sessions a day per version, over 14 days. Reading only at the end, 4.9% flagged a false winner, close to the expected 5%. Calling it on the first day the dashboard showed significance, the figure was 21.7%.

Some methods are built for continuous reading, such as sequential testing, and certain tools already use them. If yours does, great: check its documentation. If you don’t know, assume it doesn’t and wait for the end date.

Worked example: how to read A/B test results

A cosmetics store gets 2,000 sessions a day, converts at 3% and has a $180 average order. The hypothesis is to show shipping on the product page, and the goal is to detect a 20% lift. Per the table, that takes 13,900 sessions per version, or 14 days. The team rounds up to 14,000 on each side.

On day five, with 5,000 sessions per version, B had 3.72% against A’s 3.00%: a 24% lift and a p-value of 0.046. The temptation to stop was real. The team waited. In the end:

• Version A: 420 orders from 14,000 sessions, 3.00%.
• Version B: 492 orders from 14,000 sessions, 3.51%.
• Difference: +0.51 percentage points, a 17% lift.
• P-value: 0.015.
• 95% confidence interval: +0.10 to +0.93 points.

B won, and the lift shrank from 24% to 17%. That is the usual pattern: results that show up early tend to be inflated. So the useful question is “how big is the likely gain?”, and the interval answers it, not the p-value.

At 60,000 sessions a month, that interval means between $10,800 and $100,000 in extra monthly revenue, centered near $55,000. That is a wide range. If rolling it out cost $30,000 a month in subsidized shipping, the decision would not be settled, significance or no significance. Run the math on margin, not revenue.

What statistical significance means, without the jargon

The p-value answers one specific question: if there were no difference at all, how often would we see a result this extreme by chance? In the example, 0.015 means 1.5% of the time.

It does not say there is a 98.5% chance that B is better, even though plenty of dashboards almost say so. It also says nothing about the size of the gain: with a huge sample, an irrelevant improvement becomes “significant.” Significance tells you whether to trust the signal; whether it is worth doing comes from the business math.

And an inconclusive result is not a failure. It tells you the change, if it does anything, sits below what the test can see. In that case, keep whichever option is simpler to maintain. Resist “one more week” with no plan: that is the same peeking in disguise, one of the biases that distort decisions.

Three checks before you trust a result

1. Did the split come out as planned? With 28,000 sessions split evenly, expect a swing of about 160 either way. If you got 14,420 against 13,580, the odds of that being chance are below 1 in a million. Something is broken, probably a redirect or tracking, and the test should be thrown out.
2. Does the gain show up in every segment that matters? A desktop-only win can hide a loss on mobile, where most of your traffic lives. Use breakdowns to investigate, not to dig up a winner after the fact.
3. Was conversion counted the same way on both sides? A duplicated event in one version manufactures a win. Check orders in your sales system, not only in the testing tool.

What if you don’t have the traffic to test?

This is more common than it sounds. In Convertcart’s review of 1,103 experiments from June 2025 to June 2026, about 1 in 6 audited stores lacked the traffic for a valid test.

Picture a B2B company with 6,000 sessions a month and 2% lead conversion. Detecting a 20% lift requires 42,200 sessions: seven months. That leaves three honest options:

• Test bigger changes. Aiming for a 50% lift, the test drops to 7,600 sessions, a little over five weeks.
• Measure one step higher in the funnel. The click on the quote button, at a 10% rate, needs 7,600 sessions for a 20% lift. The risk is optimizing clicks that never become sales, so watch the next stage too.
• Don’t test. Use interviews, session recordings and funnel analysis, and own the change as a business decision. Better that than a weak test with a stamp of science on it.

Paid media follows the same logic, and the unit that counts is the conversion, not the impression. A great CTR with 40 conversions per side proves little about cost per sale. See how to structure that reading in our guide to analyzing Meta Ads campaigns.

What win rate should you expect?

Much lower than case studies suggest. The ConversionTeam audit (2026) reports that 19.1% with significance, using a one-tailed test at 95%. Convertcart (2026) reports 37% significant improvements. The gap between them shows how much definition and method matter, and both are optimization firms whose clients already run testing programs.

If a company just starting out wins 2 tests in 10, that is normal. If it wins 8 in 10, question the yardstick before celebrating.

Frequently asked questions

How long should an A/B test run?

As long as your calculated sample requires, rounded up to full weeks. Two weeks is a good practical minimum to cover day-to-day variation. Going past four to six weeks brings another risk: cookies expire and people start landing in the other version.

How does A/B testing work in Meta Ads?

Ads Manager has its own tool that splits the audience between versions, so the same person does not see both, and it picks a winner by cost per result. Change one variable at a time, such as creative or audience. Duplicating a campaign by hand is not equivalent, because audiences overlap and the algorithm spends unevenly.

What is the difference between statistical and practical significance?

Statistical significance says the difference is probably real. Practical significance says whether it pays for the change. A 0.02-point gain can be significant on a huge base and still not cover the development cost.

Can I test more than two versions at once?

You can, but the sample gets divided and the error grows. With three new versions against the original, the chance of at least one false winner rises from 5% to about 14% if nothing is corrected. With limited traffic, two very different versions usually teach you more.

What confidence level should I use?

95% is the standard for decisions that are hard to reverse. For cheap, easily undone changes such as an email subject line, 90% is acceptable, knowing you are trading more false winners for speed.

Good testing means setting the rules before you see the result

A/B test statistics fit in one table and a calculator. What separates teams that learn from teams that collect fake wins is the discipline to set metric, sample and timeline up front, and to accept it when the data says “I don’t know yet.”

The tedious part is usually tying the test result to what happened next, in your CRM and revenue. With Sherlok, you ask in plain language and get the analysis ready, with Meta Ads, Google Ads, GA4 and CRM in the same conversation, so you can see whether the winning version also brought in better customers.