A/B testing with low traffic: how many visitors you really need

An A/B test needs as many visitors as your conversion rate and the effect size you want to detect require: at a 2% conversion rate and a target lift of 20%, that’s roughly 21,000 visitors per variation. A small site therefore doesn’t measure small differences, it measures big ones. It tests bolder changes, tests steps with higher conversion rates, and relies on qualitative data for the small stuff. The key concept is the minimum detectable effect (MDE): the smallest lift your test can reliably detect at your traffic.

All numbers in this article assume a two-sided test, a 5% significance level and 80% power, the same as our sample size calculator. The example sites are illustrative, with rounded numbers.

What are effect size and minimum detectable effect?

Effect size is the difference between variations. It can be expressed two ways, and it pays not to mix them:

absolute, in percentage points: 2.0% to 2.4% is +0.4 pp,
relative, in percent: 2.0% to 2.4% is +20%.

Tools and calculators usually work with the relative difference. When someone says “the test looks for a 20% lift”, they mean 2.0% → 2.4%, not 2% → 22%.

The minimum detectable effect (MDE) is the smallest true difference a test of a given sample size catches with 80% probability. If the true effect is smaller than the MDE, the test usually won’t find it and ends without a winner. And when it does “find” it, the measured lift tends to be overstated, because with a small sample only the results chance has inflated make it to significance.

So the MDE is a property of your test, not of your change. It tells you how big a change is worth testing.

How many visitors do you need for different effects?

Required sample grows with the square: half the effect costs roughly four times the visitors.

Conversion rate	Target lift	Visitors per variation
2%	+10%	~81,000
2%	+20%	~21,000
2%	+30%	~9,800
2%	+50%	~3,800
5%	+20%	~8,200
20%	+10%	~6,500
20%	+20%	~1,700

Two things in the table are worth remembering. A higher conversion rate means a smaller required sample, because conversions accumulate faster. And small tweaks that move results by a few percent need traffic only a small share of sites have.

Example: a site with 10,000 visitors a month

An illustrative site: the tested page gets 10,000 visitors a month and converts at 2%. With a 50/50 split, each variation gets about 5,000 visitors a month.

After one month, the MDE is about 43%. The test reliably catches only a lift from 2.0% to roughly 2.9%.
After eight weeks (about 9,000 per variation), the MDE is around 30%, from 2.0% to 2.6%.

A new headline or button colour almost certainly won’t make that kind of difference. The test would end inconclusive and you’d lose two months.

Why small sites must test bigger changes

Once you know your test’s MDE is 30%, it only makes sense to test changes with a real chance of producing that difference:

a different offer or price presentation, not a different word in the headline,
a rebuilt page structure, not a moved button,
a much shorter form or a removed step, not a renamed field,
a different main argument: lead with a different benefit, audience or objection.

The downside is that when you win, you don’t know exactly which element of the big change worked. On a small site that’s an acceptable price: knowing the new version of the page is better beats knowing nothing. You can fine-tune the details later, when you have more data.

Test earlier, higher-volume steps

The conversion rate for a full purchase is usually low. Intermediate steps convert much higher: product page to cart, homepage to pricing, form step one to step two.

In the same illustrative example: if 20% of product page visitors add to cart, 5,000 visitors per variation in a month give an MDE of about 11%. That’s a far smaller difference than 43% for purchases, and a far more realistic test.

Two rules to make it work:

The intermediate step must relate to the purchase. More add-to-carts don’t necessarily mean more orders if the change only attracts the curious. Set the primary metric on the intermediate step, but watch the final conversion too, so the variation doesn’t hurt purchases.
Test where most people pass through. The homepage and category pages collect a sample faster than the last checkout step. To find where your funnel loses the most people, follow the guide to funnel analysis in GA4.

Does it matter if variations have different sample sizes?

The statistical test handles unequal groups, so a result with slightly different counts is fine. Two situations are different:

A deliberately uneven split (say 90/10, so fewer people see the new variation) is less efficient. An 80/20 split needs about 1.6 times the total visitors of a 50/50 split, and 90/10 close to three times. On a small site, 50/50 is almost always the best choice.
An unexpectedly uneven split (you planned 50/50 and got 55/45 across thousands of visitors) signals a bug: a redirect, caching, a blocked script or bots. Such a test can’t be fixed, only discarded and rerun.

Can I lower the sample requirement?

Partly, but always at a price. Loosening the significance level from 5% to 10% cuts the required sample by about a fifth, but doubles the rate of false winners. For a cheap, easily reversible change that can be a reasonable trade; for a price or offer change it isn’t. Decide in advance, not after you see the result.

What doesn’t lower the sample: peeking and stopping the test the first time it “wins”. That only raises the share of false results; why your A/B test lies explains why. Also don’t stretch a test past about eight weeks; by then campaigns, season and the site itself have changed.

Before/after comparison: usable, with care

When you don’t have the traffic for a test, the remaining option is to ship the change and compare before and after. The problem is that you mix the effect of the change with everything else: seasonality, campaigns, outages, chance. It works as a directional signal if:

you compare whole weeks and the same weekdays, ideally at least two weeks before and two after,
you don’t change campaigns, prices or other parts of the site in either period,
you track a control metric the change shouldn’t affect (say, the conversion of a page you didn’t touch); if that moved too, something else probably caused the change,
in seasonal businesses you also compare the same period last year.

Treat the result as a signal, not proof. If conversion drops, roll the change back.

Qualitative methods: where you don’t need a test

On a small site, the most valuable thing is fixing obvious barriers without a test. When five out of five people in user testing can’t find the form’s submit button, you don’t need an A/B test, you need a fix.

Qualitative data that works even at low traffic:

user testing with five to eight people from your target audience,
session recordings and heatmaps, for example in Microsoft Clarity (compared in Microsoft Clarity vs. Hotjar),
a short on-site survey: “What’s stopping you from completing your order?”,
interviews with customers who bought, and with those who left.

Qualitative data tells you what to fix and what’s worth a big test. The test then confirms whether it actually helped.

Frequently asked questions

How many visitors does an A/B test need?

It depends on your conversion rate and the size of the effect you want to detect. At a 2% conversion rate, detecting a 20% lift takes roughly 21,000 visitors per variation, and a 50% lift roughly 3,800. On a step converting at 20%, a 20% lift needs about 1,700 per variation.

What is the minimum detectable effect (MDE)?

The smallest difference between variations that a test of a given sample size reliably detects, usually with 80% probability at a 5% significance level. The smaller the sample, the bigger the true effect has to be for the test to find it.

Can I A/B test with low traffic?

Yes, but differently from large sites. Test bigger changes that can produce a big difference, test steps with a higher conversion rate or more traffic, and base small tweaks on qualitative data instead of a test that has no chance of proving anything.

Is it a problem if variations have different sample sizes?

The statistical test handles unequal groups, but an uneven split is less efficient: a 90/10 split needs close to three times the total visitors of a 50/50 split. If you planned 50/50 and got a clearly different split, that is a measurement bug and the result can’t be used.

Is a before/after comparison a substitute for an A/B test?

Only partly. Comparing two periods mixes the effect of the change with seasonality, campaigns and chance. It works as a directional signal when you compare whole weeks, the same weekdays, no campaign changes, and a control metric the change shouldn’t affect.

The takeaway

Low traffic doesn’t mean you can’t test. It means you need to know in advance what effect your test can carry, and choose changes and funnel steps accordingly. Compute your MDE, test boldly and where the traffic is, and fix the rest based on qualitative data.

Software can do this work too. Kaen, an AI CRO agent, calculates the required sample before every test, picks a step where the result can be measured, and tells you when a change can’t be measured at your traffic. On a smaller site he runs one test and lets it run longer, until the result is clear. A demo shows what it would cost for your site; nothing goes live without your approval.