Opens in a new tab

Campaigns

Email A/B testing without fooling yourself

How to choose a metric, size the test, wait the right amount of time and read the result honestly, with a worked example and the exact settings in the Mailnaptic builder.

Mailnaptic team

Published 29 October 2026

Last reviewed 2 October 2026

10 min read

Campaign builder, step 2: A/B Test (example values)

A/B test

Enable A/B testing

On

Control (A)

Subject: Your spring order ships Friday

Preheader: Track it from your account

Variation (B)

Subject: Your spring order ships Friday, 14 April

Preheader: Track it from your account

Determine winner after

2 Days

Winner metric

Clicks

Test sample (%)

50% enters the A/B test (split 50/50 between A and B). The remaining 50% receives the winner.

50

An A/B test is a small experiment on real subscribers, and small experiments are easy to misread. Most of the bad results we see come from the same few mistakes: changing three things at once, judging by opens, stopping the moment a number looks good, and treating a 40 recipient difference as a finding. This post is a working method for avoiding them, using the settings in the Mailnaptic campaign builder as the running example.

If you are still building the rest of your programme, the 30-day email marketing plan for a small business shows where testing fits. Testing is the last thing to add, not the first.

Start with one hypothesis and one change

Write the hypothesis before you open the builder: "Naming the ship date in the subject line will get more clicks than leaving it out." That sentence tells you what to vary (the subject), what to hold still (everything else) and what to measure (clicks). If you cannot write it, you are about to run a coin flip with extra steps.

The rule that matters most is that A and B differ in exactly one respect. If the subject, the layout and the send time all change, a win teaches you nothing you can reuse.

Muddled test

A: "Spring sale" with a hero image, sent at 9:00

B: "Last chance: 20% off" with a text-only layout, sent at 14:00

If B wins, you cannot say why. Subject, layout and send time all moved.

Clean test

Hypothesis: naming the ship date in the subject gets more clicks than leaving it out.

A: "Your spring order ships Friday"

B: "Your spring order ships Friday, 14 April"

Same content, same time, one difference. A result teaches you something.

One change versus several. The example subject lines are illustrative, not from a real campaign.

Pick a metric that survives Apple privacy features

Apple says Mail Privacy Protection "stops senders from using invisible pixels to collect information about the user" and masks their IP address. In practice, the tracking image in your email can be requested on the reader's behalf, so an open can be recorded when nobody looked at the email. We cover what that does to your reports in open rates after Apple Mail Privacy Protection and what to measure instead.

For a test, the consequence is simple. If your subscribers include a lot of Apple Mail users, an open-based winner can partly measure who uses Apple Mail and how their client fetches images. Mailnaptic ignores automatic opens that it detects when it picks an open winner, but detection is not perfect, and clicks are not subject to that distortion at all. Mailnaptic's builder offers two winner metrics, Opens and Clicks, and defaults to Clicks. Leave it there unless you have a specific reason.

Metric

What it measures

Good for picking a winner?

Opens

A tracking image was requested. Mail Privacy Protection can request it with no human involved.

Weak. Inflated and uneven between Apple and non-Apple readers.

Clicks

Someone (or something) followed a link in the email.

Best default. Closer to intent, and not affected by image preloading.

Conversions / revenue

A purchase or goal after the click.

Best when you have volume and can wait days. Needs ecommerce or site tracking.

What each winner metric really tells you. Open behaviour description follows Apple's published description of Mail Privacy Protection.

Clicks are not perfect either. They are rarer than opens, so you need more recipients to see a difference, and a subject line test is really testing "does this subject make people open and then click", which mixes two behaviours. If you sell online and have ecommerce tracking connected, revenue or orders per recipient is the most honest metric, but it needs a lot of volume and a longer wait.

Work out the sample size before you send

This is the step people skip, and it decides whether the test can tell you anything. The question is: how many people must receive each version before a real difference of a given size shows up reliably?

The standard normal-approximation answer for comparing two proportions uses four inputs: your baseline rate, the smallest lift you care about, your tolerated false positive risk (usually 5%) and your desired power (usually 80%, meaning an 80% chance of catching the lift if it is real). The formula below is the textbook one for two groups. NIST's engineering statistics handbook gives the same building blocks, including the pooled z statistic for two proportions that the formula is derived from. We checked the formula by simulation: at 5,301 per version, a true 3% versus 4% difference reached significance in about 80% of 20,000 simulated tests.

Sample size table (calculated, per version)

Recipients needed per version (5% false positive risk, 80% power)

Baseline click rate

Lift to detect: +20%

+50%

+100%

1%

42,693

7,750

2,319

2%

21,109

3,826

1,141

3%

13,914

2,518

749

5%

8,158

1,471

435

Computed by us with the formula below. Relative lift means, for example, 3.0% to 3.6% for +20%.

Per version, two-sided test. Double the figure for the total test sample. These are calculations, not benchmark data.

If you want to run the numbers yourself, this is the formula:

n per version = ( z(1 - a/2) * sqrt(2 * pbar * (1 - pbar)) + z(1 - b) * sqrt(p0 * (1 - p0) + p1 * (1 - p1)) )^2 / (p1 - p0)^2 p0 = your current click rate (baseline) p1 = the click rate you hope B reaches pbar = (p0 + p1) / 2 a = 0.05 (5% false positive risk), z(1 - a/2) = 1.96 b = 0.20 (80% power), z(1 - b) = 0.84 Example: p0 = 3.0%, p1 = 4.0% -> about 5,300 per version

A caveat on the formula: it is an approximation, and it assumes you will test once at the planned size. An online calculator is a sensible cross-check.

Two things to take from the table. First, the numbers are per version, so a test needs twice the figure. Second, they are large. A baseline click rate of 3% needs about 14,000 recipients per version to detect a 20% relative lift, which most small lists simply do not have.

What this means for a small list

With a 10,000 person list and the test sample slider at its maximum of 90%, you get 4,500 per version. That is not enough to reliably detect even a jump from 3% to 4%. Small lists can still test, but only for big differences (a doubling from a 3% baseline needs about 750 per version), or by repeating the same test over several sends.

What the Mailnaptic A/B step actually asks for

Enable A/B testing in step 1 of the campaign builder and you get a Control A canvas and a Variation B canvas. Variation B needs its own subject line, and it can have its own preheader and its own content blocks. There is an AI Generate B button that drafts a variation with a stated hypothesis. Treat that as a starting point. It does not change the rules above: if it rewrites the subject, the layout and the offer, you have a muddled test.

Step 2 then asks for three things, shown in the example above. Determine winner after takes a number in hours or days. Winner metric is Opens or Clicks. Test sample (%) is a slider from 10 to 90. Whatever share you choose is split 50/50 between A and B, and the rest of the list receives the winner. So a 40% sample on a 20,000 person list sends each version to 4,000 people and the other 12,000 get whichever wins.

Note what the step does not ask for: a significance threshold. It picks a winner on the metric you chose after the time you set. That is a reasonable way to decide what to send to the remainder, but it is not a statistical verdict. Do your own reading of the numbers, as described below, before you add a rule to your playbook. If you use the builder's rehearsal feature, it has a variant switcher, so you can check both A and B before anything is sent. See rehearse a campaign before you send it.

A two-day test window (example schedule)

Tuesday 09:00: send A and B to the test sample

Both versions go out at the same moment

Tuesday evening: do nothing

Early clicks come from your most eager readers, not a fair cross-section

Thursday 09:00: winner chosen (2 days set in the builder)

Most clicks for a typical campaign have arrived by now (our heuristic)

Thursday: remaining recipients get the winner

Read the final numbers in Analytics, with the interval, before you decide what you learned

An example timeline for a two-day wait. The "most clicks have arrived" point is our heuristic, not a measurement.

How long to wait

There is no universal answer, and vendors disagree on details. Mailchimp's guidance says to wait 2 to 12 hours for an open-based test, 1 to 3 hours for a click-based test and 12 to 24 hours for revenue, and also that longer is better on smaller lists. Those figures are Mailchimp's own, drawn from their data, so treat them as a floor rather than a rule.

Our heuristic, not a measured figure: for a normal campaign, give clicks at least a full day and preferably two, which is why the builder's default is 2 Days. Early clicks are dominated by your fastest, most engaged readers, and a version that happens to land at a better moment can look like a winner for the wrong reason. Pick the duration in advance and do not shorten it because the early numbers look decisive.

A vs B results (hypothetical numbers)

Example results, 5,000 recipients per version (sample data)

Control A clicks

150

3.0%

Variation B clicks

190

3.8%

Observed difference

+0.8 pts

z = 2.21

A: clicks per 100 recipients

3.0

B: clicks per 100 recipients

3.8

Statistically distinguishable from zero at the 5% level, but the plausible true lift ranges from almost nothing to 1.5 points.

Hypothetical results. 150 of 5,000 clicks (A) versus 190 of 5,000 (B), z = 2.21.

Peeking, and why you should not stop early

Significance maths assumes you fixed the sample size first. Evan Miller's well-known write-up shows what happens if you check repeatedly and stop at the first significant reading: with a 5% target, testing after every observation produces false positives 26.1% of the time. The fix is boring: decide the sample and the wait time up front, then read the result once.

Significant is not the same as worth acting on

"Significant" means the observed difference would be unlikely if A and B were truly identical. It says nothing about whether the difference is large enough to matter. The z statistic NIST gives for two proportions divides the observed difference by its standard error, using the pooled rate in the denominator. In our first example that is 0.8 points divided by about 0.36 points, giving z of 2.2, which clears the 1.96 bar for a 5% two-sided test.

The interval is more useful than the verdict. A 95% interval for the difference uses the same ingredients: the observed difference plus or minus 1.96 standard errors, where the standard error combines both versions' variance. For test 1, that gives +0.09 to +1.51 points. The lift is probably real, but it could be as small as a tenth of a point or as large as a point and a half. For test 2, the interval runs from minus 0.6 to plus 2.6 points. B looked better by a full point, yet the data are consistent with B being worse. NIST's handbook does not print a confidence interval for a difference of proportions, so this one is the standard textbook construction rather than a quote.

To decide whether a lift is worth acting on, convert it to money or customers. A 0.8 point click lift on a 20,000 person list is 160 extra clicks per send. If a click is worth a little and the change costs nothing to repeat, adopt it. If it needs a design rebuild every time, a lift that might be 0.1 points is not worth it.

Confidence intervals for the difference (hypothetical numbers)

Same observed lift, different amounts of evidence (sample data)

Test 1: 5,000 per version, 150 vs 190 clicks

difference +0.8 points, 95% interval +0.1 to +1.5

Test 2: 1,000 per version, 30 vs 40 clicks

difference +1.0 points, 95% interval -0.6 to +2.6

-1 pt

0 (no difference)

+3 pts

Test 2 looks like a bigger win (1.0 point) but the interval crosses zero: it tells you almost nothing.

Hypothetical results. The red line is zero difference. Interval 1: 0.8 points, +0.09 to +1.51. Interval 2: 1.0 point, -0.61 to +2.61.

Repeat tests and keep a log

One test is one sample. A result that shows up in three separate sends is a finding. A result that shows up once is a lead. If you run several tests at once on the same send, remember that each carries its own 5% false positive risk. Three independent tests give a 14% chance (1 minus 0.95 cubed) that at least one looks like a win by luck.

Keep a plain log: date, hypothesis, versions, sample per version, metric, difference, interval, what you decided. Put your next subject line experiments there too. For help generating variants that are honest, read subject lines and preheaders as a pair and using AI for subject lines without making claims you cannot back up, because a test will happily crown a misleading subject line if it gets more clicks.

What each plan can test

A/B testing arrives on the Growth plan with subject line and content tests. Pro and Business have the same campaign tests. Content tests change blocks inside the email, which makes the one-change rule harder to keep, so be strict with yourself. Inside an automation you can also split contacts into 2 to 5 branches, and the sample size table above applies to every extra version you add.

Plan

A/B testing in Mailnaptic

Starter ($14)

Not included

Growth ($29)

Subject line and content tests

Pro ($49)

Subject line and content tests

Business ($99)

Subject line and content tests

A/B testing by plan. Check the pricing page for the current plan list.

A starting routine

Test one subject line change per campaign for a few sends. Use Clicks as the metric, give it two days, keep the sample as large as your list allows, and log the result with its interval.

Try it in Mailnaptic

Turn on A/B testing in the campaign builder, write a Variation B, pick your winner metric and send. Growth and above include subject line and content tests.

Keep reading

Rehearse a campaign before you send it

Check both versions before you spend your test sample on a broken one.

Open rates after Apple Mail Privacy Protection and what to measure instead

Why opens make a poor winner metric.

Subject lines and preheaders as a pair

What to put in B when the subject line is your test.

Using AI for subject lines without making claims you cannot back up

Keep generated variants honest.

Email marketing for a small business: a 30-day plan

Where testing fits in the bigger programme.

Sources