Email A/B Testing: What to Test and How
Email A/B Testing: What to Test and How
Email A/B testing is how you replace guesswork with evidence. Instead of arguing about whether a subject line or a button color works better, you send two versions, measure what happens, and let your actual audience decide. Done right, it compounds: small, validated wins stack up into materially better campaigns over time.
Done wrong, it’s worse than no testing at all — you draw confident conclusions from noise and “optimize” your way backwards. This guide covers what to test, how to run a fair test, and how to read the results honestly.
What an A/B test actually is
An A/B test splits your audience into two random groups. Group A gets version A, group B gets version B, and everything else stays identical. Because the only difference is the one thing you changed, any meaningful difference in results can be attributed to that change.
The discipline is in that word identical. Change two things at once — a new subject line and a new send time — and you’ll never know which one moved the numbers.
What to test (and in what order)
Test the elements that affect the most people first, then work down the funnel.
1. Subject line
The subject line decides whether the email gets opened at all, so it’s usually the highest-leverage test. Things worth comparing:
- Short vs descriptive
- Question vs statement
- Personalized (name, company) vs generic
- With vs without emoji or numbers
Avoid testing spammy tactics. A clickbait subject might win on opens and lose on trust and complaints.
2. Sender name
People decide whether to open partly based on who it’s from. Test “Acme” vs “Sara at Acme” vs a product name. A recognizable, human sender often beats a faceless brand.
3. Preview text
The snippet next to or under the subject line is prime real estate that many senders waste. Test a complementary teaser vs a continuation of the subject.
4. Body content and copy
Once people open, the body decides what they do next. Test:
- Long vs short copy
- Single CTA vs multiple links
- Plain-text feel vs designed template
- Different value propositions or angles
5. Call to action
Small change, big effect. Test button wording (“Get started” vs “See pricing”), button vs text link, and placement.
6. Send time and frequency
Test mornings vs afternoons, weekdays vs weekends. Just remember timing results are audience-specific — there’s no universal best time. (If you’re rethinking cadence overall, see how often to send marketing emails.)
How to run a fair test
A clean test is the difference between learning something and fooling yourself.
Change one variable at a time
If you change the subject line, keep the body, CTA, send time, and audience identical. One variable, one conclusion.
Decide your metric before you send
Pick the metric that matches the change. A subject line test is judged on opens. A CTA test is judged on clicks. A whole-campaign test is judged on conversions or revenue. Don’t move the goalposts after the fact.
Split the audience randomly
The two groups must be random and comparable. Don’t send version A to your most engaged segment and version B to everyone else — that’s not a test, it’s a confound.
Give it enough sample and time
A test on a few hundred recipients usually can’t tell signal from noise. The smaller your list and the smaller the expected difference, the larger the sample you need. Let the test run long enough to capture normal open and click behavior rather than calling it after an hour.
Reading results without fooling yourself
This is where most testing goes wrong.
- Beware tiny differences. A 41% vs 39% open rate on a small list is probably noise, not a winner.
- Watch the downstream metric. A subject line that lifts opens but tanks clicks didn’t actually win.
- Don’t peek and stop early. Checking constantly and stopping the moment one version is “ahead” inflates false positives. Decide your sample and duration up front.
- Expect some tests to be inconclusive. “No clear difference” is a valid, useful result. It means spend your energy elsewhere.
A win you can’t reproduce wasn’t a win. When a result genuinely matters, it’s worth confirming on a later send.
Build a testing habit
One test won’t transform your program. A steady cadence of tests will.
- Form a clear hypothesis (“a shorter subject line will lift opens”).
- Test one variable against a control.
- Pick the success metric in advance.
- Use a large enough random sample.
- Read the result honestly, downstream metrics included.
- Roll the winner into your default, then test the next thing.
Keep a simple log of what you tested and what you learned. Over time it becomes the most valuable document in your email program — and it keeps Vaillant flying toward the inbox with messages that actually perform.
FAQ
How big does my list need to be for A/B testing? There’s no fixed number, but very small lists struggle to produce reliable results because random variation swamps real differences. The smaller the list and the smaller the effect you’re hoping to detect, the more recipients you need. On small lists, focus on big, obvious changes rather than fine-tuning.
What should I test first? Start with the subject line and sender name, since they determine whether the email gets opened at all. Once opens are solid, move down the funnel to body copy and calls to action.
Can I test more than one thing at once? In a standard A/B test, no — change one variable so you know what caused the result. Testing many combinations at once (multivariate testing) is possible but needs a much larger audience to be reliable.
How long should an A/B test run? Long enough to capture normal opening and clicking behavior for your audience, and long enough to reach a meaningful sample. Avoid stopping early just because one version is briefly ahead — decide your duration before you send.
Bottom line
Email A/B testing turns opinions into evidence: test one variable at a time, pick your metric in advance, use a large enough random sample, and read results honestly. Start with high-leverage elements like subject lines, build a steady testing habit, and let validated wins compound. Pair it with strong fundamentals from our email marketing guide to get the most from every send.