A/B testing broadcasts
By Clint ParsleyLast updated
In short. Test two subjects or two templates on part of the list, measure for a window, and the winner goes to the rest automatically. No clear winner means variant A ships, never a coin flip.
A broadcast can run as a holdout test: part of the list gets the variants, SEMAOS measures for a window you choose, and the rest of the list gets whichever variant won. Turn it on in the broadcast composer with "Test on part of the list, then send the winner to the rest." The whole point of the design is that deciding is automatic and conservative; you set it up and the machinery does the honest thing.
What can I test?
Two kinds of variant. Subject tests send the same body under different subject lines and pick the winner by open rate. Content tests send different templates and pick by click rate, because when the bodies differ, opens can't tell the bodies apart; only what recipients did inside the email can. The composer switches metric automatically based on what you varied.
A variant's click rate counts the people who clicked, out of the delivered emails that had a tracked link in them. The unsubscribe link isn't tracked, so an email whose only link is the unsubscribe link isn't counted, and a variant with no tracked links shows a dash instead of a click rate. A content test's winner is still chosen on clicks per delivered email, so the 100-delivered floor below applies to the number that decides.
How is the winner decided?
At the end of the measurement window, a winner must clear two bars:
- The floor: at least 100 delivered emails per variant. Below that, rates are noise; a 3-of-7 open "lead" means nothing. The composer warns you upfront when your test-group size can't reach the floor.
- The margin: a real lead over the runner-up. Subject tests need an open-rate lead of at least 8 percentage points; content tests need a lead of at least 2 points in clicks per delivered email, since click rates run much lower than open rates.
If either bar isn't cleared, variant A (your control) ships to the remainder. Deliberately not "whichever was ahead": shipping a statistically meaningless leader would launder noise into a decision. Writing your safer, known-good version as variant A is therefore part of setting up the test.
What does a decision look like with real numbers?
Say you broadcast to 5,000 contacts with the test group at 20%. About 1,000 people get the test, split between two subject lines, so each variant lands with roughly 500. Three ways the measurement window can end:
| Variant A (control) | Variant B | What the rest of the list gets |
|---|---|---|
| 490 delivered, 147 opens (30%) | 495 delivered, 198 opens (40%) | B: both cleared 100 delivered, and B leads by 10 points, past the 8-point margin |
| 490 delivered, 147 opens (30%) | 495 delivered, 173 opens (35%) | A: B leads by 5 points, short of the 8-point margin |
| 40 delivered, 12 opens (30%) | 42 delivered, 21 opens (50%) | A: neither variant reached 100 delivered, so the 20-point lead counts for nothing |
The last row is a 400-contact list at the same 20%. It is why small lists rarely get a winner: to put 100 delivered emails on each of two variants at 20%, the list needs a little over 1,000 contacts.
What are the knobs?
The test-group percentage (how much of the list tries the variants), the measurement window (minimum one hour; longer windows suit lists that open slowly), and approve-first mode. With approve-first on, nothing goes to the remainder until you review the results and click send; the winner's numbers wait for you on the results page. With it off, the remainder goes out automatically the moment the window closes.
What does the results page show?
Per-variant delivered, opens, and clicks, with the winner badged once decided. The remainder's sends are attributed to the broadcast as a whole rather than to a variant, so variant stats stay a clean read of the test cohort only; the winner badge tells you what the remainder received.
When is a test worth running?
When the remainder is big enough to benefit. If your whole list is 300 contacts, a compliant test barely leaves a remainder to optimize for, and you're better off sending your best guess to everyone and applying what you learn next time. The floor-and-margin math starts paying for itself in the low thousands of recipients.