Owlat Docs

A/B Testing

Compare two variants of a campaign on a test group, then automatically or manually send the winning version to the rest of your audience.

A/B testing lets you compare different versions of your email to find out what resonates best with your audience. Instead of guessing which subject line or layout works better, you test both with a small group and send the winner to everyone else.

How A/B testing works

  1. You create a campaign with two variants — variant A and variant B (e.g., different subject lines).
  2. A small percentage of your audience (the test group) is split randomly between variant A and variant B.
  3. After the test duration, Owlat picks the best-performing variant based on your chosen metric (or you pick the winner yourself).
  4. The winning variant is sent to the rest of your audience.

This way, the majority of your audience always gets the version that performed best.

Setting up an A/B test

1. Create your campaign

Start a new campaign as usual in Campaigns > All Campaigns. Fill in the basics (campaign name, from name, from address) and audience on the Setup step, then your subject line on the Content step.

2. Add variants

On the Setup step, open the Add an A/B test expander and choose a test type for variant B:

  • Subject line test — same email content, a different subject line for variant B. This is the most common and easiest test to run.
  • Content test — variant B uses a different saved email template (not free-form content). Use this when you want to test entirely different layouts or approaches.

Test one thing at a time. If you change both the subject line and the content, you won't know which change made the difference. A clear, single hypothesis per test gives you actionable results.

3. Configure the test

SettingWhat it means
Test Split PercentageThe percentage that receives each variant, set with a slider in 10% steps from 10% to 50%. At 20%, variant A and variant B each go to 20% of your audience, and the remaining 60% gets the winner.
How the winner is selectedBest open rate, best click rate, or I'll choose manually. Pick open or click rate to have Owlat decide automatically; pick manual to choose the winner yourself on the campaign report.
Test Duration (hours)How long Owlat waits before auto-picking the winner. Choose from preset options of 1, 2, 4, 6, 12, or 24 hours. This field only appears when the winner is selected by open or click rate — manual tests have no timer.
Auto-send is tied to how the winner is chosen

There is no separate "auto-send" toggle. If you pick best open rate or best click rate, the winner is sent automatically once the test duration elapses. If you pick I'll choose manually, no timer runs — the held-back audience is sent only when you declare a winner on the campaign report.

4. Launch the test

Send or schedule your campaign as usual. The test group receives their variants immediately. For automatic tests, the test-duration countdown begins; for manual tests, the campaign waits for you to pick a winner.

Choosing the right settings

Test audience size

  • 10-20% works well for large audiences (10,000+ contacts). You get reliable data while keeping most recipients for the winner.
  • 30-50% is better for smaller audiences. With fewer contacts, you need a larger sample for meaningful results.

Test duration

  • 2-4 hours is usually enough for subject line tests, where most opens happen quickly.
  • 12-24 hours works better for click-based tests, since clicks tend to come in more slowly.
  • For a monthly newsletter, a 4-hour test window is usually enough to see clear differences in open rates.

Setting the test duration too short may lead to unreliable results. If only a handful of people have opened the email, the "winner" might just be random noise. Give recipients enough time to engage. If neither variant has any engagement when the timer fires, Owlat defaults to variant A so the remaining audience is always sent.

Reviewing results

After the evaluation window closes, check your test results:

  1. Go to Campaigns > A/B Results to see outcomes across all tests.
  2. Open the specific campaign report for detailed variant-by-variant metrics.

For each variant you'll see:

  • Sent count
  • Open rate
  • Click rate
  • Winner indicator — which variant was selected

For a manual test, the report shows Choose Variant A and Choose Variant B buttons while the test is still running. Picking one declares the winner and sends it to the remaining audience.

Best practices

  • Start with subject lines — they're the easiest to test and have the biggest impact on open rates.
  • One variable per test — change only one thing between variants for clear, actionable results.
  • Run tests regularly — what works with your audience evolves over time. Regular testing keeps your email strategy sharp.
  • Document your learnings — keep track of what subject line styles, content approaches, or CTAs perform best with your audience.
  • Don't over-test — not every campaign needs an A/B test. Use them for important sends where optimization matters.

Next steps

Next Automations