Create an A/B Test

Each time you create an email, you may wonder whether your content is presented in the best way for your audience. Use an A/B test to compare how small changes can impact your results.

In this article, you'll learn how to create and edit A/B tests.

Before you start

• Be sure you're familiar with how A/B tests work.
• Depending on your plan, you may not have access to A/B tests. To find out what features are included in each plan, check out our pricing page.
• Create a Multivariate test to compare multiple variables in a single campaign. This feature is included with the Standard plan or higher.
• We recommend you send each combination to at least 5,000 subscribed contacts to get the most useful data, but smaller tests can still help you learn about your audience’s preferences.
• Subscribed contacts may not interact with your email right away, so test data may take some time to populate. We recommend you wait at least 4 hours after you send an A/B test to send the winning combination. Winner notification emails are sent to any user on your account with a user level of Manager or higher.
• When you test content as the variable in an A/B test, choose the winner by click rate, as opposed to open rate, since opens occur before a subscribed contact views your content.
• If you change your mind during the test phase, you can convert to manual selection to choose the winner from the report.

Create an A/B test

A/B tests are built in a series of steps. First you'll create and name your test, then you'll choose your recipients, decide which variable to test, choose your setup, and add your content. Last, you'll confirm that everything looks good and send.

To create a new A/B test, follow these steps.

1. Click the Create icon, then click Email.
2. Click A/B test.
   
   If you see the Multivariate option instead, your plan includes advanced testing options. To learn more, check out Create a Multivariate Test.
3. Enter a campaign name and click Begin.

Recipients step

On the Recipients step of the builder, choose the audience or segment you want to send your campaign to. You can configure a new segment on this screen, or choose a saved segment.

Variables step

On the Variables step, you'll choose whether to test your subject line, From name, content, or send time.

To choose the variable you want to test, follow these steps.

1. On the Variables step, click the variable you want to test.
   
2. Click the plus or minus icons to add or remove variations of the variable. Test 1 variable with up to 3 variations.
   
   Each variation will generate a combination as shown in the Summary section. For example, to test 3 subject lines, we'll create 3 combinations of your email.
   
3. In the What percentage of your recipients should receive your test combinations?, click and drag the slider to set the percentage of your recipients you want to receive your test combinations.
   
   Review the number of recipients who will receive each combination in the Summary section. We require that you send to at least 10% of your recipients for any type of A/B test, so the slider will not go below 10%. If you're testing Send time, you'll always send to 100% of the subscribed contacts in your chosen audience or segment.
4. Choose whether to determine a winner based on open rate, click rate, total revenue, or manually. Then, choose how long you want the test to run.
   
   At the end of the test phase, we'll send the winning combination to your remaining recipients. If you choose to send test combinations to 100% of your recipients, you won't see this option.

Setup step

Depending on which variable you test, the Setup step will present multiple Email subject, From email address and name, and Subject line fields.

To choose your setup, follow these steps.

1. Type your Email subject, From email address, and From name in the fields provided. Send time variation settings also appear on this step.
   
2. Check the boxes next to your preferred settings or tracking options to include. Track opens and Track clicks can't be disabled, since they're used to determine the winner of the test.
   

Content step

The Content step is where you add content for your test. What you see on this step will vary if you choose to test content.

Test subject line, From name, or send time

To add content when your variable is subject line, From name, or send time, follow these steps.

1. On the Template step of the content builder, choose your layout.
2. On the Design step of the builder, design and test your content.
3. When everything is how you want, click Confirm Before Sending.

Test content

If your variable is content, the Content step shows a preview for each variation.

To add content to each variation, follow these steps.

1. Click Add Content for a variation of your email.
2. On the Template step of the content builder, choose your layout.
3. On the Design step of the content builder, design and test your content. When everything is how you want, click Next.
4. On the Description step of the Content Builder, type in text to describe the variation of the content. Your subscribed contacts won't see this description, but it will help you remember the differences between each variation.
   
5. Click Continue to content to return to your content variations.
6. Repeat steps 1-5 for each content variation. On the Template step, use a different template or select a variation you already designed in the Previous Variations tab.
   
7. When everything is how you want, click Next.

Confirm step

The Confirm step is where you'll review the combinations that make up your A/B test and send them when ready.

To confirm and send your A/B test, follow these steps.

1. Review the pre-delivery checklist. Click Resolve or Edit to make changes.
2. After you're finished, click Send to send your test now, or click Schedule to schedule it for a later date.
   

Account users with a Manager user level or higher will receive an email when a winner has been automatically chosen, or after 4 hours if you haven't yet picked a winner manually.

Manually choose a winner

If you chose Manual Selection from the drop-down menu on the Variables step of the builder, you'll be able to view the report after tests are sent, and choose a winner.

To manually choose a winner, follow these steps.

1. Click Analytics, then click Reports.
2. Click View report for the A/B test you want to work with.
3. An alert will tell you how long into the testing phase you are and recommend that you wait at least 4 hours.
   
4. Review the report for each combination. Under Finalize your test, click the radio button next to the combination you want to send to your remaining recipients.
   
5. Click Send To Remaining Recipients.
6. In the confirmation pop-up modal, click Send Now.

Convert to manual selection

After you send your test with the choice to have the winner chosen by open rate, click rate, or total revenue, you may decide that you want to manually choose the winning combination instead.

To convert to manual selection after you send your test, follow these steps.

1. Click Analytics, then click Reports.
2. Click View Report for the A/B test you want to work with.
3. Click Convert to manual selection.
   
4. In the pop-up modal, type CONVERT in all caps and click Convert.
   
5. When you're ready, choose a combination to immediately send to your remaining recipients.