How Long Should You Run Your A/B Test?

What is an A/B test?

An A/B test, or a split test, is a process of utilizing at least two or more versions of a particular variable to determine specific metrics and/or results. For instance, creating two separate landing pages for a website with subtle differences in color scheme and formatting can help gather insight as to how users access, browse, and navigate the web page (on mobile and desktop).

A/B tests can provide a plethora of useful data ranging from engagement levels and CTR (click-through rates) to which emails have generated spam responses or have caused users to unsubscribe. It is even possible to use A/B testing to help determine what type of messaging is most likely to lead to the sale of your product or service.

A few variables that are often considered with A/B testing when it comes to launching a campaign online might include:

Email title

Slightly changing the wording of an email's title can have a significant impact on how many opens an email receives.

Messaging

The way in which a message is delivered and read can also impact how a customer views a business. Some messages may appear direct and persistent, while other messages may seem passive and non-threatening. Knowing your audience can go a long way, which is why A/B tests are so valuable.

Colors

When changing the colors of logos or specific marketing campaigns, you can determine which type of visual messaging works best to elicit a response. Understanding your audience on a visual level is also extremely important, as it can help with devising future marketing campaigns and promotions, digitally as well as locally.

Voice

While some businesses that sell skateboards may do well with a casual voice, other financial institutions typically stick to a corporate and professional tone. Choosing the right voice for your brand matters, especially if you are seeking a genuine connection with your audience.

Why are A/B tests important to conduct?

A/B tests are vital for just about any business model today. The use of A/B tests is not just to find interesting information about a particular audience. In fact, A/B tests can help you to optimize all of your marketing efforts and campaigns, ultimately helping to save money and time.

When A/B tests are conducted properly, it is much easier to gain insights into the demographics of a particular business or brand. Using A/B tests can also inform you of how your target audience interacts and engages with the type of campaigns you create and launch. The use of A/B testing is imperative for those who want to hone in on specific demographics to reduce advertising costs and overall spending.

What is the goal of A/B tests?

The goal of A/B tests is to determine what type of messaging, direction, narrative, and visualization efforts work best to appeal to a particular group or audience. By using multiple variables with A/B tests, it becomes much more clear what an audience desires from a particular brand or business.

Ultimately, because A/B tests are statistical hypothesis testing, A/B testing can be used to achieve numerous goals. From determining what type of voice your target audience is most likely to engage with to understanding the visual styles of art your audience prefers, the options are virtually endless when it comes to launching A/B tests.

Using A/B tests properly can help to reduce costs of marketing and advertising while also assisting you in your efforts to better connect with your audience and prospective customers. With the right A/B tests, maximize your reach and online visibility while also only attracting those who have a genuine interest in what your business or brand has to offer.

How to analyze A/B test results?

Analyzing A/B test results is necessary in order to make any additional decisions regarding your business. In order to begin analyzing A/B test results of your own, keep the following in mind:

Set a time to collect data

Setting a specific time to collect data is a way to launch and track an A/B test. Using a set period of time to track a specific variable with your A/B test will also help to keep an eye on any changes you see during each individual day (based on time zones and online visitor locations).

Consider set goals

What goals did you have in mind for your A/B test before launching it? Were you interested in learning more about the demographics you are targeting, generating sales, or even attracting subscribers to your newsletter? Consider the goals you had set for each individual A/B test you have launched prior to beginning your evaluations.

Evaluate various factors

When evaluating an A/B test, it is advisable to evaluate numerous factors, based on the type of test you are running. If you have changed the messaging and title of an email you have sent, you may wish to compare the open rate, conversion rate, click-through rate (CTR), and even the overall engagement you receive on your website or social media based on the included message.

Use visual representations

The use of visual representations can help you to gauge the type of messaging, formatting, or colors that work best to resonate with your target audience and/or the demographics you intend to reach.

Benefits and drawbacks of split testing?

Using A/B testing is highly recommended for just about any business or brand with an online presence today. The use of split testing is extremely beneficial, although there may be drawbacks at times. Knowing both the benefits and potential drawbacks of A/B testing can help you to prepare for potential challenges that you may face when launching your own A/B tests.

Advantages of A/B Testing

Split testing can help with planning marketing campaigns and connecting with customers, ultimately resulting in sales. Some of the most notable benefits of A/B testing include:

Evidence-backed results

The use of A/B testing provides evidence-backed results. Use collected data to eliminate the guesswork involved in crafting new campaigns and connecting with your target audience.

Gain valuable insights

Discover more about the audience you intend to reach with A/B tests that vary in language, visual styles, and overall message tone. Use variables to further connect with those who want to learn more about your business and brand and what it has to offer.

Optimize your online presence

The use of A/B testing is invaluable as it helps in the process of optimizing your online presence. From launching an official website and updating your landing pages to developing the presence of your social media pages, use A/B testing to better understand the wants and needs of your intended users and/or customers.

Drawbacks of A/B Testing

As with any marketing testing tool, A/B testing has a few drawbacks to consider, such as:

Time

Launching A/B tests can be extremely time-consuming. In some cases, each A/B test is only checked for one variable at a time, which can require plenty of time to conduct and complete.

Resources

Using manual resources is often necessary when launching and managing A/B testing. Typically, at least one individual will be responsible for creating, launching, and monitoring the A/B test results of each individual campaign you launch. If you do not have another employee available to help, you will need to do this yourself.

Requires specific goals

With A/B tests, you will need to define specific goals and variables you are interested in. It is not possible to launch one test to receive all of the information you are looking for in most cases.

Test what you’re trying to convert

Before you set up an A/B test, it’s important to decide the goal—and the intended outcome—of your campaign. There are plenty of possible reasons to choose one winning metric over another, but these 3 scenarios can give you an idea of how to pick a winner based on your goals:

• Drive traffic to your site. Perhaps you run a website or blog that generates revenue by hosting ads. In this type of situation, your winning metric should be clicks.
• Have subscribers read your email. Maybe you’re sending a newsletter that contains ads that pay out by the impression, or you’re simply disseminating information. In those instances, you should use opens to decide the winning email.
• Sell stuff from your connected store. If you’re using email to promote your newest and best-selling products or you’re testing different incentives to encourage shoppers to buy, you should use revenue as the winning metric.

Why does this matter? The table below shows the amount of time you should wait for each testing metric before you’ll be confident in the outcome, based on our research.

You’ll notice the optimal times are quite different for each metric, and we don’t want you to waste your time or choose a winner too soon! Now, let’s dig into the data to take a closer look at how we came up with our suggested wait times—and see why it’s so important to use the right winning metrics.

Clicks and opens don’t equal revenue

Since it takes longer to confidently determine a winner when you’re testing for revenue, you might be tempted to test for opens or clicks as a stand-in for revenue.

Unfortunately, we found that opens and clicks don’t predict revenue any better than a coin flip!

Even if one of the tests clearly emerges with a higher click rate, for example, you are as likely to select the test that generates more revenue as you are the test that generates less revenue, if you choose the winner based on clicks. It’s a similar story when trying to use open rates to predict the best revenue outcome. So, if it’s revenue you’re after, it’s best to take the extra time and test for it.

How long does it take to run A/B test?

We looked at almost 500,000 of our users’ A/B tests that had our recommended 5,000 subscribers per test to determine the best wait time for each winning metric (clicks, opens, and revenue). For each test, we took snapshots at different times and compared the winner at the time of the snapshot with the test’s all-time winner.

For each snapshot, we calculated the percentage of tests that correctly predicted the all-time winner. Here’s how the results shook out.

For opens, we found that wait times of 2 hours correctly predicted the all-time winner more than 80% of the time, and wait times of 12+ hours were correct over 90% of the time.

Clicks with wait times of just 1 hour correctly chose the all-time winner 80% of the time, and wait times of 3+ hours were correct over 90% of the time. Even though clicks happen after opens, using clicks as the winning metric can more quickly home in on the winner.

Revenue takes the longest to determine a winner, which might not be surprising. Opens, of course, happen first. Some of those opens will convert to clicks—and some of the people who click will end up buying. But, it pays to be patient. You’ll need to wait 12 hours to correctly choose the winning campaign 80% of the time. For 90% accuracy, it’s best to let the test run for an entire day.

A quick recap

So, what are the key takeaways from this data? When you’re running A/B tests, it’s important to:

• Pick a winner based on the metric that matches your desired outcome.
• Remember that clicks and opens aren’t a substitute for revenue.
• Be patient. Letting your tests run long enough will help you be more confident that you’re choosing the right winner. We recommend waiting at least 2 hours to determine a winner based on opens, 1 hour to determine a winner based on clicks, and 12 hours to determine a winner based on revenue.

Keep in mind that while this data is a great starting point, our insights are drawn from a large, diverse user base and could differ from the results you’ll see in your own account.

Each list is unique, so set up your own A/B tests and experiment with different metrics and durations to help determine which yields the best (and most accurate) results for your business.

And if the size of your list or segment doesn’t allow for our recommended 5,000 subscribers in each combination, consider testing your entire list and use the campaign results to inform future campaign content decisions.

Learning the A/B testing definition and why A/B testing framework is so invaluable is essential as a business owner today. Whether you are running a local business or if you are launching an international brand online, understanding your audience, as well as their wants and needs, is imperative to increase the overall longevity of your operation.