Experimental subjects
Your subjects are the members of your target audience. Depending on what you're testing, they may be customers, sales prospects, or job applicants. In many marketing experiments, subjects are divided into groups to compare the results of different experimental conditions.
Conditions
The conditions of your experiment are the variables that you change in order to see which one works best. These may include an email's subject line, the location of a search bar on your website, the graphic style of a social media post, or the price of a product.
For example, does offering a one-month free account option to your subscription service increase paid sign-ups after a free trial is over? You can gain valuable insights by comparing the data from two different months in which only that variable is changed.
Although there are many variables you can test, it's usually a good idea to keep it simple and change just one thing in each experiment to make the results more clear. If you run an experiment with more than one variable, it can be hard to know which variable caused the difference in responses.
Effects
The results you get from your experiment are called the effects. Depending on what you're testing, the effects may be sales, newsletter sign-ups, engagement, or customer feedback. In a good marketing experiment, the effects are measurable, allowing you to apply the results of your experiment to drive your marketing decisions.
The benefits of marketing experiments
Marketing experiments can do more than just show you which copy works best in a message. Here are some ways that experimentation can help your organization.
Develop new ideas
There's rarely a bad idea for a marketing experiment because you're gathering data rather than committing to a new, long-term marketing tactic. Sometimes the creative idea for your email marketing campaign ends up yielding much better results than existing ones.
Test new strategies
If you're ready to change up your marketing plan, try out more effective marketing messages, or figure out the best digital channels for your product, marketing experimentation allows you to test the waters before committing fully to something new.
Save time and money
It's always great when you have an idea of what will succeed before you invest your business's valuable time and money. Running experiments gives you some insight into what messages customers respond to or what promotion schedule is most effective. Then you can focus your marketing message in a way that your audience will respond the best to, making the most of your resources.
Optimize marketing campaigns
Even the most well-designed marketing campaigns sometimes fall short of expectations. Maybe newsletter sign-ups are lagging behind projections or your sales conversion rate isn't as high as you would like it to be.
There are many reasons why a campaign might not be working—a marketing experiment is a great way to isolate different factors and determine which changes will improve your marketing metrics most effectively.
If you find that your campaigns generate very few leads, you can test different approaches on a small scale to find something that's more effective. Once you have enough data, you can launch a new approach with confidence.
Types of marketing experiments
Depending on what you want to test, there are different types of experiments you can run. Understanding the benefits of each one will allow you to choose the option that's right for you.
A/B testing
One of the most common kinds of marketing experiments is A/B testing. This kind of experiment allows you to test two (or sometimes more) versions of a single variable. Your target audience receives a random version of your message, with one element changed.
For example, in A/B testing you may send out an email to your subscriber list with the same body copy for everyone, but two different versions of your subject line. Or a link in a social media post may lead people to one of two different landing pages.
It's important to make sure that the material your subjects see is delivered as randomly as possible to avoid accidentally influencing their behavior. It's impossible to control every possible condition, of course, but the more you can randomize who receives your messages the better.
If you show one landing page to the first 100 people who click through your social media post and a different one to the next 100 people, you can't be sure that their responses weren't influenced by something other than the page itself. Perhaps those who view your post in the morning are already more-eager consumers of your brand than those who don't interact with it until the evening. Making the test as random as possible helps to eliminate that influence.
Multivariate testing
Sometimes it makes sense to test more than one variable at a time through multivariate testing rather than A/B testing. This works best when you want to test different variables that go together like updated copy and a fresh visual style to give your brand a more contemporary, casual feel. Testing a major revamp of your website or a new pricing structure might also be good times to consider multivariate testing.
Multivariate testing can help you optimize more than one variable while running fewer tests. However, it often requires a larger sample size and a way to analyze the data that can single out the effects of individual variables in a statistically significant way.
The 5 steps of a great marketing experiment
Running a marketing test should be simple and straightforward if you follow these 5 easy steps.
Step #1: Decide what to test
What do you want to learn from your experiment? There are many elements you can measure. Some ideas include how many people click through to your site from an email newsletter or social media post, how long visitors stay on your website, the percentage of site visitors who make a purchase, or how many new people you reach with online ads.
You may want to analyze historical data for your business to see what has worked well for you in the past. If you find that shorter and more conversational subject lines result in a higher conversion rate for your email marketing campaigns, you might decide to experiment with subject lines that take your casual tone even further.
If your sales tend to drop off in the summer, testing different discount offerings at that time of the year can help you determine the one that boosts your sales the most. Maybe customer feedback indicates that they love the blog content on your website. Trying different post lengths—from short, snappy updates to long-form articles—will give you a sense of how in-depth your audience wants to go.
One tip here is to think about your value proposition—the benefits that you offer to your customers with your product or service. How can the benefits that are part of your value proposition inspire ways to reach out?
Step #2: Make a hypothesis
It may be tempting to run an experiment and just see what happens. But you'll get better results if you think ahead about what you expect to happen (and why) and then compare the results to your hypothesis.
For example, if your historical data shows that discount offers emailed to your most loyal customers result in a low response rate, try to brainstorm some reasons why that may be the case. If those offers are usually sent at the beginning of the week, you may suspect that your customers have busy schedules that distract them. Thus, your hypothesis may be that the later in the week you send the email, the more successful it will be.
Make sure to use a measurable hypothesis so you have data that you can test. After you've run the experiment, you can see which schedule produces the best results and compare it to your hypothesis, giving you valuable insight into hypotheses you can test in future experiments.
Step #3: Design the research
One of the most important things when designing a marketing experiment is to keep it simple. Choose your independent variable—the variable you change—and try to keep the other aspects of your different versions the same.
If you're sending a discount code via email and want to test two different discount amounts to see which one generates the most revenue, make sure to send both versions of the message at the same time to randomly generated groups of your customers.
In addition, think about the length of time the experiment will run. You want to make sure you get enough data to analyze, but you should also have a clear beginning and end to the experiment.
Step #4: Run the experiment
Here's where you get to put your hypothesis to the test! Create and launch your experiment by sending emails, publishing the two different versions of your landing page, or offering different discounts.
Follow your design parameters and collect research data that you'll use in your next step. Make sure to let the experiment run its course and collect all the data you require before analyzing the results.
Step #5: Analyze your results
Before you make any changes, it's important to think about how to measure success. What are you measuring—the number of people who make a purchase? Sign-ups for your newsletter? Click-throughs to your website?
Sometimes the results will be simple to understand: if one call-to-action line generates significantly more click-throughs than another, that one is clearly the winner. But if an experiment uses multiple variables or there are other factors that may affect the results, you may need to use a statistical analysis tool to help you isolate the data you need.
There are numerous software programs and online tools to assist with data collection and statistical analysis. Popular options include Minitab, Segment, and Google Analytics.
Does the data match your hypothesis and expected results? If so, great! Now you know one way you can boost your marketing efforts. If not, that's valuable information, too because it will allow you to change course, run future experiments, or give other options a try before you commit time and resources. Successful experiments and those that don't turn out as predicted both have valuable insight to offer.
What can marketing experimentation test?
There are many things that marketing experimentation can test. Popular options are different methods of communication or the messages themselves. But experiment ideas are endless and with a well-designed experiment you can also test things like communication frequency, product prices and discounts, and the use of social media influencers versus real customers.