Skip to main content

Running Marketing Experiments That Work

Learn all about successful marketing experiments, then design and run your own research to generate traffic, boost sales, and grow your business.

There are so many ways to market your business that it's hard to know where to focus your limited time and resources. Should you send out text updates? Can you expand your social media presence? Which headlines are most likely to get customers to read a blog post and generate more sign-ups for your newsletter?

Rather than taking a wild guess and hoping for the best, conduct a few well-designed marketing experiments and then make your next marketing move with confidence.

What are marketing experiments?

Marketing experimentation is a process to test marketing messages and approaches on a small scale to see what generates the best response. A well-designed marketing experiment can validate existing marketing strategies or help you understand where you can refocus your marketing budget.

Even a small increase in your ability to target potential buyers can make a big difference in your bottom line, making marketing experiments important as a way to optimize your reach, sales prospects, and customer follow-through.

Marketing experiments vs. market research

While marketing experiments are more specific, focused efforts to get specific information, market research is a broader area of marketing study in general. It can provide information on a specific market segment, the key metrics that you want to target, and even insights from other industries that may be useful for your own business.

Market research is an opportunity to get information about your market and generate ideas for marketing experiments. It often uses methods like data analysis, surveys, and focus groups to gather and synthesize information.

Most marketers use market research to generate ideas about which areas of their marketing plan could be boosted by more effective marketing campaigns, subject lines, or landing pages. For example, if a market research focus group reveals that customers find your website too dense, you may ask your marketing content team to streamline your copy and measure the responses to see whether the new version is more effective to increase sign-ups.

Marketing experimentation, on the other hand, tests one or more variables for a specific hypothesis. It is designed to generate specific data that you can use to help guide your marketing plan.

The elements of a marketing experiment

Whether you're testing different headlines, social media platforms, or your website's visual style, there are a few elements that all marketing experiments include.

Every marketing experiment is unique, but all of them include some basic components.

Experimental subjects

Your subjects are the members of your target audience. Depending on what you're testing, they may be customers, sales prospects, or job applicants. In many marketing experiments, subjects are divided into groups to compare the results of different experimental conditions.


The conditions of your experiment are the variables that you change in order to see which one works best. These may include an email's subject line, the location of a search bar on your website, the graphic style of a social media post, or the price of a product.

For example, does offering a one-month free account option to your subscription service increase paid sign-ups after a free trial is over? You can gain valuable insights by comparing the data from two different months in which only that variable is changed.

Although there are many variables you can test, it's usually a good idea to keep it simple and change just one thing in each experiment to make the results more clear. If you run an experiment with more than one variable, it can be hard to know which variable caused the difference in responses.


The results you get from your experiment are called the effects. Depending on what you're testing, the effects may be sales, newsletter sign-ups, engagement, or customer feedback. In a good marketing experiment, the effects are measurable, allowing you to apply the results of your experiment to drive your marketing decisions.

The benefits of marketing experiments

Marketing experiments can do more than just show you which copy works best in a message. Here are some ways that experimentation can help your organization.

Develop new ideas

There's rarely a bad idea for a marketing experiment because you're gathering data rather than committing to a new, long-term marketing tactic. Sometimes the creative idea for your email marketing campaign ends up yielding much better results than existing ones.

Test new strategies

If you're ready to change up your marketing plan, try out more effective marketing messages, or figure out the best digital channels for your product, marketing experimentation allows you to test the waters before committing fully to something new.

Save time and money

It's always great when you have an idea of what will succeed before you invest your business's valuable time and money. Running experiments gives you some insight into what messages customers respond to or what promotion schedule is most effective. Then you can focus your marketing message in a way that your audience will respond the best to, making the most of your resources.

Optimize marketing campaigns

Even the most well-designed marketing campaigns sometimes fall short of expectations. Maybe newsletter sign-ups are lagging behind projections or your sales conversion rate isn't as high as you would like it to be.

There are many reasons why a campaign might not be working—a marketing experiment is a great way to isolate different factors and determine which changes will improve your marketing metrics most effectively.

If you find that your campaigns generate very few leads, you can test different approaches on a small scale to find something that's more effective. Once you have enough data, you can launch a new approach with confidence.

Types of marketing experiments

Depending on what you want to test, there are different types of experiments you can run. Understanding the benefits of each one will allow you to choose the option that's right for you.

A/B testing

One of the most common kinds of marketing experiments is A/B testing. This kind of experiment allows you to test two (or sometimes more) versions of a single variable. Your target audience receives a random version of your message, with one element changed.

For example, in A/B testing you may send out an email to your subscriber list with the same body copy for everyone, but two different versions of your subject line. Or a link in a social media post may lead people to one of two different landing pages.

It's important to make sure that the material your subjects see is delivered as randomly as possible to avoid accidentally influencing their behavior. It's impossible to control every possible condition, of course, but the more you can randomize who receives your messages the better.

If you show one landing page to the first 100 people who click through your social media post and a different one to the next 100 people, you can't be sure that their responses weren't influenced by something other than the page itself. Perhaps those who view your post in the morning are already more-eager consumers of your brand than those who don't interact with it until the evening. Making the test as random as possible helps to eliminate that influence.

Multivariate testing

Sometimes it makes sense to test more than one variable at a time through multivariate testing rather than A/B testing. This works best when you want to test different variables that go together like updated copy and a fresh visual style to give your brand a more contemporary, casual feel. Testing a major revamp of your website or a new pricing structure might also be good times to consider multivariate testing.

Multivariate testing can help you optimize more than one variable while running fewer tests. However, it often requires a larger sample size and a way to analyze the data that can single out the effects of individual variables in a statistically significant way.

The 5 steps of a great marketing experiment

Running a marketing test should be simple and straightforward if you follow these 5 easy steps.

Step #1: Decide what to test

What do you want to learn from your experiment? There are many elements you can measure. Some ideas include how many people click through to your site from an email newsletter or social media post, how long visitors stay on your website, the percentage of site visitors who make a purchase, or how many new people you reach with online ads.

You may want to analyze historical data for your business to see what has worked well for you in the past. If you find that shorter and more conversational subject lines result in a higher conversion rate for your email marketing campaigns, you might decide to experiment with subject lines that take your casual tone even further.

If your sales tend to drop off in the summer, testing different discount offerings at that time of the year can help you determine the one that boosts your sales the most. Maybe customer feedback indicates that they love the blog content on your website. Trying different post lengths—from short, snappy updates to long-form articles—will give you a sense of how in-depth your audience wants to go.

One tip here is to think about your value proposition—the benefits that you offer to your customers with your product or service. How can the benefits that are part of your value proposition inspire ways to reach out?

Step #2: Make a hypothesis

It may be tempting to run an experiment and just see what happens. But you'll get better results if you think ahead about what you expect to happen (and why) and then compare the results to your hypothesis.

For example, if your historical data shows that discount offers emailed to your most loyal customers result in a low response rate, try to brainstorm some reasons why that may be the case. If those offers are usually sent at the beginning of the week, you may suspect that your customers have busy schedules that distract them. Thus, your hypothesis may be that the later in the week you send the email, the more successful it will be.

Make sure to use a measurable hypothesis so you have data that you can test. After you've run the experiment, you can see which schedule produces the best results and compare it to your hypothesis, giving you valuable insight into hypotheses you can test in future experiments.

Step #3: Design the research

One of the most important things when designing a marketing experiment is to keep it simple. Choose your independent variable—the variable you change—and try to keep the other aspects of your different versions the same.

If you're sending a discount code via email and want to test two different discount amounts to see which one generates the most revenue, make sure to send both versions of the message at the same time to randomly generated groups of your customers.

In addition, think about the length of time the experiment will run. You want to make sure you get enough data to analyze, but you should also have a clear beginning and end to the experiment.

Step #4: Run the experiment

Here's where you get to put your hypothesis to the test! Create and launch your experiment by sending emails, publishing the two different versions of your landing page, or offering different discounts.

Follow your design parameters and collect research data that you'll use in your next step. Make sure to let the experiment run its course and collect all the data you require before analyzing the results.

Step #5: Analyze your results

Before you make any changes, it's important to think about how to measure success. What are you measuring—the number of people who make a purchase? Sign-ups for your newsletter? Click-throughs to your website?

Sometimes the results will be simple to understand: if one call-to-action line generates significantly more click-throughs than another, that one is clearly the winner. But if an experiment uses multiple variables or there are other factors that may affect the results, you may need to use a statistical analysis tool to help you isolate the data you need.

There are numerous software programs and online tools to assist with data collection and statistical analysis. Popular options include Minitab, Segment, and Google Analytics.

Does the data match your hypothesis and expected results? If so, great! Now you know one way you can boost your marketing efforts. If not, that's valuable information, too because it will allow you to change course, run future experiments, or give other options a try before you commit time and resources. Successful experiments and those that don't turn out as predicted both have valuable insight to offer.

What can marketing experimentation test?

There are many things that marketing experimentation can test. Popular options are different methods of communication or the messages themselves. But experiment ideas are endless and with a well-designed experiment you can also test things like communication frequency, product prices and discounts, and the use of social media influencers versus real customers.

Marketing messages

Any way you deliver a marketing message is an opportunity for an experiment. Digital messages are often easiest to test since you can get real-time data about which ad sales copy brings more people to your site. But no matter how you market your product or service—whether it's paid media, in-person sales, or even skywriting—it's possible to design a marketing experiment for it.

Subject lines

When customers and others on your mailing list get your emails, you want them to open the messages and read the valuable information within. Subject lines can affect the appeal of your email marketing messages and your click-through rates. By testing different subject lines, you can discover which ones resonate better with your audience and optimize your email performance.

Landing pages

A landing page is where someone lands when they follow a link from somewhere else—a social media post, an email, or paid media like a digital ad. By designing two or more versions of a landing page and directing your audience randomly to one of the versions, you can see which one converts more people to registered users or revenue-generating customers.

Search engine results

Making sure that potential customers can find you is one of the most important steps you can take to grow your business. If your business isn't coming up near the top of search results, your digital marketing isn't doing all it could.

There are several factors that determine how well your website performs on a search engine: search engine optimization keywords, title tags, and your site's content. You might want to use one title tag on some product pages and a different one on another random set of pages. Seeing which tag brings more traffic will help you find the most effective way to attract customers.

Social media posts

Social media is a great place to run a marketing experiment. It has become an increasingly large part of many companies' marketing plans.

According to a survey done by the data-driven portfolio website Visual Objects, social media is the most successful digital marketing tool for small businesses. The continuous flow of new content on most social media sites means that there are a lot of opportunities to present fresh information to your target audience.

You can test different copy text, hashtags, or posting times. If your company offers business services, your hypothesis may be that you'll get more sales by offering a discount code on your social media accounts at the end of a quarter, when customers are looking to stretch their budget.

Customer service experiences

Providing great customer service is a factor in customer loyalty. Consider running a marketing experiment on your customer service process. For example, when customers contact you with a question or concern about an order, you might randomize their first contact with either a live agent or a chatbot.

By analyzing how efficiently their issue was handled or how satisfied they were with their experience, you will gain valuable data about what works best.

It may be necessary to rely on customer feedback to determine how successful your experiment is. However, you can get lots of measurable data if you analyze responses by category (positive, neutral, or negative) or ask customers to rank their experience on a numerical scale.

Visual style

Are your customers likely to spend more time on your website if it has a sleek, modern design or if it features colorful, eye-catching graphics? Trying out different visual styles in a small, focused experiment may give you surprising insights about whether your company's visual style could use a refresh or whether you're already on the right track.

Communication frequency

Finding the right frequency for your company's emails, social media posts, and text updates can boost your marketing campaigns and optimize activities like purchasing behavior. You can use marketing tests to experiment with different frequencies.

Does a weekly email about product updates generate more click-throughs per month than a message once a month? Study your existing data and look for opportunities to change how often you get in touch.

Tips for successful marketing experiments

Marketing experiments can be a powerful tool! You probably want to start deploying them immediately to develop new marketing strategies and boost your business growth.

Read on for some things to keep in mind to make sure every experiment is a successful marketing experiment.

Keep it simple

It can be tempting to try to get as much data as possible from an experiment, but trying to test too many variables at once is a bad idea. It can leave you with inconclusive results or sample sizes for each variable that don't provide enough information.

Instead, run experiments that focus on your most important or impactful options and then design another experiment to test another hypothesis.

Get creative

This is your chance to take a chance. Because you're testing multiple options, you're not yet committed to any one choice. If you think potential customers might respond more to humor in your sponsored social media posts, you can place two different versions of your post and measure which one leads to more click-throughs.

Use the right sample size

Deciding how many people you need to reach to test your marketing hypothesis is important. If your sample size is too small, your results might not be accurate. But if the sample size is too large, you might be spending more time and effort than you need to.

The right sample size will be determined by your total number of users, how precise you want the results to be, and how confident you want to be in the results.

Know what you can control

Even if you run a well-designed marketing experiment with an appropriate sample size, a single variable, and a clear hypothesis, it's impossible for it to exist in a vacuum.

For example, if you test an email that asks users to sign up for membership at your outdoor aquatic club on an unseasonably cold day, your results will probably reflect that circumstance, which you can't control.

Tools and resources for your marketing team

Luckily, when it comes to running successful marketing experiments, some of the work has already been done for you. There are several options to help you test different messages and digital channels so you can focus on developing the best ideas for your business and putting the winning growth strategies into practice.


If you're considering running a marketing experiment on your email newsletter or social media, Mailchimp has tools to help you set up, send, and publish your content and then gather the data so you can study the effects and use the results to guide your marketing.


Wynter is a platform focused on testing business-to-business (B2B) messages. Wynter has tools to help you design and run marketing experiments with templates and guidelines to test which subject line, blog post, or paid media content resonates most with your target audience.


You may want to use a service to collect website data you need for an effective marketing experiment. hotjar analyzes and tracks user interaction and experiences with your website and other online products. This can give you valuable insight into which marketing copy, images, or pages are attracting the most interaction and click-throughs.


Userpilot offers tools to help you run marketing experiments and personalize your user interface, focusing on things like user retention and how many users with a free trial become paid subscribers.

When it comes to experiment ideas, the sky is the limit. Whether you use a tool or design your own experiments from scratch, marketing experimentation can help you make smarter business decisions and reach your target audiences more effectively.

Start coming up with new ideas today to reach your customers and optimize your marketing efforts!

Share This Article