en DE RU
Request Demo
Bloomreach Acquires Exponea. Learn more

Table of Contents
  1. Obstacles to Testing
Listen this article on:

Email A/B testing – The quickest method to increasing your email performance and revenue

Mar 17, 2020 Gavin Sherry 8 min read

You’ve got a great idea for an email campaign. You’re sure the content is perfect, the product is irresistible. You’re ready to launch it. 

You just need a subject line. You’ve got two ideas. Which should you use?

Most marketers already know the answer: you should test both of them. Create an audience, then split it in two. Half the audience gets one subject line, the other half gets a different one. Let your campaign run for some time, then check your results (those could be open rates, or revenue, click-through rates, or something else).

That procedure is an A/B test.

Obstacles to Testing

All email marketers know the benefits of email testing. Yet so few actually implement testing.  Why not? It usually comes down to these obstacles:

1.    Don’t have the ability to test

2.    Can’t afford the time to test

3.    Don’t know what to test

We’ll address all three here.

1. Ability to A/B Test

Regarding the first point, Exponea has introduced a super easy testing mechanism which allows you to test an email test concept (the challenger) versus a control. The most straightforward example is testing one subject line against another. Exponea’s clever functionality allows you to set up the test variants and run a live test to a percentage of your list before deploying the winning campaign to the rest of your audience (for a more in-depth look at this feature, download our 1-pager).  It’s that easy. Although we call it A/B testing, it could in fact be A/B/C/D/E/F/G testing, as you can run multiple test options and the functionality will still deploy the most successful one to the remaining hold out group.

You can see how easy this is to set up in Exponea in the recordings below.

How to do it in a Scenario (which can be used for omnichannel campaigns)

How to do it in a dedicated Email-only campaign

2. Time to A/B Test

Let’s come to the second point: “Can’t afford the time.”  The fact is that to not test is to walk away from additional revenue (that’s ultimately what we want the email to drive).  Here’s the visual impact of what testing can do. It starts with the funnel activity which is email marketing.

This looks like a familiar enough process.  However, the benefit of testing is that even if we increase activity further up the funnel and do nothing else, we still make more revenue.

Here’s the starting point before testing.

Now what happens if we focus on increasing just our open rates?

Just by getting more people to open the email, we see an impact on what happens further down the funnel.  Keep an eye on those conversions!

Next, let’s see how it looks when we implement a testing plan to increase both open and clicks.

The numbers build even more momentum and the conversions are significantly up again!

Finally, after implementing a comprehensive testing plan, where we optimize opens, clicks and conversions.

From no testing to implementing a comprehensive testing plan, we have increased conversions by 80%!!!  YOU ARE AN EMAIL HERO!

Of course, you won’t get there overnight, and we recommend you focus on just one testing concept at a time.  So, where to begin? The starting point needs to be a goal or an objective, like “I want to increase my open rates from 20% to 25%”.

Further Reading: We’ve put together a comprehensive guide on email deliverability to help you get your emails to the inbox.

3. What to A/B Test

The factors that have an impact on open rates are quite limited, and so they are a good place to start.  In essence, there are just 4 things to test with regards to open rates:

  • Subject lines
  • Pre-headers
  • Time of sending
  • Sender name

That’s it.  There’s nothing else that you can test with regards to open rates.  So, let’s jump right in.

Subject lines:

A good subject line can increase opens by as much as 15% to 30%. The role of the subject line is crucial. It’s about grabbing recipients’ attention in a very busy inbox with many other brands all crying out “Open me.” For this reason, subject lines are not subtle. In addition to grabbing attention, it needs to communicate the number 1, stand out reason, why your customers should open this email.


Because we all mostly view emails on our mobile devices, the pre-header has become a very important part of email anatomy. In our inbox view, it sits under the subject line and acts almost like a secondary subject line. It can have 2 distinct purposes. 

1.  It can boost the impact of the subject line with an additional supporting message.

Subject Line: “Footwear lovers, get ready”

Pre-header: “We’ve added over 100 gorgeous new shoes!” 

2.  It can add a secondary reason to open the email.

Subject Line: “Footwear lovers, get ready”

Pre-header: “Plus, check out these statement dresses!”

If you have more than one content piece in the email, the second option can capture the interest of more recipients (which will drive more opens).

Time of Sending:

Identifying the best day and time to send emails can significantly boost your open rates.  When tested it can increase open rates by up to 20%. If you’re sending at 7AM each morning, so are many other brands.  What happens when we send at lunch time, or after the working day? For each brand the ideal send time is going to vary.  However, it’s a very impactful element to test.

With Exponea, you have the option to personalise email send times for every individual customer with just one click. You can optimize predictions based on open-rates or click-through rates according to your goals.

The amount of messages that you are sending to customers is also important.  In Exponea you can easily set up frequency management policies that determine how many messages a customer from a particular segment can get, and by doing so you can avoid spamming and overwhelming your customers.

Sender Name:

Slight amends to the sender name (and not the email address) can be very significant.  The sender name is your normal brand name. When tested, however, certain variants of the brand name can drive up open rates.  We need to make sure we don’t confuse recipients, but variants of the sender name worth testing include:

“Brand X – New in”

“Brand X – Exclusives”

“Brand X – The Big Sale”

“Brand X – The Style Edit”

You get the idea.  It’s definitely worth testing.

Beyond optimizing open rates, a good testing plan will also move towards the content of the email.  Here there are many more factors to test. However, it is possible to increase click activity by 25% when you learn how to present your content in email.  Some of the most impactful tests are:

  •   Buttons (CTAs) versus no buttons
  •   Links versus buttons
  •   “Shop new” versus other CTA vocabulary
  •   Price versus no price
  •   Stylized banners versus product photography
  •   Images with supporting text versus image only
  •   More products versus less products
  •   Recommendations versus no recommendations
  •   Using abandon cart items in your newsletter versus not using them
  •   Browsed item category “New in” versus no browsed item category content
  •   Clearance items versus no clearance items (people still love bargains!)

There are more areas to test.  However, these are definitely some of the most impactful ones.

Finally, we arrive at factors that might have an impact on conversion rates.  Strictly speaking, these sit outside of the remit of the email. If the recipient has clicked a link, the email has done its job.  However, it’s most definitely worth testing the following:

  •   Landing on the product page versus landing on the category page (please note the actual product clicked needs to be at the top left of the category view page)
  •   Adding delivery options on the product pages versus not adding delivery options on the product pages
  •   Category pages sorted by new in
  •   Category pages sorted by best sellers

Hopefully, the funnel analysis above confirms how implementing a testing plan can be a great boost to email performance and revenue.  However, before you go, please review some best practices for A/B testing.

A/B Testing Best Practices

  •   Make sure you’re following best practices regarding email deliverability. There won’t be much to test if your emails don’t make it to the inbox.
  •   Test one email element at a time.  To test subject lines AND the use of pre-headers at the same time is not going to inform you as to which test element had the most impact.
  •   However, it is possible to come up with several testing concepts for one test element.  E.g. with subject lines, you can test several at once. Or CTA language, again you can test several in one test.
  •   You’ve struck gold and you’ve found a test element that really works.  Great. Typically, successful content tests allow for the broad roll-out of that winning tactic.  However, be careful with winning subject lines, pre-headers and even sender names. Overuse of these tends to erode performance over time.
  •   When setting up your A/B testing in Exponea, you need to set the functionality to focus on one metric only in order to identify the most successful campaign. So, for subject lines, that needs to be open rates. For email content ideas, that is most likely to be click through rate.  However, it could also be conversion rates. When testing price shown versus price not shown, the price shown generated less clicks but higher conversions. However, testing results will vary for each brand. Test for your own results to ensure the best outcome.
  •   You will need to allow a window of time to allow the test recipients to open and engage in the email before deploying the most successful campaign to the hold out group.  The longer the window of time, the better the odds are that the campaign deployed is actually the best performing one. Our recommendation is a 4 hour wait.
  •   To gain confidence in testing and the benefits it can bring, start with simple tests before doing more ambitious ones.  The tests for opens are a great place to start.
  •   Plan your content tests in advance in order for your content team to arrive at the test variants you want to run.
  •   Testing is very impactful to email performance and revenue.  But not only is it easy within Exponea, it’s fun. So, start planning and get testing.

Going Beyond Standard A/B Testing

A/B testing isn’t going anywhere, but it is evolving. Exponea offers a new feature that’s a revolutionary change to the old standard.

Predictive Bandits

Predictive Bandits are a big upgrade to A/B testing (and in some ways, an alternative to it). In regular A/B testing, once you’ve determined a winner, you send your traffic there…and discard the losing variant.

But what if the losing variant was actually a winning variant for some (smaller) portion of your audience? You don’t need to get rid of the losing variant. You just need to find the right audience for it.

Predictive Bandits do this for you automatically. They determine which variant is right for each customer, based on what’s known about the customer as well as contextual clues. This can give you a serious boost over standard A/B testing.

Predictive Bandits are only possible with a platform that can build a complete picture of every customer, and then engage with them on the proper channel. Exponea can do exactly that. If you want to see how tools like Predictive Bandits can help your business, request a demo today. We’d be happy to speak with you.

meet the author
Gavin Sherry
Senior Deliverability & Strategic Consultant at Exponea
With over 20 years of experience in strategic marketing and e-commerce solutions, Gavin now serves as Exponea’s Senior Deliverability & Strategic Consultant, leveraging his expertise in digital marketing planning & implementation, B2B brand management, and email marketing. Specialties include: retail, travel, hospitality, financial services and IT.

Watch Exponea demo video!
Explore the Customer Data & Experience Platform B2C Leaders Love to Use

MISSGUIDED Victoria Beckham Desigual
ebuyer River Island

We rely on cookies

to optimize our communication and to enhance your customer experience. By clicking on the Accept and Close button, you agree to the collection of cookies. You can also adjust your preferences by clicking on Manage Preferences. For more information please see our Privacy policy.

Manage cookies
Accept & close

Cookies preferences

Accept & close
We use cookies to optimize our communication and to enhance your customer experience. We also share information about how you use our website with our third parties including social plugins and analytics. You consent to our use of cookies if you continue to browse our website. You can opt out of our cookie use on the Do not Sell my Personal Information page. For more information please see our Privacy Policy.