Juliane Heise

// Country Marketing Manager D/A/CH

We talk a lot about how you can optimize your email campaigns for your audience. We hardly do anything else. But without all of this talk, you probably would end up completely clueless…OK, maybe not completely, but you wouldn’t be an email master either.  

If you’re smart, you followed our Mailjet Advent Calendar and read every single email tip. In Tip # 8, we said you should A/B test your campaign to maximize results. Maybe you were asking “How in the world I can do that?” Well, we hear you and help is on the way. We proudly present to you our almighty A/B testing beginner’s guide. Put your last New Year rockets in your closet and get ready for the magic of A/B testing.


A/B testing – the hidden champion you should not ignore

Let’s start with the definition of what A/B testing is. Ok, even we agree that definition is boring but let’s face it: it’s necessary.

A/B testing means the comparison of two or more versions of one piece of content to similar persons at the same time. You can use it to test everything from website copy, to paid search ads and sales and marketing emails. In the context of email, A/B testing means that you create one variation of your campaign to test which version performs best for your audience in terms of open rate and click rate.

Split testing vary from simple to complex testing. Simple A/B testing includes one or two elements which are easy to customize like subject line, button color and size. More advanced testing includes the customization of multiple elements in your email campaign like picture placements, general wording or different email templates against each other.


The benefits of split testing your email campaigns

To get the best results possible, you need to test and analyze all your important email campaigns (marketing email, transactional emails and email automation workflows). While the cost of acquiring new customers and newsletter subscribers can be huge, the cost of increasing your email conversion is minimal.

A well-planned split test increases the effectiveness of your email marketing efforts. By using controlled tests, you will figure out which content and visual arrangements work best for your target groups. If you know what works best, then it is easier to make decisions and creating more effective email campaigns for your audience. Even with small testing and optimization afterwards, the Return On Investment in doing A/B testing can be massive. Split testing allows you to significantly increase the open rate and click rate. The result is a significant increase of leads, sales and revenue.


What you can test

You can test almost everything, isn’t that great? ;) You can test design and text related elements. In fact, to define the best campaign designed for your audience, you need to test both.

Make sure that your audience falls in love at first sight. You can reach this with a killer designed email. However, the design may be the reason they do not delete your email right away, but what really thrives the conversions is the content you provide. In the following section we give you a step-by-step guide showing you on which elements you should focus when you’re starting your A/B testing carrier.

The very first thing you should test is your subject line. The long hard working hours may be for nothing if your audience does not even open your email. So, make sure they do. Besides the fact that the mailing needs to get into the inbox, its headline needs to win them over. You can try out clear messages (“Our special Christmas offer for you”) or subject lines which are more mysterious (“You really do not want to miss this offer”). You can even play around with Emojis.

A/B testing Emoji in subject line

Congratulations, thanks to your fantastic subject line, your audience opened the email. Pictures and other visuals will be the first things that catch the their attention. Images have a great effect when used appropriately. Try different banner, product pictures etc. (even GIFs may work) in different sizes and see what visual style performs best, but make sure the items is aligned with your brand identity.

A/B testing picture sizes

The internet is in love with video, and we assume your audience is too. So, do you include videos in your marketing campaigns? If not, then you really should. It can really make the difference. And if you do so, then try some A/X testing. You can start testing the size and placement.

Sometimes, fancy visuals and funky subject lines aren’t enough to convince your audience and your paragraph texts need to win them over. Try different wording, text length and placement. Focus on the key messages and wrap the other elements around it.

A/B testing paragraph texts
If you are a online shop, then why not test your prices? Try different sizes, colors, placement and even the price itself.

A/B testing prices
When your headlines, sublines and paragraph text are optimized, then your audience is willing to click on your Calls-to-Action (CTA). CTAs buttons are one of the most important elements in a marketing email. This is what it all comes down to. Your mailing is meant to get their attention, but above all, it has to generate leads to your website. So, do not miss this in your testing. Play around with colors, sizes, text, placement etc. Keep in mind that the CTA text also needs to be on point.

A/ testing Call to Action buttons

Besides CTAs, there are other links you can include and test. An example of these links is Social Media buttons. Getting in touch on multiple channels is necessary, even for companies nowadays. We bet that you are on Social Media. Why not include Social Media buttons linked to all your channels and test these if they are attractive enough to click on them? You can test various sizes, pictures and arrangement.

A/B testing Social Media Buttons

Last but not least test the sending time. Does your audience prefer to open their emails in the morning, in the evening, during the week or at the weekend? Use your testing to find out.


Follow the path – don’t worry, you won’t end up in Mordor

The best way to run a successful A/B test is to follow a strict process. It will help you to get profound insights from your testing. A proper process need to include the following steps:

1. Problem identification:

Study your email campaign statistics. Define the user’s behaviour and find the problem areas in your conversion funnel. Include the landing pages your audience reaches after clicking a link in your email.


2. Defining a Hypothesis:

Based on your analysis, build a hypothesis. Define which result you expect from which changes. For example, “My customers do not like to scroll down. Putting the Call-to-Action button at the top will increase their attention and results conversion.”; “Most of my readers open my newsletter on their smartphone. Increasing the size of the CTA button will make it easier to them to click on it which results in more conversions.”

3. Testing the Hypothesis:

Based on your hypothesis, set up the split testing. Create a variation and A/B test it against your current email template.

4. Analysing the test date and draw conclusions:

You successfully sent out your split email campaign to the defined target groups. Now it is time to monitor the results. Which variation performs best? If there is a clear winner, then go ahead with its implementation. If the test remains inconclusive, go back to step number two and rework your hypothesis.



7 magical A/B testing rules you need to follow

To do a proper split test, you need to follow these 7 rules.


Rule 1: Know what and why you want testing your email campaigns

Testing without a specific goal is just wasting time. Don’t run around testing like a headless chicken. Know the reason why you want to use split testing (increase open rates, increase click rates) and think about what changes may get you the desired results.


Rule 2: Focus on frequently sent emails

The moment you start doing A/B testing, you will be on fire and want to test every single email campaign you are sending. But stay calm, take a deep breath and rap on your knuckles. At the beginning only focus on the emails you are sending most frequently.


Rule 3: Split your list randomly

Choose a smaller, randomized portion of your contact list to test for the most optimized email version before sending the campaign to the rest of your contact list. To get conclusive results make sure you choose the same sample sizes.


Rule 4: Test one element at a time

With our fantastic A/X testing tool you can easily split test multiple elements. We know that you are on the starting line – ready to go. But once again stay calm, take a deep breath and rap on your knuckles. Focus on just one element at a time and leave all other variables the same.


Rule 5: Wait the optimum amount of time

The optimum time depends on your email campaign and target group. After sending out your A/B testing mail lean back and wait until you have at least 100 total conversions.


Rule 6: Check if results are statistically significant

The struggle with doing A/B testing is having a large enough sample size. Use a A/B Sample Size Calculator to find the right sample size.


Rule 7: Test again

After the testing comes more testing. Now you know the best subject line, Call-to-Action etc. for your audience. Move over to another element and start testing again. Then repeat the testing until you have tested all relevant elements.  

7 magical A/B testing rules

Now you know everything you need to start A/B testing your marketing campaign: the elements you should focus on, the best practice process and the 7 rules you need to follow. Log in into your Mailjet account and let’s get the split testing party started.


What is your experience with A/B testing? Which elements do you want to test? What improvements do you see? Tell us all about it on Twitter with the Hashtag #emailmarketing.