1. Mailjet
  2. Support
  3. Documentation

A/B Testing

Summary

Introduction

• Have you ever wondered which subject line results in more opens?
• Would a red ‘Get Started’ button produce more clicks than a green one?
• Does having your company name in the sender name really make a difference?

Stop wondering and start getting answers with Mailjet’s A/B Testing!

A/B Testing

What does A/B Testing mean exactly?

Through Mailjet’s A/B Testing (AB Testing), you are free to test almost anything within your email and compare up to 10 versions simultaneously.

Once you decide on what elements (such as subject line, sender name, images, buttons, …) you want to test and how many versions of the email you want to send, you are ready to launch the A/B Test.

You set the testing criteria (open rate, click rate, ...) and then each version is sent to a small but equal number of recipients from the same contact list.

Our powerful analytics tool shows real data in intuitive graphs, telling the story of which version resonated best with your customers.

You can then choose to send the winning version to the remaining recipients on that contact list, or have Mailjet do it for you automatically!

The bottom line with A/B Testing, you benefit from very efficient email testing and together with other features of Mailjet (Segmentation, Campaign Comparison), you will know how to design the most effective email and send it to the most rewarding group of recipients.

Creating Campaign Versions

Using the Mailjet’s A/B Testing feature is quite simple:

  • Design your campaign
  • Create one or more variations of that campaign
  • Define the test criteria and recipient size
  • Review the results

Let’s start by going to your Campaigns page and clicking the ‘New Campaign’ button.

[top]

Step 1: Setup

Enter a name for your campaign and select a contact list. Click the ‘Save and continue’ button.

A/B Testing

[top]

Once you have selected your template, you will enter the email and sender details. Click the ‘Start designing your email’ button.

A/B Testing

Step 2: Design

Customize your campaign by adding new elements with a quick drag-and-drop. Click in any section to edit it. (For detailed information on how to use all the features in Passport, please read our Passport Guide.)

A/B Testing

Excellent! You have created your campaign, also known as Version A.

To create a new version, click on the drop down list, and select 'Create a new version'.

A/B Testing

And Version B has been created.

A/B Testing

Anytime a new version is created, it is an exact copy of the immediate previous. If you were to create Version C, it would be an exact copy of Version B, and so on.

To change the subject and sender details for Version B, click on the Edit icon.

A/B Testing

In this example, we have changed the subject, and the sender name.

A/B Testing

To delete a version, simply click on the drop down list and select ‘Delete this version’.

(When you delete a version, any subsequent versions are renamed accordingly.)

A/B Testing

Be careful on the number of changes you make in a single version. Having too many variables will make it difficult to determine which one variable made the most impact. Remember, you can create up to 10 versions of the same campaign.

Once you are ready to send your campaign, click on ‘Review & Send’.

A/B Testing

[top]

Step 3: A/B Test Sample

Use the slider to select the percentage (or number) of contacts that will receive your campaign versions. The total percentage or number of contacts will be divided by the numbers of versions you have.

In this example, the test will be sent to 40% of the contact list; 162 contacts will receive version A, and 162 different contacts will receive version B. The statistics from this test will be then collected to determine the winning version.

A/B Testing

In the event that the number of campaign versions cannot evenly divide into the number of contacts selected, the remainder will be added to the last version.

Click ‘Continue’ to choose how the winning version will be selected.

[top]

Step 4: Winner Selection

You can now select how the winning version will be sent: Automatically or Manually.

With the Automatically mode, you select the criteria that will be used to determine the winning version at the end of the testing period (defined in the next step).

Once a winning version has been decided, Mailjet will automatically send it to the remainder of the contact list, and you will be notified via an email message.

In the event of a tie, Mailjet will choose the winning campaign with the best global score and you will be notified via email.

A/B Testing

With the Manually mode, you will review the test results to select & send the winning version (within 7 days of the initial sending).

It is required to let the test run at minimum 5 hours before reviewing any results. And the winning version needs to be sent within 7 days of the initial sending. (If after 7 days the winning version is not sent, you will receive notification that your campaign has been cancelled.)

A/B Testing

Choose the desired mode, and then click ‘Continue’.

Step 5: Review & Send

Once you’ve reviewed your A/B campaign details, the last step is choose when the A/B test will be sent to your sample list: immediately or scheduled at a later time.

A/B Testing

Click the ‘Send A/B test now’ button.

A/B Testing

Agree to the confirmation modal and your first A/B campaign is now underway!

If you selected Manual mode, you will receive your reminder to review and analyze the results. If you selected Automatically, then sit back and let Mailjet do the work :)

Accessing your A/B campaign

Regardless if your campaign is ongoing or completed, you can view it just like a regular campaign:

From your Campaigns page, simply click on the ‘A/B test’ tab then on your campaign title.

A/B Testing

[top]

Viewing an ongoing A/B Test Auto Mode

A/B Testing

With an ongoing A/B Test in Auto mode, you can:

  • Change from Automatic to Manual mode, and then you can decide the winning version.
  • Adjust the winning version criteria.
  • Change the date or time when the winning version will be sent.
  • Cancel the A/B Test altogether.

    If no messages have been sent, your campaign will be saved as a draft that you can send at a later date. If messages have already been sent, your campaign will be saved as a normal campaign.
  • Grab a coffee and wait for Mailjet to pick the winner!

[top]

View an ongoing A/B Test Manual

A/B Testing

With an A/B in Manual mode, you can select the winning version at any time: From the drop down list, select the winning version and then click ‘Send This Version’.

A/B Testing

The other options with an ongoing A/B in Manual mode are:

A/B Testing

  • Change from Manual to Automatic mode. You would need to select the winning criteria and the date & time when the winning version will be sent.
  • Change the date or time of your reminder email.
  • Cancel the A/B Test completely.

    If no messages have been sent, your campaign will be saved as a draft that you can send at a later date. If messages have already been sent, your campaign will be saved as a normal campaign.

[top]

View completed A/B Test – Manual or Auto mode

Once the A/B Test has been completed, you will be able to view the powerful data to gain insight to what your customers liked and loved about your different versions.

The graph displays the selected data metric whereas the chart shows the details of all statuses for each version.

A/B Testing

A) Change the display unit, time range and chart type.

B) Select the desired metric to be displayed in the graph. And if you put your mouse over the graph, a popup will show the KPIs for each campaign.

C) Toggle the campaign display on the graph by clicking on the name.

D) Kick up your analyzing power to another level by comparing various campaigns and benchmarks. Please read the Campaign Comparison Guide for more information.

You are now ready to test and optimize your campaigns to your heart’s content… or at least on a regular basis depending how often you communicate with your customers!

As always, our support team is available 24/7 should you have any questions or issues.

Happy A/B Testing!

[top]

Try Mailjet now ?
Create an account - no engagement, no hidden fees