In this article, we'll be looking at how to conduct A/B tests using Marketo and Seventh Sense. If you're new to Seventh Sense, it may be helpful to familiarize yourself with the following articles before attempting an A/B test using Seventh Sense and Marketo:
The problem with traditional email A/B testing
Conducting A/B tests to optimize content, subject lines, style of email or other variables is an important part of running a successful email marketing program. Unfortunately, most marketing automation platforms don't give you very much control over picking winners or analyzing the results.
In addition, using traditional blast A/B methods usually results in some, or most emails going out with poor timing since the system has to wait for a certain amount of opens to occur before sending the rest of the emails.
With Seventh Sense, your emails are automatically spread out over time to optimize each person's delivery time. This allows you to conduct A/B tests without compromising on send time or deliverability.
For example, if you start to see one subject line start to really take off (and be sure to check statistical relevance), you can then switch the email that is being sent by modifying your smart campaign mid-delivery.
How to set up an A/B test using Seventh Sense and Marketo
Creating your Source List: The first step to setting up an A/B test is to create your source list like you normally would when sending a batch campaign. If your list is large, you will want to allow some time for it to build before proceeding to the next step.
Splitting your Source List: Once you've verified that your list has completed building in Marketo, navigate to the lists screen in Seventh Sense. Click "Split". This will create a "Cohort A" and "Cohort B" list that are equal, not only in list size but also the amount of engagement and the last period of engagement.
If your list size is large, it can take some time to build your new A and B cohort lists. Once they have finished building, they will appear at the top of your lists and have a "count" next to them. You can track the progress of the split in the spinning wheel on the menu bar of the screen.
Setting up your A/B test in Marketo: Our next step is to create two independent emails that have the different variables you're testing. Once you have one template built, you can simply clone the email and change the variable that you're testing.
When the emails are built, we recommend naming them something similar to:
Next, you'll need to create two Smart Campaigns and two delivery lists, one for Cohort A, and one for Cohort B. You'll set these up just like you normally do when sending a batch campaign from Marketo using Seventh Sense except we will not need to create source lists, because we will be using the two new split source list cohorts we just created in Seventh Sense.
Each smart campaign will have an associated email and an empty delivery list. The smart campaign will have a flow step that sends the email when a person is enrolled in the delivery list.
For clarity, you'll want to make sure your Smart Campaigns and delivery lists are named appropriately. We recommend calling them something similar to:
Scheduling your A/B Test: Once you've set up your two Smart Campaigns, emails, and empty delivery lists in Marketo, navigate to Seventh Sense and click "Personalize" on each of your A and B Cohorts to schedule them to their appropriate Smart Campaigns.
You'll be able to analyze the results in realtime by looking at the open and click rate on your A and B test emails in each Smart Campaign. Once you've identified a statistically significant winner, simply update the flow step in the losing Smart Campaign to send the winning email.