The below covers a number of questions we often receive from HubSpot users when using the deliverability audit tool. If we haven't covered a question, please do reach out and we'll get it added for everyone.
Do I need to create a separate email for each branch when performing email related tests?
Short answer - Yes, you will need to create a separate email for each branch.
Split test automation has many use cases and assuming you are testing something that is specific to an email test, you need different variations of the email to report on the results of your test. The easiest way to accomplish this is to create an initial version of your email and then clone the email x number of times with each having its own testing variable changed. We do recommend that you name the emails something that is clearly defined for reporting purposes. For example, if you are testing different times of day to see how your emails perform, you may name them something similar to <email name> - Morning, <email name> - Afternoon, <email name> - Evening.
Does split test automation have its own reporting system?
Short answer - No.
Split test automation was not designed to be a reporting tool, but rather a tool that randomly splits an audience into two, three or five cohorts and then assigns them to their respective cohort.
Pro Tip - In order to easily evaluate performance, we recommend using the "Compare email" feature within the HubSpot email dashboard.
Why are the number of contacts assigned to each cohort not completely equal?
Short answer - Split test automation uses a randomization algorithm to assign each contact to a cohort.
Because randomization is used as the algorithm to assign a contact to a cohort, the number of contacts that flow to each cohort will not be exact. This is expected due to the way randomization works. Long and short, it’s not like dealing cards where one contact goes to the A cohort, one goes to the B cohort, etc. Rather it’s like the system is performing a coin flip for each contact to assign them to their respective cohort.
Does split test automation work with “Regular” emails or only “Automated” emails?
Short answer - Your emails do need to be emails that are published for automation.
Because split test automation only works with HubSpot workflows, your emails do need to be published for automation. If you’ve already created the emails as “Regular” emails, you can always convert them in the HubSpot email drafting tool.
Pro Tip - To convert an email, open up the email editor, click on “Actions” in the top right of the window and click on “Convert to automated email”.
Do I need to have actions underneath the “None met” branch similar to the cohort branches?
Short answer - No, however, it is recommended.
While it’s a bit of belt and suspenders approach if there is ever a communication failure between HubSpot and split test automation, any contacts that get caught in the failure will flow down the “None met” branch. While this is highly unlikely, we still recommend doing this.
Pro Tip - To alleviate extra work, you can always use the “Go to action” workflow action step to push anyone who happens to go down the “None met” branch to one of the other branches.
Can I create a second or third group of cohorts underneath each cohort using another split test automation action?
Short answer - Yes.
You can perform as many splits as you’d like under each cohort. While this may get unruly or confusing, one potential use case for this is macro-throttling email sends.
Pro Tip 1- If you deploy a split test automation action underneath another split test automation action's cohorts, be sure to select the corresponding split when creating the value equals branch.
Pro Tip 2- If you're looking for email throttling for HubSpot, use Seventh Sense’s flagship product, email delivery optimization!
Can I change the naming convention for the cohorts that are created with the value equals branch?
Short answer - No.
Unfortunately, HubSpot does not have a mechanism to allow you to change the name of a cohort. These are predefined within Seventh Sense and HubSpot. While we have communicated this feedback to HubSpot's engineering team, it's not possible at this time.
I see some alerts / errors being reported in the HubSpot workflow stating that some deliverability audit workflow actions didn’t execute. Should I be concerned?
Short answer - No.
At times there can be transient network errors in the communication between split test automation and HubSpot. The two systems were designed to handle these types of failures through a timeout mechanism which means if a contact fails at the split test automation step, they will flow to the “None met” branch.
Pro Tip - If you ever see more than a 1% failure rate, please open a support ticket with us at email@example.com.