10.02.2021

Marketing & Creative

Should you split test or AB test marketing & sales emails?

9 min read

Verity

A/B testing - or split testing - email marketing is one of the few practices marketers employ to improve the performance of their email campaigns. How to split test your emails to improve the response rates of your email marketing strategy.

 

Understanding and utilising concepts like multivariate testing and conversion rates can significantly uplift your campaign's success.

This article explores the nuances of email optimisation through A/B and split testing, emphasising statistical significance and the role of testing tools in refining your marketing strategies.

Understanding the basics: A/B and multivariate testing

Before diving into the intricacies of email optimisation, it's crucial to differentiate between A/B testing (also known as split testing) and multivariate testing.

A/B split testing involves comparing two versions of an email - the original version and a variant, to gauge which one performs better in terms of conversion rates.

Multivariate testing, on the other hand, takes this a notch higher by examining how multiple variables interact and affect user responses.

There is a delicate balance between single-element tests and complex multivariate tests. For instance, changing a single element like button colour might offer clear insights into what works better.

However, when combining multiple variables such as design elements, email subject, and even the layout of action buttons, the analysis becomes more intricate but potentially more revealing, especially when assessing statistical significance.

The methodology also extends to understanding your audience. Through split testing tools, marketers can segment their target audience, tailoring messages to different groups based on their preferences and behaviours.

This segmentation not only enhances the relevance of emails but also helps in fine-tuning the overall marketing efforts.

What is statistical significance?

Statistical significance is a term used in statistical hypothesis testing to help determine if a result is not likely to occur randomly, but instead is likely due to a specific cause. 

In simpler terms, it's a measure of whether the results you are seeing in your data are meaningful or could have happened by chance.

When conducting a test, such as an A/B split testing in marketing, researchers set a significance level (commonly 0.05, or 5%). 

This level represents the probability of observing a result as extreme as, or more extreme than, the observed data, assuming that the null hypothesis (the hypothesis that there is no effect or difference) is true.

If the p-value (the probability of obtaining the observed results, or more extreme, under the null hypothesis) is less than the set significance level, the result is considered statistically significant. 

This means that it is unlikely the observed effect or difference is due to chance, and there is evidence to suggest that a real effect or difference exists.

For example, in the context of email marketing, if you conduct an A/B split test to see which email subject line yields a higher open rate, and the result is statistically significant, it suggests that the difference in open rates between the two versions is likely due to the change in subject lines and not just random chance.

The role of conversion rates in email optimisation

At the heart of email testing lies the goal of conversion rate optimisation.

Whether your focus is on the email subject line, the design of action buttons, or even the button colour, each element plays a critical role in influencing conversion rates.

By methodically testing variations of these elements, you can understand what resonates best with your target audience, ultimately leading to improved marketing campaign effectiveness.

Leveraging testing tools for insightful decisions

The market offers a plethora of testing tools, including specialised split testing tools and comprehensive platforms like Google Analytics or HubSpot.

These tools provide invaluable insights into user behaviour, helping you tailor your emails for maximum impact.

They enable you to track user interactions with different elements of your email, be it the control version or a newly introduced variant, and measure these against your conversion goals.

Analysing results: from data to decisions

The crux of successful email testing lies in analysing the results. Tools like Google Analytics and HubSpot play a pivotal role here.

They help in tracking various success metrics, from basic open rates to more advanced metrics like user engagement levels.

By correlating these metrics with specific changes in the email's design or content, marketers can make data-driven decisions to optimise their campaigns.

Best practices and advanced concepts in email testing

As we venture further into the realm of email optimisation, certain best practices emerge as vital.

Firstly, maintaining a representative sample size in each test ensures that the results are reliable and can be scaled to a larger audience. This is particularly important when dealing with complex multivariate tests, where multiple variables are at play.

Another advanced concept is understanding the sales funnel and how each stage might require different email strategies.

For instance, emails aimed at the top of the funnel should focus more on user engagement and brand awareness, while those at the bottom might need to be optimised for direct conversions, like form completions or ancillary purchases.

The impact of social proof and user experience

In today's digital landscape, incorporating elements of social proof, like customer reviews or social sharing buttons, can greatly enhance the effectiveness of your emails. These elements not only increase trust but also encourage user interaction, contributing to a more engaging customer experience.

Furthermore, optimising emails for different devices, particularly for mobile users, is crucial. This includes considerations for visual elements, button size, and overall layout, ensuring that the user experience is seamless across all devices.

How to conduct a split test on HubSpot?

Hubspot makes it super easy to split-test your regular emails (emails that are sent as a one-off).

To set up your A/B split test:

  1. Prepare for your A/B split test:

    • Determine the goal: Before setting up your test, decide what you want to achieve with your email. This could be increasing engagement, responses, clicks to the website, or driving sales.
    • Select elements to test: Based on your goal, choose elements to experiment with, such as the email subject line, imagery, content, or calls to action.
  2. Create your A/B split test emails in HubSpot:

    • Navigate to Marketing > Email in your HubSpot account.
    • Click "Create email" to make version A of your A/B split test.
    • In the top left of the editor, click to create an A/B test. Enter a name for the variation B email and click OK.
    • Consider varying aspects such as offers (e.g., eBook vs. whitepaper), copy formatting and style, email sender (personal vs. generic address), imagery, subject line length or personalisation, and the entire email layout for comprehensive testing.
  3. Configure test settings:

    • Set A/B distribution using a slider to decide the percentage of contacts that will receive each version. You can opt for a 50/50 split or other distributions.
    • Choose a winning metric (open rate, click rate, or click-through rate) to determine the more successful variation.
    • Set test duration to gather data before deciding on the winner. If results are inconclusive, HubSpot defaults to sending Variation A to the remaining recipients.
  4. Finalise and send the test

    • After configuring the test settings and reviewing your email, click "Save changes."
    • Review your email settings and configure your recipients, then click "Review and send" to initiate the test.
    • Note that you can run a full-split (50/50) A/B test with any sample size. Other sample splits require a list of at least 1,000 recipients, excluding contacts that previously hard-bounced or unsubscribed.


A/B Testing Screenshot

Examples of elements you can test:

Subject line

When you split test email campaigns, the subject line is always a good place to start. You can experiment with capital letters, emojis, questions, and shorter or longer sentences. This helps to establish how your email open rate is affected.

Imagery

Imagery is a subconscious area and the images that you choose have an impact on your readers’ responses. 

Testing different types of images is always a good way to see what content your readers are attracted to and engage with the most. 

Try testing differences such as image size, people vs non-people, bold or pale colour palettes and humour.

Content

Your brand messaging and the information that you share all make an impression on your audience. 

Without wandering too far from your company voice, you can experiment with altering your choice of language, colloquialisms and tone of voice to see if it increases your click-through rates.

Call-to-action

Once a subscriber has opened your email, it’s essential to have an effective call-to-action (CTA) so they take the intended next step. 

Often, people limit CTA usage to simply changing the button or text colour, but there’s a lot more than you can test to improve your click-through rates.

  1. Location: The location of a CTA can make a big difference in click-through rates. If people don't scroll down to the end of your email, they might miss it. So, why not try it out in different places to see what kind of change that makes?
  2. Images: Experiment using image CTA’s instead of just a button with text. Images are likely to attract more people and may encourage them to click through.
  3. Wording: Experiment with changing up the text on your CTAs.BUY NOW may work for some communication, but for prospects who aren’t at that stage of the buyer’s journey, you may have more luck with BROWSE MORE. Your audience can tell you what works best for them, and the stats will show this after you send the email.

How Huble can help you optimise your email campaigns

While the strategies and insights outlined in this article provide a robust foundation for enhancing your email marketing campaigns, the journey towards optimisation is continuous and dynamic.

This is where Huble steps in. As a seasoned marketing consultancy, we specialise in helping businesses like yours navigate the complexities of email campaign optimisation.

Leveraging tactics like split testing, multivariate testing, and a deep understanding of user engagement and conversion metrics, our team is adept at crafting email strategies that resonate with your audience and drive results.

If you're ready to elevate your email campaigns and harness the full potential of targeted, data-driven marketing, we invite you to speak with our team.

Let's explore how we can help you achieve your marketing goals and transform your email initiatives into powerful tools for business growth.

Latest Insights

Service & CX

7 min read

Why you need unified data for effective AI in customer service

Explore how AI for customer service can transform B2B customer retention strategies and drive revenue growth for lasting customer relationships.

Read more

Sales & Revenue

11 min read

The ROI of AI in sales: A numbers-driven look at productivity gains

Explore how AI tools can save sales teams up to 4,000 hours per month, and how AI solutions boost efficiency, quality, and revenue in sales operations.

Read more

Marketing & Creative

7 min read

Transform your business with AI: The SPARK AI Framework from Huble

Explore Huble’s SPARK framework, guiding businesses through AI readiness, strategy, implementation, and continuous improvement for lasting success.

Read more

HubSpot Implementations

18 min read

HubSpot Data Management: 6 strategies for success

This article dives into the central role of data management in maximising CRM potential and best practices for data management in HubSpot.

Read more

HubSpot Implementations

15 min read

How to plan for a successful HubSpot Implementation

Key aspects of planning and managing HubSpot implementation. We'll guide you through every step to ensure a successful HubSpot implementation.

Read more

HubSpot Implementations

13 min read

HubSpot CDP: the future of unified customer data

Unlock the potential of a HubSpot CDP. Discover how to unify customer data for enhanced insights and personalised marketing strategies.

Read more

HubSpot Implementations

12 min read

8 key steps for a successful HubSpot change management strategy

Explore essential steps of HubSpot change management for CRM migration, ensuring a transition that minimises disruption and drives user adoption.

Read more

HubSpot Implementations

10 min read

How to choose a HubSpot partner in Belgium

Explore the key takeaways from DMEXCO 2024, including the importance of customer retention, AI-driven personalisation, and ethical considerations.

Read more

Marketing & Creative

9 min read

8 key lessons from DMEXCO 2024 for Digital Marketers

Explore the key takeaways from DMEXCO 2024, including the importance of customer retention, AI-driven personalisation, and ethical considerations.

Read more

HubSpot Implementations

12 min read

How sales teams can benefit from HubSpot Sales Hub AI tools

Explore the key takeaways from DMEXCO 2024, including the importance of customer retention, AI-driven personalisation, and ethical considerations.

Read more