How to Do A/B Testing: 15 Steps for the Perfect Split Test

Summary

This video script offers an insightful guide on A/B Testing, a powerful tool for businesses to optimize content and enhance key performance indicators such as engagement and sales. It explains the process, from choosing test items and determining sample size to analyzing data for statistically significant results. The script highlights common mistakes to avoid and emphasizes the importance of testing single elements for reliable outcomes. With examples and a mention of HubSpot's free A/B Testing Kit, it provides a practical approach to making data-driven decisions to boost business performance.

Takeaways

• 🔍 A/B Testing is a method to compare two versions of an element to determine which performs better for business goals.
• 📈 It helps in optimizing content to increase engagement, sales, clickthrough rates, and other key performance indicators.
• 🧪 Think of A/B Testing as a marketing experiment where you split the audience to test different versions of the same element.
• 🎯 Common goals for A/B Testing include increasing website traffic from emails, improving conversion rates, and reducing bounce rates.
• 🛠️ HubSpot offers a free A/B Testing Kit with guides and templates to assist in the testing process.
• 📝 The first step in A/B Testing is choosing the appropriate test items that will impact the identified goal.
• 📉 Determining a sufficient sample size is crucial to ensure the results are not skewed and are statistically significant.
• 🔄 It's important to test only one element at a time to avoid unreliable results and to identify the exact impact of changes.
• 📊 Analyzing the data after the test helps in planning and making informed changes for future improvements.
• ❗ Common mistakes in A/B Testing include testing more than one variable at once, having a small sample size, and making changes before the test is over.
• 📈 A/B Testing is accessible to businesses of all sizes and offers a data-supported way to find low-cost methods for growth.

Q & A

What is A/B Testing?

A/B Testing is a marketing experiment where you split your audience to test two different versions of the same element, such as an email subject line, website font, or call to action placement, to determine which version performs better.

How does A/B Testing benefit a business?

A/B Testing benefits a business by helping to better understand customer habits and behaviors, allowing for content optimization that can increase engagement, sales, clickthrough rates, and other key performance indicators.

What is the purpose of testing different elements in A/B Testing?

The purpose is to determine which version of the tested element performs better in achieving the set goals, such as increasing website traffic, conversion rates, or reducing bounce rates.

What is the importance of sample size in A/B Testing?

Sample size is crucial in A/B Testing to ensure the results are not skewed and are statistically significant. A larger sample size provides more reliable data to make informed decisions.

Why is it important to test only one element at a time in A/B Testing?

Testing only one element at a time ensures that the results are reliable and that you can accurately determine which specific element impacted the outcome. Testing multiple elements can yield unclear results.

What is the significance of statistical significance in A/B Testing?

Statistical significance, typically aimed at 90% or higher, indicates that the results have a definitive winner and did not occur by chance, providing confidence in the decision-making process.

How does A/B Testing apply to an E-commerce business?

For E-commerce businesses, A/B Testing can help find what kind of product images attract customers or what checkout designs reduce cart abandonment rates, ultimately optimizing the user experience for increased sales.

What is the first step in approaching an A/B Test according to the script?

The first step is choosing the appropriate test items by listing out elements that will impact the identified goal, such as testing different versions of a landing page for conversion rates.

What is the role of the HubSpot A/B Testing Kit in the A/B Testing process?

The HubSpot A/B Testing Kit provides a how-to guide and templates for conducting A/B Tests, including tools for calculating sample size and determining statistical significance.

What are some common mistakes to avoid when conducting A/B Tests?

Common mistakes include testing more than one variable at a time, only testing minor changes, having too small of a sample size, making changes before the test is over, and only running a test once without replication.

How can A/B Testing help in optimizing a business's email strategy?

A/B Testing can help optimize email strategies by testing different elements such as subject lines, sender names, email formats, layout, and timing to determine what increases open rates and clickthrough rates.

What is the recommended approach for analyzing A/B Test results?

The recommended approach is to focus on the goal metric, use tools like the HubSpot A/B testing calculator to determine statistical significance, and make decisions based on clear winners.

Outlines

00:00

🔍 Introduction to A/B Testing and Its Benefits

The first paragraph introduces the concept of A/B testing, a method used to understand customer behavior and optimize content for better engagement and sales. It explains A/B testing as a marketing experiment that involves comparing two different versions of the same element to determine which performs better. The speaker outlines common goals for A/B testing, such as increasing website traffic from emails or improving conversion rates, and highlights the importance of testing elements like email subject lines, website fonts, and call-to-action placements. The paragraph also mentions the value of A/B testing for E-commerce businesses and introduces HubSpot's free A/B Testing Kit as a resource for guides and templates.

05:01

📈 Steps and Considerations for Conducting A/B Tests

The second paragraph delves into the process of setting up an A/B test. It begins by emphasizing the importance of choosing the right test items and determining a sufficient sample size to avoid skewed results. The speaker discusses the necessity of ensuring statistical significance in the test results, aiming for at least 90% to confirm a definitive winner. The paragraph also touches on the importance of keeping control variables constant during the test and choosing an appropriate timeframe to avoid seasonal biases. It advises against testing multiple elements simultaneously to prevent unreliable results and concludes with the importance of analyzing data to inform future strategies. The speaker also provides a brief overview of common elements to test, such as Call-to-Actions, emails, and landing pages, and how to approach testing different aspects of these elements.

Keywords

💡A/B Testing

A/B Testing is a method of comparing two versions of a webpage, email, or other content to determine which performs better. It is central to the video's theme, as it is presented as a tool for optimizing business strategies. The script describes it as a 'marketing experiment' where an audience is split to test two different versions of the same element, such as a website landing page or email subject line, to see which one yields better results like higher engagement or sales.

💡Key Performance Indicators (KPIs)

KPIs are metrics used to evaluate the success of a business or specific initiatives. In the context of the video, KPIs like engagement, sales, and clickthrough rate are optimized through A/B Testing. The script mentions that A/B Testing helps to increase these KPIs by understanding customer behavior and making data-driven decisions.

💡Call-to-Action (CTA)

A CTA is a prompt designed to inspire action from the audience, such as making a purchase or signing up for a newsletter. The video discusses CTAs as a common element for A/B Testing, with variations in placement, size, color, copy, and graphics being tested to determine what drives the most user engagement.

💡Sample Size

Sample size refers to the number of observations or subjects in a study that is used to make estimates or inferences about a population. The script emphasizes the importance of having a sufficiently large sample size in A/B Testing to ensure that the results are statistically significant and not skewed.

💡Statistical Significance

Statistical significance in A/B Testing means that the results of the test are unlikely to have occurred by chance. The video aims for a 90% statistical significance, indicating that there is a 9 out of 10 chance that the observed results are a true reflection of the population and not due to random variation.

💡Landing Page

A landing page is a standalone web page, designed to promote a specific action or offer. In the script, the speaker uses a landing page as an example for A/B Testing, comparing a version with multiple product images (Version A) against the existing page with a single product image (Version B) to determine which converts better.

💡Conversion Rate

The conversion rate is the percentage of visitors to a website who take a desired action, such as making a purchase. The video's example of A/B Testing a landing page is aimed at increasing the conversion rate of website visitors into paying customers.

💡Bounce Rate

Bounce rate is the percentage of visitors who leave a website after viewing only one page. The script mentions testing different design features to lower the bounce rate, indicating that A/B Testing can help identify elements that keep visitors on the site longer.

💡E-commerce

E-commerce refers to the buying and selling of goods or services using the internet, as well as the transfer of money and data to execute these transactions. The video uses an e-commerce business as an example to illustrate how A/B Testing can help find optimal product images and checkout designs to reduce cart abandonment.

💡HubSpot

HubSpot is a CRM platform that provides tools for marketing, sales, and customer service. The video mentions HubSpot's free A/B Testing Kit, which includes guides and templates for conducting A/B tests, and the HubSpot CMS, which facilitates the setup and analysis of A/B tests.

💡Optimizely

Optimizely is a platform for A/B Testing, personalization, and experimentation. The script refers to Optimizely's sample size calculator as a tool to help determine the appropriate sample size for A/B Testing, which is crucial for obtaining reliable and statistically significant results.

Highlights

A/B testing is a simple tool to boost business by understanding customer habits and behaviors.

A/B testing helps optimize content to increase engagement, sales, and key performance indicators.

It involves marketing experiments by testing two different versions of the same element to determine which performs better.

A/B testing can be applied to email subject lines, website fonts, and call to action placements.

Gathering data through A/B testing allows for better decision-making to positively impact business goals.

A/B testing can increase website traffic from emails by testing different headlines.

Testing CTA placement can lead to a higher conversion rate from website visitors to sales leads.

A/B testing can identify design features that lower bounce rates and drive traffic away from sites.

For E-commerce, A/B testing can find optimal product images and checkout designs to reduce cart abandonment.

The approach to A/B testing includes choosing test items, determining sample size, and ensuring statistical significance.

HubSpot's free A/B Testing Kit provides guides and templates for conducting A/B tests.

Common elements to A/B test include Call-to-Actions, emails, and landing pages.

Testing different parts of a CTA, such as color, size, and placement, can optimize user engagement.

A/B testing emails can involve format, layout, timing, subject lines, and sender names.

For a candle business, A/B testing a new store landing page can determine the impact of product images on conversions.

When setting up an A/B test, it's crucial to test only one element at a time for reliable results.

Analyzing A/B test data helps plan and make future changes based on statistically significant results.

Common mistakes in A/B testing include testing multiple variables at once and having an insufficient sample size.

A/B testing should be replicated with the same parameters to ensure consistent and reliable results.

A/B testing is accessible to businesses of any size and offers a data-supported way to grow and optimize operations.

Transcripts

- I'm gonna show you exactly how A/B Testing works,

why it can benefit your business?

And all of the common mistakes to avoid.

But what exactly is A/B Testing?

(screen whooshes) Okay.

So A/B testing is an invaluable yet simple tool

that can boost your business

by helping you to better understand

your customer's habits and behaviors.

It allows you to optimize your content

in a way that'll increase engagement,

sales, clickthrough rate, (text tinkling)

or any number of other key performance indicators.

Think of it like a marketing experiment.

You split your audience

to test two different versions of the same thing.

It can be anything from a simple email subject line,

to the font on a website landing page,

or the placement of a call to action.

The goal is to determine which one performs better.

That's it.

Gathering data like this

will allow you to make the best decisions moving forward

to positively impact your goals.

Some A/B Testing goals could include

increasing website traffic from emails

where you test different headlines

to catch the reader's attention

and clickthrough to your website.

Or maybe you're looking for a higher conversion rate

from website visitors to a sales lead.

Testing the placement of your CTA

to submit their contact information.

It can help you test what's driving traffic

away from your site by testing different fonts

or design features

just to see what helps lower your bounce rate.

If you're an E-commerce business like me,

A/B Testing can be a great asset

for finding what kind of product images

best catch your customer's attention,

or what kind of checkout designs

lower your rate of cart abandonment.

Okay, so this all makes sense,

but where do I even start? (screen whooshes)

Here's a quick summary of how I approach an A/B Test.

Oh, by the way,

everything I'm gonna be talking about

is available in HubSpot's free A/B Testing Kit.

That includes a how-to guide

and templates available for download

if you wanna follow along.

The first step is choosing the appropriate test items

by listing out elements that will impact the goal

that I've identified.

For me, I'm trying to determine if a landing page

with more product images on it

will better convert website visitors into paying customers.

So I'm gonna test two different versions of my landing page.

Here's version A,

a brand new design with multiple product images.

And version B is going to be my existing landing page

with only one product image.

Next, I have to determine my sample size.

I wanna make sure the sample size isn't too small

because it could skew the results.

Tools like Optimizely sample size calculator

can even help determine what size is best for your business.

Just like determining sample size,

I also need to be able to verify my data

by making sure it's statistically significant.

That means your results actually have a definitive winner

that did not occur by chance.

Typically, we're aiming for 90% statistical significance.

To help increase statistical significance

while testing variables,

keeping my controls the same is crucial,

including when and for how long I run the test.

So I'm gonna choose a timeframe

that represents an average time for my sales,

instead of running the test

during the holiday shopping season, for example.

There are lots of things you can run A/B Tests on

so it's important that I only test one element at a time.

If you do more than one, it'll yield unreliable results

because you won't be able to determine

exactly which element impacted those results.

Finally, analyzing the data is gonna help me plan

and make changes to how I do things in the future.

So what are some of the most common elements

you should consider when A/B Testing?

(screen whooshes)

The most common things you'll A/B Test

include Call-to-Actions, CTAs for short.

Emails and landing pages,

which is what I'm focusing on today.

If you're looking to A/B Test a particular CTA,

like a contact form to generate leads

or a subscription link,

things like placement size, color, copy

and graphics are all prime elements for testing.

Should you use a bold color

or something that seamlessly blends into the page.

Should it be positioned at the top of the page

or as a sidebar?

Testing different parts of the CTA one at a time

could help you find what's optimal for your users.

If it's emails you're testing,

a lot of the same elements could apply

in addition to format, layout,

timing, subject lines and even senders.

Is your open rate better when the email comes

with the name of a personal sender instead of a company?

Do your email subscribers tend to open more emails

with emojis in their subject lines?

Or just simple text?

A/B Testing can help you pinpoint

the sweet spot you're looking for

to increase whatever metric you're measuring.

For my candle business,

I'm focusing on A/B testing a new store landing page.

Again, I'm trying to determine

if a landing page with more product images on it,

will better convert website visitors into customers.

The elements I could test include

offers, copy, form fields, (text popping)

or even just the entire page,

like the color, the layout, and so on.

So what I'm gonna do here

is run two different versions of my landing page

to see which, if any, will result in more sales.

So what are the steps

to getting my actual test up and running?

(screen whooshes)

Okay, so I've already completed steps one and two.

I picked my variable, and I identified my goal.

Now I've gotta create my control and my challenger.

Basically, my challenger is the new landing page

in which I've added more product images.

My control is a landing page as it currently exists.

Now, I'll need to split my sample groups equally

and randomly, as well as determine my sample size.

If this was an email,

it'd be a finite number of emails I'm sending.

But since this is a landing page,

the length of my test will determine my sample size.

It'll vary for each business

based on the kind of traffic your website gets.

I've gotta be sure that I let the test run long enough

to gather a substantial sample size.

Otherwise, it'll be hard to confirm

the statistical significance.

Next, I have to decide how significant my results need to be

to declare a true winner.

Typically, I'm looking for 90% at minimum,

but you may decide that a lower percentage is okay.

Okay, so I'm finally ready to launch, what's next?

(screen whooshes)

An A/B testing tool

like the one available in the HubSpot CMS,

makes it super easy to set up.

That's what I'm doing here.

But you can also set one up through Google Analytics.

Unless your A/B Test is time related,

like testing the best time of day

to send an email, for example,

then you should be running both variations

of the A/B test at the same time to ensure accurate results.

So I'm gonna let my A/B test run for three months,

both versions running at the same time,

just to make sure my sample size is big enough

to get plenty of useful data.

So now that I'm ready to launch the test,

it's time to see how it plays out.

See you soon.

(screen whooshes)

Welcome back.

I'm ready to check the results on the A/B Test

of my store's landing page.

First, I'm gonna focus on my goal metric,

which was to increase sales.

So even though I collected a lot of other data

like bounce rates and email subscription signups,

I'm not gonna put much weight on those

because they're not the data I was testing for.

So using the HubSpot A/B testing calculator,

I can determine if the results

will actually help me make a final decision.

Are they statistically significant

and therefore suggest a clear winner?

So I'm plugging in the results,

in this case, the number of visitors

who went to the landing pages and conversions.

In other words, the number of those visitors

that actually became customers.

You can see that I've plugged it all in

and the spreadsheet does the calculations for me.

Version A converted almost 10% more than version B,

which means it's 95% certain

that changing to version A will increase my sales.

95% is statistically significant.

So it's time to take action based on these great results.

I mean, version A is the clear winner,

so I'll disable the losing design

and let the winner take over full time.

Finally, while I wait for the sales to roll in,

I'll use the HubSpot A/B Test Tracker

to track the results and plan my next test.

CTA placement on the product landing pages.

But what if you're not getting

statistically significant results?

Well, you might just be bumping up

against some of the most common mistakes of A/B testing.

(screen whooshes)

Common mistake number one,

testing more than one variable at once.

Remember, the whole idea is to run tests

on specific elements, one thing versus another.

If you try to test a new font

and a new font size at the same time, for example,

it's more difficult to determine which specific element

is actually responsible for any change,

or if that change is even statistically significant.

Common mistake number two,

only testing the small things.

I know I've mentioned a lot of small variables

like font sizes and image count,

and those are all super important,

but you should also periodically test more

radical changes to your pages.

This can be a little more upfront work, I know,

but if you're not getting statistically significant results

for your small changes,

trying something bigger might lead you down a new path.

Common mistake number three,

having too small of a sample size.

You wanna be sure your sample size is big enough

to get reliable results

that are statistically significant.

For example, if I only ran my test for an hour,

getting a hundred visitors with a 10% increase in sales,

that's actually only 10 people

who responded positively to the test.

But since I tested for three months

and got around 158,000 visitors,

that means 15,800 people responded positively to the test.

See the difference the sample size makes for your data.

Common mistake number four,

making changes before the test is over.

I know it can be hard to be patient, trust me,

especially when you're testing something

that can make a huge impact on your business.

But it's critical that you let the test play out

even if it seems like there's a clear winner early on.

If you jump the gun, you might be compromising your results.

And finally, common mistake number five,

only running a test once.

The more A/B testing you do,

the more likely you'll encounter a false positive

at one time or another.

I mean, user behavior can be unpredictable after all.

But just like a science experiment,

it's always best to replicate your tests

with the same parameters in place

just to make sure you get the same results.

This is especially important

if the margin of improvement

of one version over another is minor.

(screen whooshing)

A/B Testing is a simple data supported way

of finding realistic, low-cost methods

for growing your business.

This is something a business

of really any size can take advantage of

in order to optimize every part of their operation.

Click on this link for HubSpot's free A/B Testing Kit,

which includes the spreadsheets I used here today,

along with a comprehensive guide

to everything you need to know about A/B Testing.

And if you're already an A/B Testing pro,

let us know how it's helped you optimize your business

in the comments below.

As always, please like and subscribe

to HubSpot on YouTube (mouse clicks)

for more great marketing how-tos.

I'll see you next time.

- I can't find this client info.

- Have you heard of HubSpot?

HubSpot is a CRM platform,

so it shares its data across every application.

Every team can stay aligned.

No outta sync spreadsheets or dueling databases.

HubSpot, "Grow better." (bright music)