Why you probably shouldn’t be split testing

 |  Duncan Jones - Get more articles like this sent to your email here


Hi, I’m Duncan and today I’m going to be talking to you about split testing – or more accurately when you shouldn’t be split testing.

Quite a few years ago, I discovered split testing and instantly I was hooked… the fact that you could create two versions of the same thing with just one difference and data could tell you which one was the best was awesome.

So I set about testing everything across all the campaigns I was running – big changes, small changes, even button colours… The problem was though I shouldn’t have been testing a lot of the time and years later I’m still seeing marketers make the exact same mistake.

The reason every company can’t split test is how much volume you need to get statistically significant results.

What this means is that you need enough data to know to a high degree of certainty, that luck or chance has not influenced the results.

A simple example of statistical significance is if you throw a dice.

If say you only rolled it 6 times a 4 could come out half the time by chance, if you rolled it 60,000 times however the 4 would come out closer to its actual probability.

The same thing applies for split testing and this means that companies with not much data are essentially running tests that are like throwing the dice 6 times – that’s not the level of data you should be making business decisions from.

I can already hear you saying, “If I don’t have enough traffic, I can just run the split test for longer until I hit statistical significance.”

Well, I thought the same thing back then and I started running tests for 2, even 3 months.

The issue with that?

Time will influence your split tests – the longer you run the test the more likely it is that external factors, such as temporary website changes, technical issues and things like users deleting their cookie will come into play and this can make your test pretty inaccurate.

There is also a huge opportunity cost of waiting 2-3 months for tests to end as during that time you won’t be able to touch or improve the page using other methods.

What that means is that you could get to the end of 3 month period, have a test that made no difference and you’ve essentially wasted quarter of a year.

There are a lot of tools out there to calculate the amount of time it will take you with your current traffic to get to statistical significance such as this one by VWO.

When using these tools ensure you are using just the numbers from the page that you’re testing and not from your whole site.

If you get a result that shows over 4 weeks, I wouldn’t recommend split testing.

So what can you do?

One of the main ways you can improve your website and landing pages without split testing them is by doubling down on the qualitative research you’re doing this will give you the confidence that they’re the right changes prior to making them.

The research could include:

1. Running polls and surveys and asking open questions that will give you insights and ideas.

2. Heat mapping, click tracking and recording visitors as they use your website through a tool such as Hot Jar.

3. User testing your website to find out what your target market are actually thinking as they use it via a service like usertesting.com or even in person.

And make sure you don’t forget actually picking up the phone and talking to your customers or prospects.

These insights that they give you can be invaluable.

With enough qualitative data you can work out the actual issues and start improving them straight away.

You should also be doing the same thing with quantitative data and looking for device or browser compatibility issues, page speed problems and any other errors that you can confidently fix without testing them.

Along with research, I’d recommend monitoring your top competitors who get a lot of traffic.

If they’re big, chances are they are doing a lot of testing and if you closely monitor the tests they run and then the website changes they make you can see what is working and can get ideas on what to implement.

For example, if you’re in the hotel booking industry it would be very worthwhile reviewing booking.com’s website & checkout process regularly.

They reportedly have over 1,000 tests running every single day and are constantly improving every aspect of their site.

They also regularly speak at CRO events around the world, which is another sign of a company to follow.

Whilst you can’t blindly implement competitor’s changes it will give you some great ideas of things you can implement and it can also help back up your qualitative findings.

There are a few tools available to help you with the monitoring of your competitors websites such as kompyte.com or visualping.com.

When making untested changes to a low traffic site make sure to use annotations in Google Analytics for every change you can then keep an eye on conversion rates and the results from your marketing campaigns over time and revert things back if need be.

If after watching this video you disagree with me and still want to split test with a low traffic site, just make sure you are split testing huge changes such as a total redesign.

The bigger the improvement you can make, the fewer conversions it will take you to get to statistical significance and thus the shorter it will be.

That’s it for now, thanks for watching.

Duncan Jones

Duncan Jones

Head of Strategy + Growth at Webprofits
Duncan has been building websites and online businesses since the days of the 56k modem and made his first $1 online from annoying popup ads. With over 10 years experience, he leads Strategy + Growth at Webprofits and still loves the thrill of launching digital marketing campaigns, getting the first conversion and then rapidly optimising and scaling them for clients across the company.

Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Meet Webprofits

85+ full-time marketers. Offices in Australia, USA and Singapore. Helping challenger brands drive growth since 2006.

One Team. One Brief. One Objective

Webprofits is a digital growth consultancy with fully-integrated, end-to-end, agile digital marketing teams that challenger brands can use to drive rapid growth in a complex and fragmented digital landscape.

Find out more