How many A/B tests should you be running?

A/B test volume is often a topic of discussion in CRO. If you listen to certain presentations at conversion conferences, you may be alarmed to hear that your competitors are running 75 tests each month. Does that mean they’re learning 75 times more than you?

Chances are they aren’t. In fact, it’s more likely that they’re muddying their data by running overlapping experiments that impact each other, or they’re concluding tests too soon, or the bulk of their experiments are extremely small or have little thought behind them.

The reality is that website optimisation takes time and quality has to be the most important metric. If you make CRO a numbers game where the aim is simply to run as many tests and make as many changes as possible, you compromise on results and end up with no actual insight to help grow your business.

One way you can increase your test throughput is by understanding your realistic testing velocity for your traffic volume. This isn’t just something to view at an overall level – effective segmentation is key to planning simultaneous tests that don’t overlap.how many a-b testings should i run

For example, you might run tests on PPC traffic hitting your landing pages, or on mobile product pages only, or only the users who haven’t already seen another existing test on your site, and so on. As long as you have enough visitors to generate a robust result within a realistic timeframe, do it.

It’s important to do this upfront planning to prevent you from having to run tests for months at a time – either because you included too many variants or because there simply wasn’t enough traffic present to run a basic A/B test in the first place.

Fortunately, as conversion optimisation practitioners, we don’t have life or death situations in our hands every time we come to work. If we’ve proved beyond doubt that a particular change will improve site conversion, we can get that change live so we can start reaping the rewards in full and move on to the next test. There are too many variables in eCommerce for us to try to pin down the precise uplift created by an individual test.

Ultimately, the bigger picture is what matters and a CRO programme delivers over a year or longer – not just over the course of a couple of tests.

If you believe your business could improve conversions through A/B testing, contact our CRO team by email at [email protected]