Comparing Multivariate & AB Testing Tools

It is time now to select an AB or multivariate testing platform to start running your tests. There are over thirty different tools available in the marketplace for you to choose from. So, you might ask:

• What is the right split testing tool for my website?
• Will I affect the success of my conversion optimization program by choosing one particular tool?

The market is full of testing tools that range from simple free tools to sophisticated enterprise split testing software. This article will walk you through the pros and cons of different split testing tools available.

We will also suggest criteria to help you select the software that best fits your needs, considering ease of use, the agility of results, cost of running, testing support, operational issues, and testing analytics.

The different types of AB or multivariate testing tools available

2018 data from datanyze shows the following top AB testing tools:

We classify A/B testing tools based on different criteria:

• Based on tool price
• Based on how the split tests are implemented on a website
• Based on the scale of software (enterprise vs. smaller scale software)

Before we discuss different ways to classify different split testing software, let’s start with a brief discussion on in-house AB testing tools.

In-house AB testing tools

Few companies build their in-house testing tools to conduct AB or multivariate testing. There are two main reasons for using an in-house AB testing tool:

• legacy built testing tools;
• the complexity of website architecture

There is one more reason to consider for developing an internal testing tool. Many AB tests that generate significant results require considerable technical implementation. And while A/B testing tools are great in randomly splitting website visitors between the different variations, they are not that great in doing a post-test analysis to determine which visitor segments react positively to a particular design within the test. This is another reason that some large e-commerce and SaaS companies opted to build their own testing tools that integrate directly with their platform.

In-house testing tools account for less than 2% of the overall market. We expect this segment to remain at the same level.

Classifying A/B testing tools based on price

Split testing tools vary in price from the free of charge to tools that can cost tens of thousands of dollars.

1. Tools priced based on site metrics

These tools are priced based specific website metric. Different tools use different metrics to set the price including:

1. The number of tested visitors in a time period (typically a month)
2. Number of the website monthly visitors
3. Number of monthly page views
4. The number of calls to the testing tool.

These tools will vary in price and range anywhere from $50/month to tens of thousands of dollars.

2. Free AB testing tools

These tools provide the customer the ability to conduct split testing free of charge, but they offer limited sets of features, including a smaller set of segmentation, targeting and reporting functions. The leading player in this space is Google Optimize which continues to add more features on a regular basis.

Classifying AB testing tools based on testing technology (client side vs. server side)

There are two different types of ways testing tools are used on a website:

1. Client-side testing tools

In these tools, the testing scripts are executed by the browser of the website visitor. When a visitor comes to a webpage, the testing software javascript is executed, and the particulars of the test are loaded on the page.

Client-side testing tools typically come with a point and click visual editor that allows users to manipulate a web page.

In theory, these tools require limited involvement from the user’s technical team. Creating new designs in these tools is much faster compared to their counterparts.

Client-side testing tools are great for creating new designs for one page, but they are less suited to conduct more complex testing such as visitor flow tests. The cornerstone “visual editor” of these platforms breaks when modifying websites that rely heavily on javascript or dynamic content.

These tools come with a low-cost monthly subscription, starting at $50 per month. The subscription cost will vary based on the number of visitors a website receives in a month.

Amongst client-side testing tools, you can find:

• Optimizely
• Visual Website Optimizer
• Convert
• Omni convert

2. Server-side testing tool

In these tools, the testing script runs on the testing platform as opposed to the visitors’ browser. The visitor’s browser gets the final version to be displayed.

There are several ways testing is implemented using server-side tools. Adobe test and targe, for example, relies on the marking made by users on specific areas of the tested webpage. Variations of the marked areas are then created in the testing software by the optimization or development team. Each new test will require involvement from the development team.

Server-side testing tools frequently involve custom configuration to retrieve meaningful visitor tracking data. They also require organizations to have a full-time dedicated resource(s) to manage the software and to analyze the results.

While many of these tools do not come with an easy to use “visual editor” to make quick changes on a webpage, they do provide detailed reporting and targeting features.

Monthly plans for these tools start around $1,000 and vary based on the number of page views a customer uses in a month. In addition to the software cost, we estimate a $100,000 annual in-house cost of running tests on enterprise testing packages.

Server-side testing platforms are useful in conducting sophisticated testing such as multi-page testing, mobile application testing or visitor flow testing. However, they are an overkill when doing quick A/B testing for a webpage design due to the substantial cost associated with the deployment of each test. The leading player in this field is Adobe Test and Target. 

Selecting the right testing tool for your website

With many tools available, the following criteria can help you choose the best split testing software to meet your requirements.

1. Ease of use: How fast can you deploy a test?

We evaluate the ability of a testing tool to deploy tests quickly as a way of measuring its ease of use.

When you are conducting a conversion optimization program, your goal is to implement tests efficiently and not to have technology as a barrier to your efforts of increasing conversion rates.

Testing tools vary tremendously when it comes to ease of use. Some of the more sophisticated tools require complicated test setup, which can take days. Client-side testing tools outperform any of their counterparts in this area.

2. What is the cost of running the AB testing software?

The cost of running testing software varies from zero (Google Optimize) to thousands of dollars per month.

In addition to the cost of running the software itself, there might be additional required investments. Some split testing software is too complicated and will require having full-time staff to manage in addition to involvement from your development team. That, of course, will increase the cost of running your testing program.

3. Testing support

Sooner or later you will need some help from the testing software company. Many smaller companies ignore this one area when they are doing the initial assessment of the testing packages. We highly recommend evaluating the different alternatives available.

4. Operational issues (scale, performance, and high availability)

How long does it take the testing software to load up a particular design? Does the testing software use CDN (content delivery network) to deliver designs to visitors who live in different parts of the world?

We highly recommend evaluating the response time for each of the different software packages. Avoid any testing package that takes longer than 500 milliseconds to deliver a design.

5. Testing analytics

In addition to reporting the conversion rate for each variation, some split testing platforms will also report the following metrics for each of the variations:

• Revenue per visit
• Bounce rate
• Exit rate
• Visitor type (new vs. repeat)
• Traffic source
• Traffic medium

This data is essential to understand the impact of the new designs on all aspects of the visitor experience on your website. We recommend evaluating what metrics the testing software reports in a test as that can impact your final analysis.

6.  What multivariate approach does the software use?

This point is particular in evaluating multivariate testing software. When launching a multivariate test, the software will display the original and new variations to different visitors to determine the winning design. To select the winning design, the software will use either full factorial or fractional factorial testing:

Full Factorial Testing

The testing software will test all of the different combinations of elements and their alternatives. So, if a test has four elements and each element has three different combinations in it, the testing software will test all possible 3^4= 81 designs.

Fractional Factorial testing (Taguchi method)

The testing software will select a subset of all possible combinations of the different elements and their alternatives. So, in our example above, the testing software will fully test a smaller set (less than 30) to determine the winner. Fractional factorial testing allows tests to run faster since they do not fully test all possible combinations. Critics of this method point out that it is less accurate compared to full factorial testing.

The debate between full factorial vs. fractional factorial has been going on for years.

Most MVT testing software uses full factorial in determining the winner of an experiment or a mix between the two different methods. While this might be a sticking point for some conversion experts, for 90% of online businesses, it is not a critical point in selecting which software to use.