A Beginner’s Guide to A/B Testing: Frequently Asked Questions

Posted by Florencia Carrizo Heit on May 31, 2017

Most businesses today, regardless of size, location, or industry, are aware, at the very least, of the basic traffic numbers for their website, like how many visitors they have to their site every month or how long those visitors stay on the site. But even some companies’ most sophisticated traffic reports overlook the most important metric of all – conversion rates. Analyzing conversion rates means measuring how many of your site’s visitors convert into paying customers.


Why is this so important? Conversion analysis gives you a true understanding of what “success” looks like for your website. It also uncovers any glitches or roadblocks in your sales funnel so that you can optimize your website with key, strategic improvements to achieve higher conversions (more sales) and, as an obvious result, increased overall revenue for your company.


So, exactly how do you get those higher conversion rates (and increased sales revenue)? One of the most common and most effective methods for boosting conversion rates on your website is A/B testing. It’s a cornerstone of any successful digital marketing strategy, but to a beginner, it can be a bit overwhelming. Newsflash: it doesn’t have to be.


Here are the most frequently asked questions we get about A/B testing, and if you don’t see an answer to a question you have about A/B testing, conversion rates, or optimization, you can always email us at optimization@multiplica.us and we’ll add it to our FAQ list!:


 1) What is A/B Testing?

When you run a basic A/B test on your website, you’re comparing two variations of an experience, element, or page on your site to determine which version performs better with your customers. Typically, this means showing the original version of the page to one half of your visitors and a new, modified version to the other half. By tracking the performance of the two versions, you will learn what types of experiences your users prefer on your site so that you can then make any necessary optimizations to ultimately show them what they want and need in order to become converting, paying customers.

AB Test


2) What should I test?

One of the easiest and most effective ways to make quick improvements to your site is to test your call-to-action (CTA) buttons, which are prompts that inspire or guide your users to take some sort of action (like “click here,” “add to cart,” “buy now,” or “learn more”). By testing these buttons, you might find, for example, that simply changing the color of a CTA from yellow to blue will increase your click-through-rate (CTR) by 15%, or that changing the wording from “buy now” to something more industry-specific – like “book my trip” for a travel website – more than doubles your online sales. Whether you’re testing color, copy, placement, shape, font, or size, the options are plenty when you’re testing CTAs, and positive results tend to happen fairly quickly once you optimize the CTA. Gotta love those “quick wins,” right?

We also recommend testing:

  • Your website’s images. Test images to determine what grabs your visitors’ attention and gives them the most positive experience of your site.
  • Social icons or buttons. Not only is the placement of your social icons on the page pretty important, but you should also test their size and the look & feel of the buttons or its surrounding content (like white space). This is especially good to test if your business is struggling with engagement on social media and you want to try to boost traffic to your social sites.
  • Forms! Every paying customer on your site has to fill out a form at some point in the sales/conversion funnel, and poorly designed forms are one of the top reasons that online shoppers abandon a transaction. So, forms matter, and you should be testing yours. Quick tip: users like short forms that are clear and concise, avoid asking for sensitive or personal information, and don’t take long to fill out.
  • Your banner headline. This is another quick, easy win you can get through A/B testing. For example, we’ve found through years of running tests for clients that sometimes shorter headlines combined with a bolded sub-heading performs better when tested against long, stand-alone headlines.


 3) How do I interpret the results of my test and decide which “winner” to pick?

Keep in mind when you report on the results of your A/B tests that you’re calculating the ratio of difference between the two tests, not an absolute increase. For example, imagine that you tested a red CTA button on your homepage (Option A), and that resulted in a 10% conversion rate. You then compare this to the test results for a green CTA button (Option B) on that same homepage, which gave you a 20% conversion rate. So, green is the obvious winner, but by how much? If your answer is that the green CTA performs at a 10% higher conversion rate than the red CTA, sorry, but you’d be wrong… Instead, you should calculate the ratio difference between Option A and Option B by the rate, not the absolute increase. By doing this, you learn that there is a 100% in conversion rate with the green CTA button over the red.


The Winner Is....


4) Can I test more than two things at the same time?

Yes, and this is called a “multi-variate test” (MVT). For example, you may simultaneously test two different versions of headline copy in your banner and two different colors for your CTA buttons – this is multi-variate testing. Keep in mind, though, that if you’re new to testing altogether, this is a more advanced method of testing (because the more you test, the more combinations of results optimization strategies you get), and it may require additional outside help depending on the resources you or your company have.


 5) How often should I test?

It’s no surprise that websites need to be improved and updated regularly to keep up with ever-changing user behaviors, needs, and trends. So, as soon as you have a clear, specific plan for what you’d like to test and why, you really can run as many tests as you like – testing and analyzing the results is one of the best was to really hear what your users want from your site and what they like (or dislike) about it as well, so you’d be hard-pressed to listen to them too much. Just keep in mind, though, that the ultimate objective of A/B (or MVT) testing is to make sure your customers have the best journey possible on your site and in your sales funnel.


Need help with testing on your website? Reach out to us at optimization@multiplica.us!


Happy Testing!

Topics: Test and Optimization