Your Gut Instinct is Wrong – So Try A/B Testing

From an early age, we’re taught to go with our gut instinct in the face of uncertainty. Whether it was the Grade 6 Spelling Bee or the high school multiple choice exam, we’ve learned to follow our intuition when we’re unsure of the right decision.

Of course, this tendency carries on into our professional lives as well. Globally, 58% of executives said they “went with their gut instinct” when making their last big decision. PwC’s global survey also revealed that just 33% of executives described decision-making at their companies as “highly data driven.”

This all makes sense, though, because we’re living in a world where information is everywhere, and with such an overwhelming amount of data available, we’re more easily distracted than informed. With so much data out there to be analyzed, it’s not uncommon for decision-makers to get muddled in trying to make sense of it all, causing them to fall back on experience and intuition when making everyday decisions.

Moving From Your ABC’s to A/B Testing

In the absence of complete information, we’re naturally inclined to rely on heuristics and gut feelings to make important decisions. But as an astute, forward-thinking professional, it’s important to perform regular gut-checks that test your intuition using a process known as A/B testing.

Put simply, A/B testing involves comparing two or more variables under the same conditions, with the goal of identifying the most successful alternative. But A/B testing is more than just a technique… it’s best described as a way of thinking, a dogma to live by in our professional lives.

To buy into the idea of A/B testing, we must first come to the realization that our gut instincts aren’t always perfect. Research shows that the best decisions are made by blending intuition, experience and fact; but when we don’t have the facts and experience, we tend to make decisions simply on a hunch, leaving us vulnerable to a lapse in judgment that could cost us down the road. Ultimately, it’s in our best interest to test our assumptions and hunches in order to uncover which alternatives actually provide the most return.

A/B testing was primarily the domain of market research and advertising firms up until about a decade ago, when it began to spread to new industries and new applications. From retail through to software, A/B testing is the ‘new’ secret weapon for lean companies looking to find significant value without much overhead.

In retail, for instance, A/B tests can be performed in a variety of ways:

• Testing two variations of a promotion to maximize gross margin dollars (i.e. $10 off when you spend $50 or 15% off when you spend $50).

• Testing in-store merchandising teams to justify the investment (i.e. calculate ROI from in-store visits and compare against nearby comp stores).

• Testing two variations of merchandise assortment (i.e. uncover which planogram assortment best displays your products and drives the most sales).

• Testing two types of packaging (i.e. launching a new type of packaging to see which is most effective).

Getting from A to B

When you take a step back and look at your most recent business decisions, you’ll likely find a variety of opportunities in which you could have used A/B testing to challenge your assumptions. While there’s no shortage of opportunities to apply A/B testing, most companies fail to get off the ground because they don’t have the right reporting infrastructure to analyze their test data and make fast, data-driven decisions.

If you believe that A/B testing can help your business, but don’t know where to begin, consider these 3 tips for getting started with the Test and Learn approach.

Streamline Data Collection

Before launching an A/B test, it’s imperative to have a reliable flow of data, as A/B testing can be difficult if data is slow to source and challenging to work with. In committing to a culture of experimentation, it’s best that data be collected automatically and in a format that is easy to analyze. This can be achieved using an analytics platform (Tableau, Askuity, etc.) or by building an in-house data warehouse.

When data collection is no longer a hassle and ad-hoc reporting is a breeze, A/B testing becomes second nature, and adoption across teams is much more seamless.

Find Comparable Groups / Environments for Testing

One of the more difficult aspects of A/B testing can be finding comparable environments to run tests. The impact of external factors such as weather and socioeconomic variables are detrimental to A/B tests, as these extraneous circumstances can influence the results and invalidate findings. This is why it’s of upmost importance to identify groups that have near-identical characteristics.

For example, when running an A/B test across two groups of stores, comp stores could be grouped by variables such as: total sales, year-over-year growth rate, store format and region.

Set Target Metrics

Target metrics can be anything, as long as the metrics are easily measured over time. Depending on the use case, this can include any number of options, from gross margin dollars to button clicks. When selecting goals, be sure to consider whether test variables are likely to have a significant impact on target metrics. To demonstrate, let’s look at two example tests:

1. You’re measuring the open rate of your emails (the target metric) by testing two variations of an email subject line (the test). Assuming your two test groups are equally random and similar, this is a good example of a test variable that is likely to have a significant impact on your target metric. Aside from the sender’s name, the subject line is the only other key variable that is likely to influence whether or not someone will open your email – making open rate an ideal target metric for this particular test.

2. You’re measuring your retail customers’ total basket size by testing two variations of a welcome sign placed on the sidewalk. In this example, the welcome signs are not likely to have a strong influence on the target metric – total basket size – making this a less than ideal test. Rather than looking at the total basket size, you may want to measure the total number of customers who shopped at the store. While it’s difficult to argue that a sign causes customers to spend more money, it’s reasonable to justify that a sign drives more shoppers into the store.

Testing for the Truth

Along with the benefits that A/B testing offers for your bottom line, the Test and Learn approach can also improve your business in other, unexpected ways. When the culture shifts to a ‘let’s test it’ mentality, you’ll begin noticing that your test results will have broad applicability to other areas of your business.

By highlighting your best initiatives, A/B testing allows you to see trends that you may have otherwise missed. As a marketer, for instance, you might realize that one subset of your target market consistently responds better to one style of messaging versus another. By uncovering this trend over a series of tests, you can apply these findings more broadly across your organization, to improve other areas such as sales and customer service.

Another added benefit of testing is that you’ll find your team spending less time deliberating in front of a white board, and more time putting tests into action. Rather than debating for hours over option A or option B, you can move ahead with trying both alternatives and observe the results in real time.

Testing also opens up decision-making to the entire team, and reduces the impact of groupthink in your organization. Instead of just one or two people making every decision, team members will be motivated to contribute and suggest alternatives, leading to a more diverse set of ideas.

Above all else, A/B testing keeps you and your team honest. By challenging your organization to make data-driven decisions through A/B testing, you’ll begin uncovering profitable tactics to optimize sales and fuel growth – without abandoning your intuition.

Image: Dilbert Comics