Yes, we can learn to love statistics and A/B testing
I am a big fan of optimization and A/B testing. But that was not always the case. When I was studying for my MBA at HEC in the 1980s, the most challenging courses I enrolled in was Statistics and Probabilities. This course, loaded with obscure equations, was meaningless gibberish to me. And I couldn’t see how it could be useful in real life except maybe for surveys.
Ironically, a few years later, I found myself starting a career in direct marketing, an area where everything was based on statistical analysis, probabilities and confidence intervals.
I quickly learned the value of optimization testing, as well as some of the pitfalls that novices face. Here are some tips to help you better implement a testing strategy. Never fear, we’ll try not to bore you with statistical concepts and equations.
Foster an optimization culture
The most successful companies understand that small gains of a few percentage points can add up to big gains when compounded year after year.
They also understand that the marketer’s gut feel can only go so far. And that our personal biases sometimes lead us to make the wrong decisions. After all, we are not representative of our customers. We must objectively validate our assumptions, through optimization testing. And to maximize our chances of success and growth, testing must become part of your DNA and your corporate culture.
Give yourself the right to make mistakes
The first step towards an optimization culture is to accept that some tests will give disappointing results. We must give ourselves the right to the error. We must also accept that our current practices may not be the best and that there is always room for improvement.
On the other hand, do not excess by multiplying frivolous tests, because each underperforming test means that your sales will be impacted.
Prioritize the A/B testing that will have the greatest impact
In this sense, we must test the hypotheses that are most likely to have a major impact on the commitment and sales. Among these, we can test:
- Targeting a campaign to identify niche markets that are more receptive to the product / service
- The elements of an offer (the price and the expression of this price, the incentives – deals, gifts, free delivery, contests, etc., the duration of the offer and the conditions of sale)
- The structure of the emails, the type of content, the relative position of the elements – in particular the CTAs.
- The graphic processing of emails including imagery and emotion emanating from it.
But do not overdo it, like testing one word vs. another, the color of a button, and so on. Use your good judgment.
Your optimization test must respect the rules of statistics
When performing optimization A/B tests, an estimate is made of the proportion of customers who will be likely to have a given purchasing behavior. If we repeat the test several times, we will observe a similar result, but different. The dispersion of these results around the mean is what is called the standard deviation.
What you need to know is that the larger the sample used in your tests, the smaller the standard deviation. So the more the result of your test is reliable. Make sure you are working with samples of sufficient size so that you can validate if the result of your test are actually different (better, or worse) from what you normally observe.
Also make sure you only change one variable at a time. You cannot change the target population, the product price and the creative concept and expect a statistically valid result.
Measure the profound impact of your tests
It is not enough to dwell on engagement rates (opening and clicking). You must also look at the conversion on the site (newsletter sign ups, potential leads, bid solicitations, purchases and customer retention) and the long-term impact on the health of the database.
It is an issue that is measured over several months, even years, and you have to be vigilant. We have already seen situations where a test before stimulated a very large increase in the recruitment of new clients, but poor quality clients whose loyalty was very low.
Create a results directory accessible to all
The knowledge that results from these optimization exercises is a competitive advantage and the result of these efforts must be preserved and made accessible to all stakeholders in the marketing team, or even senior management.
Create a secure directory that groups the results of each test. After just a few years, you will have accumulated valuable strategic information that will allow you to make better decisions. This will allow you to always go further, but also not to test the same hypotheses again. After all, repeating the same gestures while expecting a different result is the definition of madness.
Some statistical analysis resources
• Confidence interval calculator
• Sample size calculator
Do you want to implement a successful optimization strategy? Contact us to discuss it.
