Trying to find the tallest dwarf: A/B testing can be problematic
‘There’s nothing wrong with A/B testing‘. This statement is both true and not useful; better is, ‘there’s nothing wrong with A/B testing if you’re A/B testing the right bits‘.
But how do you know what are ‘the right bits’? Applied psychology is your answer.
Often we’ll speak with clients about their attempts to create change – an uplift in clicks, more of X behaviour, less of Y actions – and how their results are better but not much better.
The problem is clear; fighting to improve something that’s fundamentally poor. Or, as the famous joke about asking for directions from a local when lost in the countryside goes, “I wouldn’t start from here”.
Tom Goodwin (@tomfgoodwin) describes it well:
The culture of A-B testing and/or "test and learn" just seems like a way for people to value ideas less, believe in them less & fight for them less.
A billion A-B tests leads to the entire UI of Amazon – it's absolutely terrible. No vision or strategy or delight.
— Tom Goodwin (@tomfgoodwin) November 21, 2018
Common challenges we see are:
- Tweaking clicks on a particular page in the user flow, when the user flow itsef is burdening the user’s cognitive capacity, so the messaging kept on each page of the journey needs to change
- Improving the wording on a gain-frame message (‘Improve your…’), when even a poorly constructed loss-frame message (‘Underperforming stops here…’) would be way better
- Service descriptions that use narrative structure rather than a schematic structure ( Example of Insurance product here)
These challanges can feel quite tactical, but really they underpin both a communication strategy (a single, coherent narrative) and marketing strategy (specific executable plans for a product/service mix) – of which we work on
— Christian Goy (@GOY909) November 21, 2018
So it’s no surprise the A/B testing ‘discomfort’ permeates the whole marketing community, as Christian Goy (@GOY909) describes above.
It is clear that A/B testing is good when you deploy it in fertile spaces. Don’t spend time trying to find the tallest dwarf. Or, to mangle the joke, using applied psychology means “I wouldn’t start from here” turns into “Let’s start from over there”.
If you find yourself ‘over there’, then come over here.
WE CHANGE THE WAY
For more speak with Davina (Client Services Director) or Oliver (Founding Director)
+44 (0)843 289 2901