Trying to find the tallest dwarf: A/B testing can be problematic

Posted by | · · · · · · · · | The Hunter Blog | No Comments on Trying to find the tallest dwarf: A/B testing can be problematic

There’s nothing wrong with A/B testing‘. This statement is both true and not useful; better is, ‘there’s nothing wrong with A/B testing if you’re A/B testing the right bits‘.

But how do you know what are ‘the right bits’? Applied psychology is your answer.

Often we’ll speak with clients about their attempts to create change – an uplift in clicks, more of X behaviour, less of Y actions – and how their results are better but not much better.

The problem is clear; fighting to improve something that’s fundamentally poor. Or, as the famous joke about asking for directions from a local when lost in the countryside goes, “I wouldn’t start from here”.

Tom Goodwin (@tomfgoodwin) describes it well:

Common challenges we see are:

  • Tweaking clicks on a particular page in the user flow, when the user flow itsef is burdening the user’s cognitive capacity, so the messaging kept on each page of the journey needs to change
  • Improving the wording on a gain-frame message (‘Improve your…’), when even a poorly constructed loss-frame message (‘Underperforming stops here…’) would be way better
  • Service descriptions that use narrative structure rather than a schematic structure ( Example of Insurance product here)

These challanges can feel quite tactical, but really they underpin both a communication strategy (a single, coherent narrative) and marketing strategy (specific executable plans for a product/service mix) – of which we work on

So it’s no surprise the A/B testing ‘discomfort’ permeates the whole marketing community, as Christian Goy (@GOY909) describes above.

It is clear that A/B testing is good when you deploy it in fertile spaces. Don’t spend time trying to find the tallest dwarf. Or, to mangle the joke, using applied psychology means “I wouldn’t start from here” turns into “Let’s start from over there”.

If you find yourself ‘over there’, then come over here.

Related: 5 Things You Should Never Say (or Do) to a Little Person


1. Insight lab | 2. Co-creation & Strategy team | 3. Communication studio

For more speak with Davina (Client Services Director) or Oliver (Founding Director)
+44 (0)843 289 2901

You might like to read our published books/chapters and
twitter @HuntingDynasty

Also, join the London Behavioural Economics Network Meetup group and Facebook group

No Comments

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.