Skip to main content

An argument against A B split testing

programming and human factors
by Jeff Atwood
On a recent airplane flight, I happened to catch the movie Groundhog Day. Again.
If you aren't familiar with this classic film, the premise is simple: Bill Murray, somehow, gets stuck reliving the same day over and over.
Phil doesn't just go on one date with Rita, he goes on thousands of dates. During each date, he makes note of what she likes and responds to, and drops everything she doesn't. At the end he arrives at -- quite literally -- the perfect date. Everything that happens is the most ideal, most desirable version of all possible outcomes on that date on that particular day. Such are the luxuries afforded to a man repeating the same day forever.
This is the purest form of A/B testing imaginable. Given two choices, pick the one that "wins", and keep repeating this ad infinitum until you arrive at the ultimate, most scientifically desirable choice. Your marketing weasels would probably collapse in an ecstatic, religious fervor if they could achieve anything even remotely close to the level of perfect A/B testing depicted in Groundhog Day.
But at the end of this perfect date, something impossible happens: Rita rejects Phil.
The next time you reach for A/B testing tools, remember what happened to Phil. You can achieve a shallow local maximum with A/B testing -- but you'll never win hearts and minds. If you, or anyone on your team, is still having trouble figuring that out, well, the solution is simple.
Read more at www.codinghorror.com

Comments

Popular posts from this blog

MarsEdit 4

I've been using MarsEdit for just over seven years. I started with version 2, and then upgraded to version 3 in 2010.Now, in 2018, I just upgraded to version 4. This is my first post with the upgrade and I'll report on it in a little while as I get used to the new version.Here is what Daniel of Red Sweater Software has to say about his product:Browser-based interfaces are slow, clumsy, and require you to be online just to use them. Browsers are perfect for reading web content, but not ideal for creating it. If you're serious about writing for the web, you need a desktop blog editor. If you're lucky enough to have a Mac, nothing is more powerful, or more elegant than MarsEditRich and Plain Text Editing MarsEdit's editor switches easily from Plain to Rich Text, so you can work in whichever format you prefer. Love Markdown? MarsEdit can preview it and convert it to HTML if needed.Wildly Compatible Works with WordPress, Blogger, Tumblr, TypePad, Movable Type, and any …

Resistance Is Futile

Some of the trains that run from London Kings Cross to Cambridge divide at Cambridge. Part of the train goes on to Kings Lynn and the rest stops at Cambridge and does the run back to London. Perhaps it picks up other carriages coming from somewhere else, I don’t know but it seems likely.When you get on the train and before it sets off, there’s a public announcement announcing how the train divides. It explains that the train is composed of eight carriages and that passengers wanting to travel beyond Cambridge should ensure that they are in the front four carriages.The rear four carriages are of course nearest the barrier, so we choose to sit in one of those carriages if for no other reason than that we don’t have to walk so far to find a seat.So there we are and the voice comes over with the announcement. And then there is a pause and then a voice tells you what carriage you are in. So we hear ‘Six of Eight’ or ‘Seven of Eight’ – and every time I hear it I think of Star Trek, ‘Three o…

Business For Britain Is Concerned With Business For Britain

This report in the New York Times today


LONDON — Is British business fretting about the risks of the country drifting out of the European Union? Or does it crave a looser relationship with Continental allies, one free from meddlesome regulation?
The answer to that question remained unclear Monday after a newly formed group of business leaders argued for a renegotiation of Britain’s membership terms — echoing the policy of Prime Minister David Cameron, who in January promised voters a referendum on whether the country would remain in the Union.
The new group, called Business for Britain, is intended to counter the intervention of pro-E.U. business leaders who have warned of the dangers of Britain slipping out of the 27-nation bloc and its single market of 500 million people. A statement released Monday to announce the group’s formation was signed by about 500 executives.
I think this opinion in the New York Times article is interesting:

Never much attracted to the idea of European unity,…