Skip to main content

An argument against A B split testing

programming and human factors
by Jeff Atwood
On a recent airplane flight, I happened to catch the movie Groundhog Day. Again.
If you aren't familiar with this classic film, the premise is simple: Bill Murray, somehow, gets stuck reliving the same day over and over.
Phil doesn't just go on one date with Rita, he goes on thousands of dates. During each date, he makes note of what she likes and responds to, and drops everything she doesn't. At the end he arrives at -- quite literally -- the perfect date. Everything that happens is the most ideal, most desirable version of all possible outcomes on that date on that particular day. Such are the luxuries afforded to a man repeating the same day forever.
This is the purest form of A/B testing imaginable. Given two choices, pick the one that "wins", and keep repeating this ad infinitum until you arrive at the ultimate, most scientifically desirable choice. Your marketing weasels would probably collapse in an ecstatic, religious fervor if they could achieve anything even remotely close to the level of perfect A/B testing depicted in Groundhog Day.
But at the end of this perfect date, something impossible happens: Rita rejects Phil.
The next time you reach for A/B testing tools, remember what happened to Phil. You can achieve a shallow local maximum with A/B testing -- but you'll never win hearts and minds. If you, or anyone on your team, is still having trouble figuring that out, well, the solution is simple.
Read more at www.codinghorror.com

Popular posts from this blog

My Blog Is Carbon Neutral

I am pleased to be able to make the statement in the title to this article and I want to explain how I have done it and how I came to hear about the scheme that enables it.

The first step along the path that brought me to this happy state is that I have joined Seeded Buzz.

The idea behind Seeded Buzz is for the members to plant seeds - which means telling other bloggers about their blog posts and inviting them to continue the conversation on their blogs with a link back to the original post on their blog.

Well I have found a blog post from one of the members that covers a subject that interests me.


Seeded Buzz points out that better the Seed conversation and the more extendable / debatable it is, the more other bloggers will Buzz about it on their blogs and link to the original post.

And as we all know, links are the engine that pumps searches - and these are the lifeblood of the internet.

The seeder profile I found was from Thomas Chasm who blogs about a lot of different categories …

Giveaway Tools

After looking at an article on InsightScope about contest giveaways, I just read the FAQs for one of the tools, which is KingSumo Giveaway and I see that as at the time the FAQs were prepared, the tool doesn't integrate with Mailchimp. Instead you have to download a CSV and upload.Also, there is a warning that the tool may not be compatible with Facebook's terms in the future.Finally, there is a long, convoluted way you have to deal with duplicate content, which is described in the FAQs as follows: I’m concerned about duplicate pages for SEO We don’t create new pages, just add a parameter to your URLS. Google just sees the original page and URL structure. What we recommend if Google results are important is to run the tool, then once it looks like a winner I’d change the Title to the winner and the original URL. Remove the other titles. We are working on a fix to make this easier.Advanced users:Utilize the parameter tools in Google’s Webmaster Tools and set the Headline plugi…

Jean-François Millet and John Everett Millais

Jean-François Millet and John Everett Millais - How not to mix them up

John Everett Millais John Everett Millais (1829 – 1896)was a Pre-Raphaelite painter (one of the founders of the Pre-Raphaelite Brotherhood) who painted colourful (too colourful?) paintings - mostly of people. His most famous is probably Ophelia, lying back arms surrendering to the current.

Jean-François Millet
Jean-François Millet (1814 – 1875) was a French painter and one of the founders of the Barbizon school in rural France. Millet painted realistic rural scenes - peasant farmers, sheep, trees - in a muted pallette that were nontheless romantic.