3 Things Marketing Experts Don't Tell You About Split Testing

Best Home business in 2007

3 Things Marketing Experts Don't Tell You About Split Testing
By: Jim Stone

I've been split testing for a while now, and even use some pretty advanced methods these days. It has worked out well for me, but I still remember many early disappointments.

Many very smart marketers will tell you there's a lot of power in A/B split testing.

They'll even tell you that you CAN get 1000% improvements by changing a single word -- or even a single letter.

These marketers are right. But if you think they're telling you the whole story, you're in for a rude awakening! I found out there were a few things the experts hadn't told me.

Here are three of the most important things:

1. It's More Difficult To Set Up And Run A Simple Split Test Than You Might Think:

In order to set up a good split test you must:

* upload a new sales page with just one change to it. You might also have to create two different "thankyou" pages, depending on how you set up your tracking.
* get a script that splits traffic between the two pages. Ideally this traffic splitter will split the traffic randomly, and not just in an "every other" fashion.
* make sure your script sets a cookie on the visitor's machine and serves the same page to them the next time they come to the site.
* put a tracking system in place that will track your visitors and your sales for both pages separately.
* set up a database (perhaps).

And getting set up is just part of the hassle. It's also difficult to run the test and interpret your results.

One time I set up a test, and started sending traffic to the pages being split tested. Then I noticed a typo on ad B. So I corrected it. Now, "strictly speaking" I shouldn't have changed anything during the test, but I was probably OK doing this, even though it's not technically good form.

That wasn't the problem. Here's the bad part: when I made the change to my page, my editing software (without my knowing it) changed the names of the fields in my opt-in form on page B, and, as a result, it didn't work any more.

I didn't realize the mistake until a week later. During the week I had marveled at how ad A had done so well, while ad B pretty much just stood still.

That was one week wasted. My results were meaningless.

(I don't know about you, but this kind of thing happens to me all the time. These are the stupid pesky gremlins that infest every new thing I try. Why can't things ever go as planned?)

Other things can happen, too. Ad campaigns can stop. Hurricanes can strike. Terrorists can strike. Christmas can strike. Your page can get listed on the front page of Yahoo, and suddenly you're getting a different kind of traffic -- and lots of it. All of these things can create "noise" that affects how your ads perform.

You also have to know how long to run your test before your results are "statistically significant".

Truly, simple split testing can be something of a hassle.

But the hassle isn't really the worst part about split testing. Here's something else the split testing "gurus" haven't told you:

2. You Can't Consistently Get Great Results On Your Own.

If you look at your landing page, and ask yourself, "How can I improve this?" You will probably come up with something to test. And then you will test it.

Good for you, right? You were Johnny on the spot. You got going right away. You didn't sit around debating the merits of your choice. You took action. What could be wrong with that?

The problem is you probably came up with some safe little change. Some timid little tweak. And it probably didn't help much, if any at all (and it might have hurt). It probably wasn't a golden nugget that significantly boosted your response. It might have been, but it probably wasn't.

The truth is, to get best results, you need to get others to help you brainstorm for testing ideas.

But the experts haven't told you that.

And there's one more thing they haven't told you:

3. With Simple Split Tests, You Should Be Prepared For Lengthy Strings Of Disappointing Tests -- Even When You Generate Great Ideas.

Did you know that most of the changes you make to your landing page will not help?

According to Charles Holland of Qualpro (Breakthrough Business Results With MVT), if you run 25 split tests, about 13 won't show any significant difference. And 6 of the proposed changes will have a negative effect. That leaves about 6 to have a significant positive effect. That's 6 improvements out of 25 tests.

But it gets worse. Did you know it takes about 100 ideas to get 25 elements that you can easily split test? That's because 75% of the ideas you will generate for improving your ad will be things that are too expensive or time consuming to implement, or can't be combined with other things you want to include on your page.

That means you must generate 100 testing ideas to get 6 that will actually make a positive difference on your page.

Please, re-read that sentence, and let it sink in this time. It's that important.

If you have ever been discouraged by a string of 5 or 6 A/B split tests that made no difference, all in a row, you know in your heart that these numbers are very close to the mark. You might have felt alone -- just an unlucky sap for whom nothing ever goes right -- but you were actually just experiencing normal, run-of-the-mill split testing.

The experts didn't tell you that, either.

They didn't tell you how much of a hassle it can be, they didn't tell you what you need to do to generate good ideas, and they didn't tell you how long it can take to get meaningful results.

So How Do You Deal With These Problems?

1. The hassle. You're just going to have to bite the bullet. Split testing is one of the best ways to improve your profits. Get some good software, and some good information about running and interpreting your tests, and just dig in. This doesn't exactly made it any easier, but at least you've been warned.

2. The help. Imagine what kind of ideas you would get if you assembled 20 people in a room. Other marketers, potential customers, existing customers, maybe your mother. And you ask them the question "How can I improve this sales page?" And you let them throw ideas out, and let other people's ideas inspire new ideas, and you do this until you have 100 ideas -- ideas that come from many perspectives, from people with nothing to lose, and from people who are thinking more and more "out of the box" as the session goes on.

If you could do THIS, then you would have a good chance of finding some truly good testing ideas.

Such a big brainstorming session is impractical for most people. That's true. But it can't hurt to be told what the ideal is. At least now you'll be thinking of ways you can approach this ideal.

For instance, you might consider checking out what your competitors are doing. You might pay better attention to the marketing books you've read. You might solicit ideas from discussion forums.

And maybe, just maybe, you will go to the trouble of actually getting a few friends and relatives to sit around with you and brainstorm a bit.

3. The discouragement. First, it helps just to be told what to expect. Winning ideas are rare, but they usually deliver big results when they come. So the hunt is worthwhile, but it can be discouraging.

But there's actually something you can do to change the situation in your favor.

There's a new method of split testing called "accelerated split testing", or "Taguchi testing", or "Multi-variate testing". It allows you to take all your ideas -- the good ones and the bad ones -- and test them all at once. Your chances of hitting a winner (or two or three) somewhere in that jumble is actually very, very good. If you set your tests up right, you should get significant improvements from almost every single test.

So there you go. Three problems the experts haven't told you about, and three ways to deal with the problems.

Now you've been told.

About the author:
Jim Stone is an accelerated split testing expert. He has developed an accelerated split testing software product, and writes regularly on the subject. You may receive his free course: "10 Steps To More Sales With Accelerated Split Testing" at:

Circulated by Article Emporium

Return to Index