The larger bias, however, is the fact that the survey is entirely self-selected–in other words, you are only hearing from the people who are willing to click on a banner ad and willing to take a survey online. Self-selected research is not by definition valueless, but it’s also kinda like, say, Pizza Hut only conducting research via the comment cards left on the tables. Chances are, you have filled one of these out only if 1) you had a lousy meal, 2) you had the best meal of your life, or 3) you are some combination of crazy/terminally bored.

So, in other words, you start with a fraction of reality–the folks who will click on ads (which is not the same as the folks who read/absorb ads), then take an odd fraction of that universe in the people who will sit down and take an uncompensated survey. So, definitely not representative or remotely projectable to the online populace under any circumstance. You might be encouraged to see that 500 people take your survey over a weekend and believe you have a smash hit–but it is possible that those 500 people represent your total potential audience–you have no way of knowing or even guessing if this is directionally accurate. I am not saying this approach is without value–I am suggesting that you could get fooled or pulled in dangerous directions unless you know how far from reality your respondents are.

Now, there is a lot of research done this way, and it isn’t without merit–you could argue that the folks who do respond to this sort of thing are exactly the people likely to be your best customers. It is true that if I click on an ad for widgets, and then take a survey on widgets, I probably loves me some widgets-so I am probably a prime candidate to sign up for widgtr.com or whatever your product is. But know that thousands of businesses have failed because they were unable to cross the chasm between the earliest adopters and even the early mainstream consumer on the consumer adoption curve. The folks who respond to this kind of research may lead you down a path that satisfies them completely, but fails beyond that. I know you said a “niche market,” but even in niche markets there is a full representation of the consumer adoption curve, with a representative bell-shape running from the earliest adopter to the folks who wait until last. Even if your product is fake plastic vomit, you cannot assume that the people who respond to this kind of research are remotely representative of all the people who might buy your yack.

Let me give this example–say Apple decided to follow this method in the initial tests of the iPod. What they would have ended up with, I would argue, would have been miles away from the iPod they launched with, which had fewer features and was more expensive than anything else on the market at the time. The “early adopter” iPod would have probably been another Rio, only with a built-in bottle opener.

So, am I suggesting this is worthless, or a bad idea? No—especially if you can get a quick head start by satisfying an early base, getting some seed capital, and then using that seed capital to grow your business-by doing some more research. By all means, do this–but know that there is no axiom in my business that states “even bad research is better than no research at all!” even though I hear it from time to time. Just treat it as a first step in the game–but be prepared to revisit it as soon as you can afford to. The Internet is littered with the corpses of various Flooz’s and Beenz’s who were probably fooled into thinking they had actual products by just this sort of research. But before you go and hire Whoopee to be your spokesperson, be sure you throw it out and go do some more.

Keep up the good work,

Tom Webster
Vice President
Edison Media Research