The U.S. presidential primaries bring out streams of commentary on market research in America, and this year already promises to uphold the tradition. eMarketer — not one of my own go-to IT research sources — is running an article today showcasing its analysts opinions on why the primary polls are so inaccurate (Lies, Damned Lies and Polls, eMarketer free newsletter). I’ve got to applaud Jon du Pre Gauntt’s comment. It is as applicable to the work of the industry analysts as it is to political polls:
“The inherent problem with polling and statistical sampling of any kind is that the respondent is asked to consider a choice either in isolation (”Do you think X is good or bad?”) or in a very restricted set (”Of X, Y and Z, which do you prefer?”). But, with very few exceptions, real life isn’t like that. Decisions of import usually have three or four or five variables in play.
The rub is that it’s nigh impossible to design a five-minute poll or a 20-minute survey that isn’t, by definition, contrived around the clutch of hypotheses the researcher wants to test. The response can either validate or invalidate the hypothesis. But it does nothing to prove whether the researcher has asked the most important questions.
Case in point: Ask most anyone whether they’d like their favorite digital music on their phone easily accessible and the data would show huge demand. Yet one of the things that bit both the operators and labels in the rear was that they assumed that data point translated smoothly into a willingness to pay.
It’s dead easy to answer a hypothetical question when you’ve got nothing on the line. It’s easy to say that you value this or that when you don’t need to pay right then and there for that statement. Or you support this or that candidate while you still retain the option to vote.
But you work with what you’ve got. To paraphrase Churchill, polls and surveys are terrible ways to try to understand the electorate or a market. But the alternatives are worse.”
In 2008, I hope we see analyst firms provide greater transparency about their credentials in market research — including staff education, professional development, and accuracy track records. It’s also high time we see more information on the research partners, and their credentials, as well.
Tags: 
