The U.S. presidential primaries bring out streams of commentary on market research in America, and this year already promises to uphold the tradition. eMarketer — not one of my own go-to IT research sources — is running an article today showcasing its analysts opinions on why the primary polls are so inaccurate (Lies, Damned Lies and Polls, eMarketer free newsletter). I’ve got to applaud Jon du Pre Gauntt’s comment. It is as applicable to the work of the industry analysts as it is to political polls:
“The inherent problem with polling and statistical sampling of any kind is that the respondent is asked to consider a choice either in isolation (”Do you think X is good or bad?”) or in a very restricted set (”Of X, Y and Z, which do you prefer?”). But, with very few exceptions, real life isn’t like that. Decisions of import usually have three or four or five variables in play.
The rub is that it’s nigh impossible to design a five-minute poll or a 20-minute survey that isn’t, by definition, contrived around the clutch of hypotheses the researcher wants to test. The response can either validate or invalidate the hypothesis. But it does nothing to prove whether the researcher has asked the most important questions.
Case in point: Ask most anyone whether they’d like their favorite digital music on their phone easily accessible and the data would show huge demand. Yet one of the things that bit both the operators and labels in the rear was that they assumed that data point translated smoothly into a willingness to pay.
It’s dead easy to answer a hypothetical question when you’ve got nothing on the line. It’s easy to say that you value this or that when you don’t need to pay right then and there for that statement. Or you support this or that candidate while you still retain the option to vote.
But you work with what you’ve got. To paraphrase Churchill, polls and surveys are terrible ways to try to understand the electorate or a market. But the alternatives are worse.”
In 2008, I hope we see analyst firms provide greater transparency about their credentials in market research — including staff education, professional development, and accuracy track records. It’s also high time we see more information on the research partners, and their credentials, as well.
Tags:
January 22nd, 2008 at 9:56 am
I’m sure many people would agree with your point that survey data is remarkably bad at predicting future behavior. But now with so many electronic transactions to mine, and so many available Gigabytes on analyst hard drives, we’re seeing usage information replace traditional surveys as a leading data source, especially in technology markets.
When clients are sifting though millions of records in their transaction and customer databases, they’re not going to buy a report because you’ve just conducted a 15 question survey. Ultimately, good analysts have to be just as creative developing new data sources as they are interpreting information after it’s been collected.
January 22nd, 2008 at 5:28 pm
Well said. The quality of the analysis needs to continue improving, especially as analysts obtain access to more data. The concept of the traditional Point Of Sale data is evolving very quickly now with wireless, mobile and RFID/NFC, and it’s important that high tech analysts keep up.
Qualitative research — such as focus groups and peer councils and case studies — will continue to play an important role in industry research and analysis. To me, that makes for a more logical research business focus going forward than the whitepaper business model we see so much of today.