Finding accuracy in IT industry research

Accuracy should be an important factor in choosing an ICT industry analyst as a trusted advisor. Analysts provide input to key decisions where accuracy counts, from negotiating prices to building business plans. That means analysts should willingly demonstrate their track record in producing accurate market forecasts, rankings, trends, and analysis.

It would be refreshing to see clear grounds for analyst bragging rights on accuracy. I’m talking about a year by year, market by market listing of the firm’s important positions, along with pointers to proof of their accuracy. Published right out there in the open, on the company’s web site.

So, why isn’t this happening?

The obstacles are not that great. It’s easy to prove the accuracy of some predictions — such as the timing of the next iPhone debut, or drop in average retail prices by year-end. For these kinds of market milestones, facts are easily verified through 3rd party sources.

Granted, it’s much more difficult to prove the accuracy of complex market forecasts, best practices, and trends. After all, there is no definitive source for 20-20 hindsight on complex market analysis. (And, if a definitive 20-20 source existed, the analysts wouldn’t want to acknowledge it — it’d put them out of business.) However, reputable analysts use a sanity check to cross-check the accuracy of their work. Publishing some of this QA to the public seems like a reasonable expectation.

So much for the obstacles. What about the risks?

Analyst claims about research accuracy can backfire. Publishing a track record can raise new sales objections. It can generate ridicule. It can undermine the perceived value of premium research content.

My sense is that the rewards will outweigh the risks within a few years. Consider the growing competition for traditional research and consulting budgets, and not just from other analyst firms. Accuracy will be easier to demonstrate than attributes like vendor independence.

Social media is another factor. Someone is going to lead the conversation about market research accuracy, and it doesn’t have to be an analyst. The Industry Standard (still in beta) Prediction Market is a good example of what comes out of left field when engaged users generate content about analyst accuracy.