Dave Rossiter has navigated the Institute of Industry Analyst Relations (IIAR) blog into an ongoing discussion about analyst ethics and independence. The post more or less recounts the usual assortment of issues and war stories, and then poses the perennial question:

“As analyst relations professionals, we face a challenge. What responsibility do we have for ensuring these practices are stamped out? Are we proactive or do we just refuse to support them? Do we have a ‘quiet word’ in the right ear? Do we out the bad apples in public?”

Whether the IIAR is truly sinking its teeth into the issue — or simply jumping the shark — remains to be seen.

The IIAR needs to focus attention on where it can make a positive impact. If you look closely, David does bring up one area where the IIAR could make an impact: the misuse of briefings as sales calls.

This is actionable because it’s a specific business practice, it’s commonplace, each occurrence is obvious, and each occurrence is easily documented. As an added bonus, there’s no “objectivity” rhetoric around it to make AR practitioners uneasy: an analyst sales pitch is attached, or it’s not.

Plus, a precedent exists. ESOMAR’s guideline, “Maintaining Distinctions between Marketing Research and Direct Marketing,” addresses similar issues within the context of consumer market research:

“Whenever researchers are acting in their capacity as researchers they must not be involved in carrying out direct marketing or other non-research activities. Such activities are by definition incompatible with Rule 4 of the International Code which safeguards the confidentiality of respondents’ personal data which have been collected for marketing research purposes and prevents these from being used for any non-research purpose.

“This Rule does not prevent researchers, when they are acting in a different capacity (e.g. as a general information manager), from being involved with the operation of marketing databases. However, in such a case they must clearly differentiate such an activity from their work as a marketing researcher and avoid any confusion arising between the
two types of activity.”

Of course, the IIAR would need to clean up some questionable AR practices around briefings, as well.

Yesterday, WSJ’s Lee Gomes revisited his 2002 criticisms of Aberdeen Group. (2008: Vendors Still Paying…; 2002: Glowing Report on Firm X Isn’t… . I respect Lee Gomes. I suggest that analyst watchers and research buyers read the piece, not only for what’s there but for what’s not.

In short: Mr. Gomes checks up on Aberdeen, now part of Harte-Hanks and risen from its own ashes (from a fire many credit Gomes with lighting). He acknowledges some key changes at Aberdeen — such as transparency on vendor sponsored research — but essentially sends readers away with the bitter scent of pay-for-play hanging in the air.

I’ve been impressed with the Aberdeen business model. I see it as one that successfully links high-volume, low-cost benchmark surveys with vendor demand-generation programs. It’s closer to traditional market research and more scalable, on paper, than many of its IT analyst competitors engaging in the same kinds of activities. Of course, final judgement on whether any of this is good or bad comes down to daily execution.

Gomes finds much to criticize about that.

Yet, he avoids a couple of elephants standing in the middle of the room.

One of these elephants is whether ICT industry analyst research warrants the kind of mandated clean-up we saw with U.S. investment industry research.

Gomes finds that 5 years of Aberdeen self-imposed improvement still come up short of the mark on business ethics. What now? Is it time we progress into a full-fledged debate on government intervention?

After all, WSJ readers are not likely to be thinking this as an isolated situation at one IT industry research company. More likely, they’re seeing it as a tacit statement on ethics across the IT industry analyst sector. And, the WSJ readership embodies a hefty chunk of the IT analyst customer base.

Another elephant in the room is whether the old distinctions between market researchers and IT industry analysts still hold true.

What is the difference? Do we understand it? Is it relevant?

Or should we put the “industry analyst” distinction on moth balls, just another Maginot Line left over from the 20th century?

More blogger reactions to Lee Gomes coverage on Aderdeen

Investile Dysfunction: IT Research Firms Get the Smackdown from the Wall Street Journal

The Whole Nine Yards: Lee Gomes vs Aberdeen: Round 1

Technobabble 2.0: Aberdeen Group - guns for hire, which includes a response from Stephen Gold, President of Aberdeen Group, a Harte-Hanks company

AttentionMax: TThe Future Of Industry Analysts In The Tech Sector

ITtoolbox: Is it really research or is it paid advertising?

Spend Matters: The WSJ Shows No Love for Aberdeen

Paula Rosenblum’s Blog: WSJ - It’s so easy to Misunderstand

UPDATE Feb 8:
Analytics Evolution: Aberdeen Group Under Fire from WSJ Reporter

The AP and Ars Technica and about a hundred other news sites are covering the MPAA’s admission this week that its commissioned study on P2P movie* piracy contained inaccurate data — almost a 30 point error in the calculations. One of many, many things that’s wrong with the entire MPAA study saga — from 2005 through to this minute — is that there has been no shoulder-to-shoulder community response from the leaders of the high tech industry analyst firms.

Background: MPAA had the study conducted in 2005 by LEK Consulting. The MPAA has been using findings from the study for political lobbying, public positioning, industry positioning — basically, every possible use for a set of “proof points”. They never released the methodology or raw data, and so no credible source could refute the findings in a point-counterpoint manner. Many credible sources did express concern, individually, about the accuracy of the data points that were released.

Granted, you can pass this off as another example of what a well financed organization can get away with when they hand off “facts” from a confidential research study to a bevvy of lawyers, lobbyists, and PR specialists. Plus, LEK Consulting is a custom research company, not an industry analyst firm.

But look again. What’s at issue here is a research study into consumer tech behaviors, devices, services, and content. Points of debate include questionable data gathering methods, questionable data analysis and conclusions, and questionable uses of data. This is the industry analysts’ backyard.

Consider the impact a joint statement would have carried, if signed by the companies recognized for their expertise in technology market research: Gartner, Forrester, IDC, Burton Group, Info-Tech Research Group, Informa TM, JupiterResearch, M:Metrics, Parks Associates, RedMonk, Juniper Research, NPD Group, Yankee Group, and tens to hundreds of reputable high tech research firms and indies.

Cooperating on a common position statement would have made a big impact. It could be a single statement presenting divergent viewpoints. However, speaking one at a time, as though no other analyst exists on the planet, had no visible impact whatsoever.

I’m not calling for pandemic love among analyst firms worldwide. I am saying that cooperating for the good of the public is good for business.

* Original post corrected from “music” to “movie” - BF

It’s that time of year again. Time to decide which industry analysts called the shots, gave the answers, and made the time often enough to earn an IT research and advisory contract for 2007. My advice to analysts this year: publish proof points showing your accuracy, timeliness, objectivity, engagement. Put forward some well researched — not just well rehearsed — reasons for us to believe.

Analyst bashups, in general, are nothing new. Historically, the most damage was done by competitive sales teams and word of mouth — the kind of thing you find in any industry. A few journalists would take the time to sleuth planned budgets or controversial practices, and that was pretty much the extent of it.

Blogs have changed the old analyst bashups. More people than ever are publishing anecdotes about smart and not-so-smart analyst opinions, research, forecasts.
Read the rest of this entry »