Yesterday, WSJ’s Lee Gomes revisited his 2002 criticisms of Aberdeen Group. (2008: Vendors Still Paying…; 2002: Glowing Report on Firm X Isn’t… . I respect Lee Gomes. I suggest that analyst watchers and research buyers read the piece, not only for what’s there but for what’s not.

In short: Mr. Gomes checks up on Aberdeen, now part of Harte-Hanks and risen from its own ashes (from a fire many credit Gomes with lighting). He acknowledges some key changes at Aberdeen — such as transparency on vendor sponsored research — but essentially sends readers away with the bitter scent of pay-for-play hanging in the air.

I’ve been impressed with the Aberdeen business model. I see it as one that successfully links high-volume, low-cost benchmark surveys with vendor demand-generation programs. It’s closer to traditional market research and more scalable, on paper, than many of its IT analyst competitors engaging in the same kinds of activities. Of course, final judgement on whether any of this is good or bad comes down to daily execution.

Gomes finds much to criticize about that.

Yet, he avoids a couple of elephants standing in the middle of the room.

One of these elephants is whether ICT industry analyst research warrants the kind of mandated clean-up we saw with U.S. investment industry research.

Gomes finds that 5 years of Aberdeen self-imposed improvement still come up short of the mark on business ethics. What now? Is it time we progress into a full-fledged debate on government intervention?

After all, WSJ readers are not likely to be thinking this as an isolated situation at one IT industry research company. More likely, they’re seeing it as a tacit statement on ethics across the IT industry analyst sector. And, the WSJ readership embodies a hefty chunk of the IT analyst customer base.

Another elephant in the room is whether the old distinctions between market researchers and IT industry analysts still hold true.

What is the difference? Do we understand it? Is it relevant?

Or should we put the “industry analyst” distinction on moth balls, just another Maginot Line left over from the 20th century?

More blogger reactions to Lee Gomes coverage on Aderdeen

Investile Dysfunction: IT Research Firms Get the Smackdown from the Wall Street Journal

The Whole Nine Yards: Lee Gomes vs Aberdeen: Round 1

Technobabble 2.0: Aberdeen Group - guns for hire, which includes a response from Stephen Gold, President of Aberdeen Group, a Harte-Hanks company

AttentionMax: TThe Future Of Industry Analysts In The Tech Sector

ITtoolbox: Is it really research or is it paid advertising?

Spend Matters: The WSJ Shows No Love for Aberdeen

Paula Rosenblum’s Blog: WSJ - It’s so easy to Misunderstand

UPDATE Feb 8:
Analytics Evolution: Aberdeen Group Under Fire from WSJ Reporter

One of the questions that lands in my inbox every month is, “Who is on your list of ‘Top 5′ industry analyst firms?” I used to treat this question without hesitation, and rattle off the five firms with the highest revenues. Like most people, my logic was straightforward: either you slice and dice by a general coverage/geographical area, or you name the global top 5 based on who’s making the most money.

Money’s not enough of a basis for the “global top 5″ anymore. As an industry, we live in more interesting times and this is changing the ICT research and advisory playing field. And, let’s face it: years of analyst watching has changed this watcher. Today will mark my 10,000th post on Industry Analyst Reporter.

Analyst company revenues and state-of-the-business continue to be important metrics in my eyes. As always, I say the best source for this kind of info is Louise Garnett at Outsell. She obtains information from privately held companies as well as the handful of public ones, and she correlates it against research into Fortune 500 corporate purchasers.

What other criteria do I use for naming a “global top 5″? I like these attributes:

Baseline coverage: does the company have the scope and depth of topical expertise and primary research for addressing all the basics of enterprise IT, networking and telecom?

Advisory/consulting availability: does the client/expert ratio facilitate meaningful discussions on applying research to decisions and practices?

Track record: what indicates proven success in research quality, accuracy, consistency, customer loyalty, and customer satisfaction?

Research expertise: does the company recruit professional market research skills and topical expertise, and invest in ongoing professional development in both research skills and topical expertise?

Research sources: what is the caliber and extent of company’s primary and secondary research sources?

Ethics: has the company made a public commitment to uphold market research industry standards and ethics, as defined by appropriate regional market research or competitive intelligence associations?

How does that line up with your thinking?

In an interview with AMR Research, Carter Lusher / Dave Eckert make a surprising assertion:

“Only one analyst firm in the IT industry has a dedicated and systematic research approach on emerging technologies. However, that one firm is the 900 pound gorilla Gartner.”

That’s not the view from where I sit.

It’s rare that an emerging technology surfaces first in Gartner research. And, Gartner’s hype cycle and cool vendor reports are hardly the end all, be all of systematic industry research on bleeding edge tech.

Emerging technologies almost always come to light first at one of the dozens of smaller, agile analyst firms offering in-depth insight into specific markets. It’s easy to see why. These analysts tend to have deeper roots in early stage technologies and markets. They are easier to get to. They have robust networks among developers, investors, other opinion leaders, and early adopters. I see it time and again, from semiconductors to Wi-Fi to 3G to graphics to collaboration software to managed services.

Gartner does a good job of discussing technologies that are ready for early enterprise adopters. They also do a good job on emerging tech providers ready for IPO or acquisition. That doesn’t make Gartner the first — or the only — analyst outfit with a dedicated focus on emerging technologies.

A Symphony Services press release just gave me a jolt. The head reads, “blah-blah-blah Featuring Leading Independent Analysts John C. McCarthy and Bruce Richardson.” What’s this? John McCarthy quit Forrester, Bruce Richardson quit AMR Research, and they’re independent analysts now? No. I misinterpreted Symphony’s use of “independent.”

“Independent” gets my vote as one of the most over-used, confused, and divisive labels applied to the industry analysts.

Many vendors regularly refer to analyst research studies as “independent”. You’ve seen it in press releases and SEM ads: “An independent study shows…” That always seems a bit odd to me. Think about it. In the first place, why would you bother to promote an analyst study if you felt that right off the bat, you need to convince people that it’s “independent”? In the second place, what’s the benefit of any vendor calling any analyst report “independent”? Either your sales target is already a skeptic and holds a negative opinion about the objectivity of the analyst firm in general (which is now rubbing off on your company by association), or you’ve just raised an objection that your sales target may not have had. You lose either way.

I’m just as guilty as any other culprit in taking liberties with the word. I say “independent analysts” when I mean “SOHO analysts.” It’s a courtesy, from one SOHO (me) to another. Plus, I’ve always disliked “boutique analysts.” What does a boutique analyst do? Sell trendy, overpriced research from Italy? Wouldn’t that mean we should refer to IDC, Gartner and Forrester as “big box analysts”?

We can agree that Mr. McCarthy and Mr. Richardson are well known analysts. We can agree that they’re competing analysts. Chances are we won’t agree on whether they are “leading independent analysts”, until we take some of the “I” out of the meaning of independent.

Jason Busch adds good context to an unusal situation in the analyst sphere: AnswerThink adopting the name The Hackett Group.

Generally, ICT research and advisory company names disappear entirely (or at least beyond recognition) following an acquisition. I’m curious to see whether Informa eventually kills off the names of its most recent shopping spree: will Datamonitor and its own recent acquisitions go the way of the Informa buy-outs before them — ARC Group (the mobile sector ARC, not the manufacturing sector ARC), Baskerville, Chorleywood, EMC…

The only other case similar to Hackett Group that I can think of is Alan Meckler. He renamed INT Media Group to Jupiter Media, following the 2002 acquisition of Jupiter Research. At the time, he commented on how much he liked the name. He wasn’t kidding: he held onto the name much longer than the research and advisory, JupiterResearch.

The AP and Ars Technica and about a hundred other news sites are covering the MPAA’s admission this week that its commissioned study on P2P movie* piracy contained inaccurate data — almost a 30 point error in the calculations. One of many, many things that’s wrong with the entire MPAA study saga — from 2005 through to this minute — is that there has been no shoulder-to-shoulder community response from the leaders of the high tech industry analyst firms.

Background: MPAA had the study conducted in 2005 by LEK Consulting. The MPAA has been using findings from the study for political lobbying, public positioning, industry positioning — basically, every possible use for a set of “proof points”. They never released the methodology or raw data, and so no credible source could refute the findings in a point-counterpoint manner. Many credible sources did express concern, individually, about the accuracy of the data points that were released.

Granted, you can pass this off as another example of what a well financed organization can get away with when they hand off “facts” from a confidential research study to a bevvy of lawyers, lobbyists, and PR specialists. Plus, LEK Consulting is a custom research company, not an industry analyst firm.

But look again. What’s at issue here is a research study into consumer tech behaviors, devices, services, and content. Points of debate include questionable data gathering methods, questionable data analysis and conclusions, and questionable uses of data. This is the industry analysts’ backyard.

Consider the impact a joint statement would have carried, if signed by the companies recognized for their expertise in technology market research: Gartner, Forrester, IDC, Burton Group, Info-Tech Research Group, Informa TM, JupiterResearch, M:Metrics, Parks Associates, RedMonk, Juniper Research, NPD Group, Yankee Group, and tens to hundreds of reputable high tech research firms and indies.

Cooperating on a common position statement would have made a big impact. It could be a single statement presenting divergent viewpoints. However, speaking one at a time, as though no other analyst exists on the planet, had no visible impact whatsoever.

I’m not calling for pandemic love among analyst firms worldwide. I am saying that cooperating for the good of the public is good for business.

* Original post corrected from “music” to “movie” - BF

Nathan Gilliatt has an interesting model for using RSS to “aggregate and redistribute market intelligence,including analyst research subscriptions, inside the corporate firewall.” He calls it IDS, or Intelligence Delivery System. Analysts are just one source category.

What I like most about his IDS model is that it enables an organization to bring together diverse information in a stable repository and strip out the costs and chaos of things like internal portals, applications and multiple 3rd party content aggregators. To be sure, I’m a fan of these tools and services today. Whatever limitations they present, they provide capabilities that many organizations simply could not justify building for themselves.

Where I disagree with IDS is the emphasis on importing and RSS for herding analyst research content. Some of the most valuable analyst content is delivered in formats that are not RSS-friendly — such as graphics, spreadsheets, databases, video, and interactive tools. Increasingly, content will be delivered in mutiple languages and with role, industry, and regional context.

Robust search applications seem the more promising way forward, for analyst research consumers and analysts alike.

Hat tip to Vinnie Mirchandani for starting this discussion.

The U.S. presidential primaries bring out streams of commentary on market research in America, and this year already promises to uphold the tradition. eMarketer — not one of my own go-to IT research sources — is running an article today showcasing its analysts opinions on why the primary polls are so inaccurate (Lies, Damned Lies and Polls, eMarketer free newsletter). I’ve got to applaud Jon du Pre Gauntt’s comment. It is as applicable to the work of the industry analysts as it is to political polls:

“The inherent problem with polling and statistical sampling of any kind is that the respondent is asked to consider a choice either in isolation (”Do you think X is good or bad?”) or in a very restricted set (”Of X, Y and Z, which do you prefer?”). But, with very few exceptions, real life isn’t like that. Decisions of import usually have three or four or five variables in play.

The rub is that it’s nigh impossible to design a five-minute poll or a 20-minute survey that isn’t, by definition, contrived around the clutch of hypotheses the researcher wants to test. The response can either validate or invalidate the hypothesis. But it does nothing to prove whether the researcher has asked the most important questions.

Case in point: Ask most anyone whether they’d like their favorite digital music on their phone easily accessible and the data would show huge demand. Yet one of the things that bit both the operators and labels in the rear was that they assumed that data point translated smoothly into a willingness to pay.

It’s dead easy to answer a hypothetical question when you’ve got nothing on the line. It’s easy to say that you value this or that when you don’t need to pay right then and there for that statement. Or you support this or that candidate while you still retain the option to vote.

But you work with what you’ve got. To paraphrase Churchill, polls and surveys are terrible ways to try to understand the electorate or a market. But the alternatives are worse.”

In 2008, I hope we see analyst firms provide greater transparency about their credentials in market research — including staff education, professional development, and accuracy track records. It’s also high time we see more information on the research partners, and their credentials, as well.

During December, two blogs written by analysts were added to Tekrati’s Directory of Analyst Blogs, a freely available directory and OPML.

The December additions are:

Company: Hurwitz & Associates
Fern Halper’s data makes the world go round
Robin Bloor’s Blog: have Mac will blog

On December 31st, the directory contained 260 blog listings. Blogs with validated feeds are included in the directory OPML.

I appreciate your help in letting me know of any blogs I’ve missed.