John Zogby's Open Letter to Nate Silver
John Zogby’s wrote an open letter to Nate Silver that is typically seen as an expression of emotion and clash of egos. But in my view, the deeper reality it revealed between the lines was that 538 has created a real business model problem for pollsters.
Silver intelligently combines multiple polls to make more accurate predictions than are usually achieved by any one individual pollster. On one hand, the math of this is irresistible – in the real world, voting models often work. On the other hand, it would be pretty uncomfortable for a pollster to combine his own results with various competitive poll results to achieve equivalent accuracy (or at least to do so transparently). So, the pollsters do all the tedious work to collect and analyze the data, and then Nate Silver comes along and creates all this value with it in a way that is hard for the pollsters to duplicate. You can see why this situation might upset the pollsters.
In every industry that combines data collection with analysis, there is an endless battle between the data collectors and the analysts. The data collectors bear the hard costs – people, office space, telecommunications, travel budgets, etc. – that are required for interviewing people, visiting stores, and so forth. Their nightmare world is to become commodity data collectors paid for their costs plus a small margin set by competitive bidding. Their typical defenses are to attempt : (1) to build proprietary methods for collecting superior data, or equivalent data at much lower cost, and (2) to integrate the analysis and the data into a single product, and forbid by contract the paying client from using this for other purposes. The analysts, on the other hand, want to have an open market in commoditized data, and compete on analytical capability.
In my experience, the value chain always unbundles eventually into separate data providers and data analysts. The lever is the client’s checkbook. It’s in their interest to push down the cost of data collection and force more competitive bidding, since this is where most of the cost, but very little of the hard-to-copy technique that produces defensible margins, sits. They can also more directly compare alternative analytical providers, and analytical innovation therefore can become more rapid. The core capabilities of a data collection operation versus an analytical operation are somewhat different, and this specialization then makes it ever harder for integrated providers to compete once the market begins to change in this way. (Though if there’s one source of revenge for the data collectors, it’s that eventually the analytical innovation plateaus, and the analysis becomes so routinized that it too becomes a commodity itself – until eventuall there is some new some new analytical breakthrough, and whole cycle repeats.)
Zogby’s open letter is the cri de coeur of an incumbent smart enough to see a chunk of his economic profits under threat as polling meta-analysts seize value, potentially increasingly forcing him to compete as a commodity supplier. What’s got to be especially galling is that (as far as I understand 538’s business model) his new competitors are literally giving it away.
We all laugh at the Luddites for smashing looms that threatened their livelihoods, but it is the nature of a market to constantly commoditize high-margin labor. It’s not so funny when you’re on the receiving end.
You’re missing a big piece of analysis here. Most pollsters don’t make any money predicting the national GOP congressional average, or even a senator’s current approval/disapproval – the tasks that Silver is taking over.
Pollsters make their money helping campaigns and organizations figure out what to do NEXT, for example by testing various arguments among different demographics. Message testing and strategy development – which is an art, not a science – is the real value of polling, and you can’t aggregate that.
— Josh · Jul 19, 06:01 PM · #
Josh,
Fair enough. I didn’t go into that mcuh. The opinion researchers I know (a small smaple) actually view all the politicla work as loss leaders – they make their money with companies. In that sense, if they lose a posiiton in the high-profile political work, it would hur them that way. But agin, I only know a few of them.
— Jim Manzi · Jul 19, 06:26 PM · #
That reminds me of the great Nielsen vs. IRI price war debacle of 1987 onward in the consumer packaged goods checkout scanner data market research industry following the Reagan Administration’s veto of their proposed merger. Even after driving a third competitor, SAMI, out of business, the two duopolists consistently failed to make profits because there was really only room for about 1.5 firms in the business because of the huge fixed cost of buying and processing checkout scanner data from a nationally representative sample of supermarkets. My suggestion was that the two firms jointly own the data collection and processing business and compete in data analysis, but that still had anti-trust problems.
— Steve Sailer · Jul 19, 08:51 PM · #
“We all laugh at the Luddites for smashing looms that threatened their livelihoods, but it is the nature of a market to constantly commoditize high-margin labor. It’s not so funny when you’re on the receiving end.”
I’ve been managing to stay about a step and half away from the receiving end for about 15 years now, and no, it’s not fun. On the other hand, there’s a nice post over at The Atlantic about soldiers DIY music videos. Fair trade, I say.
— Tony Comstock · Jul 19, 11:55 PM · #
Is this whole thing not analogous to the complaints of MSM news-gatherers against bloggers?
— Roger Evans · Jul 20, 12:43 PM · #
I admit I haven’t really paid a whole lot of attention to 538 since the presidential election… so maybe they’ve gotten better… but the problem with this theory is that Nate Silver’s analysis is fairly amateurish. I suspect you could do just as well simply using Pollster’s poll aggregator, than any of their bizarre and arbitrary weighted models. Nate Silver’s value added isn’t his analytic powers… which would could be duplicated/surpassed by any math or statistics grad student… but his commentary, which combines solid writing, partisanship, and some quantitative rigor. He’s popular because he gives liberals like me an easy place to go for charts and graphs to back up our arguments. His threat to Zogby is mainly as a guy who gets on TV to talk about the polls, not because he’s putting together data from multiple polling outfits… as I mentioned, anybody can do that with Pollster.
— J.W. Hamner · Jul 20, 01:31 PM · #
I’m going to disagree on this analysis. If pollsters shift to primarily data collection rather than analysis, the cost of high quality data collection will stabilize. As the Reasearch 2000 fiasco has shown, cutting corners on paying for data collection is very risky.
The real crunch with be on the media organizations that pay for these national polls. Why would NYTimes/CBS/Fox/WashPost… pay for a poll if they know everyone is going to look at the aggregation sites rather than their own sites for comparisons? As I’m writing this, the NY Times decision to host 538.com, makes great sense.
For the bigger picture, the question is setting up a system where neural parties want to pay for polls in the first place when they usually won’t profit from those payments. If all polls become partisan, it will be a mess for everyone.
— bsci · Jul 20, 02:06 PM · #
This is incredibly interesting, but Manzi’s point has an even broader application to the culture at large: http://www.legallysocial.com/archives/299
— Darren Cahr · Jul 20, 03:40 PM · #
I disagree about what would motivate John Zogby to write his letter. Nate Silver has been doing his analysis for some time now without inciting a response. What Zogby responded to was Nate creating an industry-wide scoreboard of results. Nate’s scoreboard makes it very easy to say which pollsters have been historically accurate and which have not, and which have been historically biased and which have not. Naturally pollsters who want to be taken seriously want to lead in both categories, but many also want to be able to ‘tweak’ their numbers to the benefit of their clients. By exposing the resultant inaccuracies, Silver threatens the ego-boosting polling model. And certainly Zogby’s poorly-rated Internet polls are, as shown by Nate’s rankings, effectively worthless.
— Peter · Jul 20, 05:06 PM · #
“What Zogby responded to was Nate creating an industry-wide scoreboard of results. Nate’s scoreboard makes it very easy to say which pollsters have been historically accurate and which have not, and which have been historically biased and which have not.”
Peter,
Bingo! We have a winner.
Silver’s simply developed a public scorecard and anyone with a computer or hand-held device that’s not his Johnson can access Silver’s database to peruse the results.
Now if someone would only do the same tedious bit of work to track and assess the accuracy of pundits and bloggers.
— Mark · Jul 20, 05:51 PM · #
J.W. Hammer says: “the problem with this theory is that Nate Silver’s analysis is fairly amateurish. I suspect you could do just as well simply using Pollster’s poll aggregator, than any of their bizarre and arbitrary weighted models.”
Well, maybe. But I remember the Democratic primaries in 2008 very well, and 538’s estimates were much more accurate than the aggregates available at Pollster.com. Freakishly so.
He also did some good reporting and writing, but the track record in the primary is what made him must reading by the time the general election came around.
— Braise · Jul 21, 12:27 AM · #
@Braise:
“He also did some good reporting and writing, but the track record in the primary is what made him must reading by the time the general election came around.”
IIRC Pollster didn’t have the feature where you could exclude consistently biased polling outfits or types of surveys in the 2008 primaries, so it’s probably true that their aggregation tools were not as useful as Silver then. However, there were a cadre of political scientists and statisticians analyzing all that polling data and blogging about it in 2008… and accumulated similarly gaudy records… Silver was simply the highest profile. I do not believe that this profile is due to the quality of his models… which were fine, but not particularly sophisticated and fairly Rube Goldberg to no apparent benefit… but mainly in his ability to talk about quantitative things in a way a layman can understand. I really just don’t think there is much evidence that modeling/aggregation is particularly threatening the polling industry… if anything I’d say they benefit from him talking about polls to what I presume is a fairly large audience.
— J.W. Hamner · Jul 22, 04:04 AM · #
Home medical equipment is a category of devices which is used for patients whose care is being managed from a home or other private facility managed by a nonprofessional caregiver or family member.
— home medical equipmentsale · Jul 27, 07:17 AM · #