Legacy of Anecdotes

Paul R. Pillar, a long-time intelligence professional, has written an article in the current issue of Foreign Affairs to defend the record of the CIA as a prognosticator of future events against the assertion in the book Legacy of Ashes that the CIA is pretty much worthless in this capacity. It seems to me that the whole debate is being conducted – at least in public – without the data that would be required to draw any meaningful conclusions.

Legacy of Ashes describes a whole series of Agency prediction failures. Pillar responds, somewhat sensibly, that prediction in this field is inherently hard, and then lists correct CIA predictions for the Six-Day War in 1967, the Tet offensive in 1968 and the social and political aftermath of the invasion of Iraq in 2003.

While Pillar is certainly right that the correct standard to apply to CIA predictions is not “correct 100% of the time or else the CIA is incompetent and worthless”, it is also not true that the right standard is “correct anything more than 0% of the time and the CIA is worth any amount of money we spend on it”. If the CIA made 3,000 predictions between 1950 and today, and got exactly the three that Pillar cites right, a 0.1% rate of correct predictions doesn’t even come close to beating just flipping coins. What we need to know is the CIA’s predictive batting average versus the best available alternative. Once we have that, we could try to evaluate the positive benefits of any increased accuracy and compare it to the costs, broadly considered, of the CIA.

Of course, one of the reasons people often cite batting averages is that baseball has lots of discrete episodes within a given game, and therefore lends itself to statistical evaluation. Intelligence is surely more like soccer, with fluid play and ambiguous beginnings, ending and results of episodes. So such an analysis of CIA predictive accuracy would therefore by necessity incorporate many elements of judgment. But without it, all we have is dueling anecdotes.

(cross-posted at The Corner)