I wrote a post arguing that much of the economics profession’s asserted confidence about its ability to forecast the effect of certain kinds of policy interventions is unwarranted, and that this problem is especially acute for predicting long-term effects. Karl Smith, an economics professor who blogs at Modeled Behavior, responded somewhat critically. (Reihan Salam weighed in as well.) I then replied to Smith.
Smith has now replied to me again. If you are at all interested in the question of the reliability of the knowledge produced by economics, I think it is worth reading his latest post in full. I complement him for being willing to engage on this at length, and in a spirit of open-minded discussion.
Smith’s latest post begins with this:
Jim Manzi responds to my post. It seems I came off a bit harsher than I intended. Other posts lead me to think that some believe I’m rejecting Manzi’s argument against overconfidence in models. Quite the contrary I am suggesting that academics don’t actually have the level of confidence Manzi and Brooks ascribe.
Fair enough. But if economists don’t really have sufficient confidence in their models to rely upon them, then how do they predict the effects of policies that they propose? Smith goes on to describe this with clarity and candor:
Manzi and Brooks seem to believe that policy advice comes plugging and chugging on the big models but instead it comes from an intuition honed by working with both the simple theoretic models and war gamming the big models. [Bold added]
Smith sounds like a very smart and sensible guy. But if the predictive method is in fact the economist’s intuition rather than some predictive algorithm that can be validated empirically, then how do we know the prediction is reliable? I guess it would help a lot if a given economist had a long track record of making accurate, value-added predictions about the impact of similar interventions – but as I’ve argued at length, it is difficult even to measure the effect of many macroeconomic policy interventions after the fact.
This is crucial. Unless Smith can demonstrate that his intuition can actually predict the effect of such interventions more reliably than some alternative method, then all we have is his opinion that macro model X is useful, or that war-gaming exercise Y is relevant, or that X and Y should be combined using method Z (which can’t even be defined explicitly if it’s intuition).
What Smith is describing here is intelligent and data-driven theory-building. What’s missing is the part where the theory is tested, and proven to be reliable.
In other words: You say that you have the ability to predict the effect of stimulus. Prove it.
At the end of his post, Smith goes into the specific example issue that I used in my original post to try to illustrate some of the enormous difficulties that formal economic forecasting of stimulus must confront. My example was basically that if we execute a policy that creates greater economic activity today, this will lead to a set of investments that otherwise would not have happened, and this will in turn change the alternatives available to us in future periods. The term for this general idea is path dependence. I argued that path dependence might turn out to have effects on total long run economic output that are significantly positive, significantly negative, or not material one way or the other.
Smith, in his original reply, said that things like this would make no difference:
For example, if you asked what effect would properly done fiscal stimulus today have on the economy 20 years from now, the fairly easy and straight forward answer is, none.
Stimulus is not central planning or industrial policy. It should have no lasting effects. If it does, then you did it wrong.
In his latest reply, he says this:
Thus monetary policy should have no large predictable effects on the economy. There are always butterfly effect type stories we could tell – a crucial company getting funding at just the right time, etc – but from our perspective this is white noise. We could just as easily crush the butterfly as set him free.
If Smith means the first sentence literally as written – that there should be no large predictable effects on the economy – then I obviously agree, as this is a simple restatement of what I have said. I’ll assume, subject to correction, that what Smith means by this paragraph is something like the following: The net path dependent effect of a stimulus action on total economic output over a period of decades is drawn randomly from a conceptual distribution that (i) is centered around zero, and (ii) will not produce impacts that are of material size.
If that’s a roughly correct interpretation, then it’s not at all obvious to me that it’s true. My request to Smith is, once again, simple: Prove it.