Putting Social Science Claims to the Test: Stimulus Edition
In early 2009, the United States was engaged in an intense public debate over a proposed $800 billion stimulus bill designed to boost economic activity through government borrowing and spending. James Buchanan, Edward Prescott, Vernon Smith, and Gary Becker, all Nobel laureates in economics, argued that while the stimulus might be an important emergency measure, it would fail to improve economic performance. Nobel laureates Paul Krugman and Joseph Stiglitz, on the other hand, argued that the stimulus would improve the economy and indeed that it should be bigger. Fierce debates can be found in frontier areas of all the sciences, of course, but this was as if, on the night before the Apollo moon launch, half of the world’s Nobel laureates in physics were asserting that rockets couldn’t reach the moon and the other half were saying that they could. Prior to the launch of the stimulus program, the only thing that anyone could conclude with high confidence was that several Nobelists would be wrong about it.
But the situation was even worse: it was clear that we wouldn’t know which economists were right even after the fact. Suppose that on February 1, 2009, Famous Economist X had predicted: “In two years, unemployment will be about 8 percent if we pass the stimulus bill, but about 10 percent if we don’t.” What do you think would happen when 2011 rolled around and unemployment was still at 10 percent, despite the passage of the bill? It’s a safe bet that Professor X would say something like: “Yes, but other conditions deteriorated faster than anticipated, so if we hadn’t passed the stimulus bill, unemployment would have been more like 12 percent. So I was right: the bill reduced unemployment by about 2 percent.”
Another way of putting the problem is that we have no reliable way to measure counterfactuals—that is, to know what would have happened had we not executed some policy—because so many other factors influence the outcome. This seemingly narrow problem is central to our continuing inability to transform social sciences into actual sciences. Unlike physics or biology, the social sciences have not demonstrated the capacity to produce a substantial body of useful, nonobvious, and reliable predictive rules about what they study—that is, human social behavior, including the impact of proposed government programs.
So begins a modified excerpt in the current City Journal from my upcoming book.
As if on cue, the Wall Street Journal has an article about the debate concerning stimulus. The WSJ article opens:
Eighteen months after President Barack Obama administered a massive dose of spending increases and tax cuts to a weak economy, a brawl has broken out among economists and politicians about whether fiscal-stimulus medicine is curing the illness or making it worse.
…
But today, neither side can say with certainty whether the latest stimulus worked, because nobody knows what would have happened in its absence.
…
The Obama administration is stocked with heirs of Mr. Keynes, including academics Christina Romer and Mr. Summers. Ms. Romer famously projected in January 2009 that without government support, the unemployment rate would reach 9%, but with support the government could keep it under 8%. It’s 9.5% today.
Some Obama administration officials privately acknowledge they set job-creation expectations too high. The economy, they argue, was in fact sicker in 2009 than they and most others realized at the time. But they insist unemployment would have been worse without the stimulus.
My article goes on to argue that very few of the many potentially useful non-obvious predictions made so confidently by social scientists in fields ranging from economics to criminology to education can survive rigorous controlled experiments, and tries to describe how the experimental revolution is now coming to this discipline. I also try to outline a few lessons I think we have learned from social science experimentation so far. Much more on this will be coming later.
When all else fails we can use reason. Reason suggests, based on human behavior, that short-term stimulus in an uncertain environment will not create real wealth and confidence, because the temporary jobs and benefits can’t be depended on (except if unemployment is extended for a long, long time, but even then consumption based on this small amount won’t be great — see below).
Reason also tells us that in a recession, establishing the structure for myriad regulations which can increase the cost of doing business creates uncertainty among business owners who have to be able to project costs before expanding and hiring. The combination of stimulus and regulation and uncertainty encourages thrift, efficiency and productivity gains, but it doesn’t encourage hiring and expanding business. On top of it, extending unemployment and other benefits indefinitely locks many low wage earners in a situation where leisure is as valuable as expending energy, and human nature will choose a state of rest over expending energy if the pay-off is practically equal. To break out of this static situation, small and medium size businesses have to be confident that expanding and hiring will be worth the effort, then workers at the bottom scale will have to be motivated by receiving significantly more money for expending energy than doing nothing, or else benefits will have to be allowed to expire. If there is no political will to let benefits expire, even if businesses begin gaining confidence, they will have a hard time filling low-paying positions. We could see around 10-15% unemployment for years to come, and stimulus will only support the problem, not resolve it.
We’re trying to get out of a mess caused by government regulation and planning through more government regulation and planning, rather than facing the fact we’ll have to dig our way out with some suffering. Creating a free market environment is the quickest way back, but too many people are convinced a free market is the enemy, so government will be relied on until we collapse. That’s not a happy ending, though.
— Mike Farmer · Aug 2, 03:19 PM · #
Do you go into exploring mixed-design studies that utilize both quantitative and qualitative data? I have a background in social science (specifically social welfare) research, and I found that a lot of my field’s claims to being data-driven were pretentious: a lot of what we do defies being easily quantifiable.
— Erik Vanderhoff · Aug 2, 05:05 PM · #
Putting aside Mike Farmer’s Mirror Universe Marxism, I don’t think there’s quite the debate among intelligent folks over the stimulus that you make out. There is some heated disagreement over effectiveness and the precise nature of the stimulus…but serious thinkers contending that the response to the economic and financial collapse should have been nothing or a mix of only tax and spending cuts? Not so much.
Mike
— MBunge · Aug 2, 06:54 PM · #
MBunge,
I couldn’t help but notice you didn’t dispute what I wrote. Plus, I don’t know of anyone who recommended doing nothing or only tax and spending cuts, yet plenty of serious thinkers recommended tax and spending cuts plus rolling back harmful regulations and creating a free market environment where economic growth can take place. Just because the “serious” thinkers you read agreed with the stimulus doesn’t mean they weren’t seriously wrong. I could simplify the issue and state they aren’t serious thinkers because all they ever recommend is government spending, but that would be unfair, wouldn’t it?
— Mike Farmer · Aug 2, 07:56 PM · #
“plenty of serious thinkers recommended tax and spending cuts plus rolling back harmful regulations and creating a free market environment where economic growth can take place.”
The folks you’re talking about are not serious thinkers. The idea that one of the chief responses to an economic collapse should have ANYTHING to do with regulations is, bluntly, insane. The utility of this or that regulation can be debated, but confronting a massive recession by cutting regulations is like trying to put out a forest fire by playing a harmonica.
Furthermore, the idea of reducing public spending at a time when private spending is on the decline may have a purile appeal, but the result of such action is pretty much one of the few things that basically anyone who knows anything about economics agrees on. There may be some debate over tax cuts, but no intelligent, educated person thinks government spending cuts are an appropriate reaction to an economic downturn.
These are not exactly diffcult concepts to grasp. But when you’re as consumed by your ideology as totally as the most devoted communist ever was, it can be hard for you to tell the difference between up and down or black and white.
Mike
— MBunge · Aug 2, 08:25 PM · #
Mr. Manzi,
You might find these works useful in writing your book. Note that only Young 2009 speaks directly to macro-economics, the rest are about social science generally.
Freese, Jeremy. 2007. “Reproducibility Standards in Quantitative Social Science: Why Not Sociology?” Sociological Methods and Research 36:153-172
Gerber, Alan S., and Neil Malhotra. 2008. “Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals.” Quarterly Journal of Political Science 3:313-326.
Lieberson, Stanley. 1987. Making It Count: The Improvment of Social Research and Theory. Berkeley Calif.: University of California.
Lieberson, Stanley, and Joel Horwich. 2008. “Implication Analysis: A Pragmatic Proposal for Linking Theory and Data in the social Sciences.” Sociological Methodology 38:1-50.
Morgan, Stephen, and Winship, Christopher. 2007. Counterfactuals and causal inference : methods and principles for social research. New York: Cambridge University Press.
Young, Cristobal. 2009. “Model Uncertainty in Sociological Research: An Application to Religion and Economic Growth.” American Sociological Review 74:380-397.
— gabriel · Aug 2, 08:56 PM · #
“The idea that one of the chief responses to an economic collapse should have ANYTHING to do with regulations is, bluntly, insane. The utility of this or that regulation can be debated, but confronting a massive recession by cutting regulations is like trying to put out a forest fire by playing a harmonica.”
Why?
“Furthermore, the idea of reducing public spending at a time when private spending is on the decline may have a purile appeal, but the result of such action is pretty much one of the few things that basically anyone who knows anything about economics agrees on.”
Not if the spending is wasteful and misdirects capital from more productive, wealth generating pursuits. Consuming resources is not the same as producing and generating new wealth.
“There may be some debate over tax cuts, but no intelligent, educated person thinks government spending cuts are an appropriate reaction to an economic downturn.”
Yes, there are.
“These are not exactly diffcult concepts to grasp. But when you’re as consumed by your ideology as totally as the most devoted communist ever was, it can be hard for you to tell the difference between up and down or black and white.”
You’ve got me there — I do believe in what I believe in, but I know up from down, and black from white. I know you still haven’t disputed what I actually wrote.
— Mike Farmer · Aug 2, 09:11 PM · #
gabriel,
Thanks very much.
— Jim Manzi · Aug 2, 09:15 PM · #
“I know you still haven’t disputed what I actually wrote.”
Only in the sense that what you wrote was mindless cant which you cannot conceive of as being disputable. You’re not offering up any factual evidence or even intelligible theory. You’re just repeating the same things people like you offer up as the response to any and all economic circumstances. You’re restating ideological precepts, not prescribing effective policy. Hence the reference to Mirror Universe Marxism.
Mike
— MBunge · Aug 2, 09:52 PM · #
“Only in the sense that what you wrote was mindless cant which you cannot conceive of as being disputable.”
Oh, I see. You can’t dispute it — you can only try to degrade me with turgid prose.
— Mike Farmer · Aug 2, 10:25 PM · #
Let’s consider the larger issue of social science predictions.
It’s actually not very hard to make accurate predictions about social outcomes sas long as you make unpopular ones. I’ve been making predictions about educational achievement since 1972, and I have a very high accuracy rate. Of course, most of my predictions are of the boring and unpopular kind, such as that Compton’s schools will average lower test scores than Beverly Hills’ schools. I’ve been right about countless predictions like that for decades, and you can see how widely beloved it has made me.
— Steve Sailer · Aug 2, 11:10 PM · #
Jim asserts:
“Unlike physics or biology, the social sciences have not demonstrated the capacity to produce a substantial body of useful, nonobvious, and reliable predictive rules about what they study—that is, human social behavior, including the impact of proposed government programs.”
That’s not true. There is a huge number of predictive rules that are useful, reliable, and nonobvious, at least to our elites. For example, a few years, Mayor Bloomberg and NYC schools supremo Joel Klein decided to fix the ramshackle admissions process to the gifted schools by imposing a standardized test on all applicants. Half Sigma immediately predicted that the percentage of Asians and whites admitted would rise at the expense of blacks and Hispanics, which would cause a sizable unexpected political problem for Bloomberg and Klein. All that has come to pass.
This inevitable outcome have been obvious to Bloomberg and Klein from a century of social science data accumulation, but it clearly was not.
— Steve Sailer · Aug 2, 11:17 PM · #
Jim,
I’ve now read your City Journal article. It’s quite informative, but it skirts around the basic issue. The biggest problem with social science research is not methodological; it’s that we don’t like the results.
I’ve been a social science stat geek since 1972. We’ve been doing experiments, artificial and natural, along with correlation studies, since well before 1972. The essential flaw is not in the social science, it is in that the elites of America don’t like what the social sciences have uncovered about, say, crime, education, discrimination, immigration, and so forth.
What we’ve learned are things like:
- Race matters – Sex matters – Class matters – Heredity matters – Character matters
And, judging by what I read every day, we hate this knowledge. Those who explain how to use the knowledge uncovered by the social sciences are routinely villified. Men of outstanding character, heroes of the human sciences such as Charles Murray and Arthur Jensen are subjected to massive campaigns of demonization.
— Steve Sailer · Aug 3, 12:00 AM · #
Jim,
Since you and I have very similar backgrounds in marketing research, our lines of thinking are similar. Because I’m an old coot, however, I’ve been thinking about these important topics longer than you have.
You rightly say in your City Journal article, “Despite confidently asserted empirical analysis, persuasive rhetoric, and claims to expertise, very few social-program interventions can be shown in controlled experiments to create real improvement in outcomes of interest.”
Quite true.
That, however, does not prove your preceding contention:
“The most fundamental lesson that emerges from such experimentation to date is that our scientific ignorance of the human condition remains profound.”
In reality, after a century of experimentation and analysis, we know an awful lot about several major human conditions, such as IQ. What we don’t know about IQ is how to intervene to reliably narrow racial gaps in average educational achievement (other than hitting Asian and white kids on the head with a ballpeen hammer).
Much the same is true regarding many other topics. As you rightly say,
“First, few programs can be shown to work in properly randomized and replicated trials. …
“Second, within this universe of programs that are far more likely to fail than succeed, programs that try to change people are even more likely to fail than those that try to change incentives.
“And third, there is no magic. Those rare programs that do work usually lead to improvements that are quite modest, compared with the size of the problems they are meant to address or the dreams of advocates.”
Given all that, you can readily arrive at my oft-repeated point of view, which is that the one potential government policy that is likely to have major long term effects on social problems, for good or for bad, is immigration. Since government policies can seldom change people, government policy should select immigrants who will likely benefit the current citizens of the nation and keep out those who likely won’t.
This has been the philosophy of Canada’s immigration system for decades, and it has worked fairly well for them. Yet, advocates of a Canadian-style immigration policy philosophy for America are routinely subjected to Two-Minutes Hates in this country. Thus, social policy discourse in the U.S. is puerile and sentimental.
— Steve Sailer · Aug 3, 12:41 AM · #
Steve,
Thanks for the series of great comments.
I think that “non-obvious” is an important descriptor of the kind of social predictive rules that we require “science” to uncover. This raises the question of “obvious to whom?” and I spend a lot of time on this in the book (the answer is slightly complicated, and related to the next post in this sequence in which I reply to Mark Kleiman).
When I said that “the most fundamental lesson that emerges from such experimentation to date is that our scientific ignorance of the human condition remains profound”, I meant that to be evidenced by the fact that we find it almost impossible to execute counter-intuitive social programs that can demonstrate in replicated RFTs that they cause material changes in targeted outcomes.
You say in response that “In reality, after a century of experimentation and analysis, we know an awful lot about several major human conditions, such as IQ. What we don’t know about IQ is how to intervene to reliably narrow racial gaps in average educational achievement (other than hitting Asian and white kids on the head with a ballpeen hammer).”
But this is the kind of thing that I mean by “useful.”
Finally in response to your description of your preferred immigration policies, here is what I’ve written about the topic:
— Jim Manzi · Aug 3, 12:55 AM · #
Jim,
Thanks.
You say:
“I think that “non-obvious” is an important descriptor of the kind of social predictive rules that we require “science” to uncover. This raises the question of ‘obvious to whom?’ … we find it almost impossible to execute counter-intuitive social programs that can demonstrate in replicated RFTs that they cause material changes in targeted outcomes.”
Yes, but what’s obvious and intuitive to you and me — that, for example, “illegal immigration of a low-skilled population from Latin America” is bad for the U.S. — was not obvious and intuitive to, say, George W. Bush, John McCain, Ted Kennedy, and Karl Rove.
Similarly, it’s obvious and intuitive to you and me that the No Child Left Behind act was always more likely to lead to cheating on tests than to actually closing the racial gaps in school achievement, but that wasn’t obvious and intuitive at all to George W. Bush and Ted Kennedy.
The central problem is that acting oblivious to the obvious is crucial to winning at the status game in 21st Century American culture. In contrast, noticing what your lying eyes are seeing leads to being denounced as stupid and evil.
If you are looking for a real challenge for the social sciences to tackle, them they should do experiments on the formation and retention of Elite Mythologies.
— Steve Sailer · Aug 3, 04:31 AM · #
Steve,
I don’t consider ““illegal immigration of a low-skilled population from Latin America” is bad for the U.S.” to be an obvious statement. There are many things that I believe to be true that do not rise to the level of obviousness (in the sense that I use the term in the book and essay). Almost any statement that is the subject of widespread politcial debate in the contemporary U.S. would not be something that I mean when I say “obvious”.
— Jim Manzi · Aug 3, 07:25 AM · #
But there isn’t much “widespread political debate in the contemporary U.S.” on the long term impact of letting in millions of illegal immigrants. There is simply one side citing social science statistics and the other, dominant side in the media, screaming That’s racist! Those are HateStats!
As your article documents, we mostly don’t know how to fix the problems caused by massive low-skilled immigration. But that doesn’t stop the conventional wisdom from exclaiming, “All We Have To Do Is Fix …
- the Schools! – Housing! – Carbon Emissions! – Inequality! – The Budget! – Traffic!
What should be obvious is that when you find yourself in a hole, the first thing to do is to stop digging. That’s not to America’s elites, however.
— Steve Sailer · Aug 3, 08:32 PM · #
Immigration fix in nine easy (and politically feasible) steps, and two hard ones:
1. Upon application and fee, grant permanent resident status to all unfelonious persons in the country legally who hold a valid work visa issued on or before August 1, 2010. (They followed the rules and should be rewarded.)
2. Simplify and accelerate the application process for work visas. Eliminate the sponsorship requirement in favor of consulate interviews, polygraph testing, fingerprinting and/or recommendation from applicant’s local constabulary. Raise the fee and shorten the time. Presume applicant’s success. Delegate in full all final decisions regarding applications to individual consulates. Install a training and grading program for interviewers. Recruit interviewers from local population. Create oversight agency to keep track of statistics that are traceable to individual consulates (e.g., how many of Consulate X’s entrants commit violent crime) and sack any administrator who regularly allow too many net-negatives into the country.
3. Expand consulate services and locations to meet demand. Install counselors at each location to help aspiring applicants navigate the system, become eligible.
4. Build a real fence on the border. Crack down hard on remaining illegals.
5. Blitz the airwaves with Spanish language advertisements explaining the new system.
6. Eliminate all internet and US-located application sites. Require all visa applicants to apply in person in their country of origin or country of domicile.
7. Make use of biometrics on the work visa identification card. Make use of implied consent law to enforce compliance with various provisions.
8. Prohibitively criminalize the hiring of illegitimate workers. Normalize and streamline the ability of employers to do their due diligence. Keep track of employment. Enforce minimum wage. Deport visa holders if unemployed after six months, make wait two years for new visa. Deport any visa holder who commits a felony, bar reentry.
9. Embrace an ethic of regulated entry for all, with no administrative penalty for denying entry with cause. “Cause” may be shown via one page affidavit. If cosigned by supervisor, no penalty may issue for rejection of application.
10. End the war on drugs, replace criminalization regime with a therapeutic regime.
11. Repeal ObamaCare.
If your problem with immigration is 1) criminality, 2) chaos, 3) anonymous entry, 4) wage depression, 5) rule breakers and free riders, and/or 5) a shadowy underclass, then this plan should work for you. If your problem is with brown people, spanish speakers, racial dilution, or “unskilled, tired, poor, huddled masses yearning to breathe free,” then I’m relatively certain that your jolly’s won’t get rocked by this proposal.
— KVS · Aug 3, 10:04 PM · #
Great article written on my life, help, Thank you for your article, hope to become friends.
— cheap wholesale clothes · Aug 4, 08:10 AM · #