Knowledge and Prejudice
While he uses some pretty inflammatory language in reacting to my article on the foundations of social scientific knowledge, Mark Kleiman asks what I believe to be some very important questions.
Kleiman quotes me as saying:
we should be very skeptical of claims for the effectiveness of new, counterintuitive programs and policies, and we should be reluctant to trump the trial-and-error process of social evolution in matters of economics or social policy … we need to keep stumbling forward with trial-and-error learning as best we can.
This leads Kleiman to ask:
What is this magical “trial-and-error process” that does what scientific inquiry can’t do? On what basis are we to determine whether a given trial led to successful or unsuccessful results? Uncontrolled before-and-after analysis, with its vulnerability to regression toward the mean? And where is the mystical “social evolution” that somehow leads fit policies to survive while killing off the unfit?
I devoted a lot of time to this related group of questions in the forthcoming book. The shortest answer is that social evolution does not allow us to draw rational conclusions with scientific provenance about the effectiveness of various interventions, for methodological reasons including those that Kleiman cites. Social evolution merely renders (metaphorical) judgments about packages of policy decisions as embedded in actual institutions. This process is glacial, statistical and crude, and we live in the midst of an evolutionary stream that we don’t comprehend. But recognition of ignorance is superior to the unfounded assertion of scientific knowledge.
Kleiman then goes on to ask this:
Without any social-scientific basis at all (unless you count Gary Becker’s speculations) we managed to expand incarceration by 500 percent between 1975 and the present. Is that fact – the resultant of a complicated interplay of political, bureaucratic, and professional forces – to be accepted as evidence that mass incarceration is a good policy, and the “counter-intuitive” finding that, past a given point, expanding incarceration tends, on balance, to increase crime be ignored because it’s merely social science?
My answer is yes, it should be counted as evidence, but that it is not close to dispositive. We can not glibly conclude that we now live in the best of all possible worlds. I devoted several chapters to trying to lay out some principles for evaluating when, why and how we should consider, initiate and retrospectively evaluate reforms to our social institutions.
Kleiman’s last question is:
Should the widespread belief, implemented in policy, that only formal treatment cures substance abuse cause us to ignore the evidence to the contrary provided by both naturalistic studies and the finding of the HOPE RCT that consistent sanctions can reliably extinguish drug-using behavior even among chronic criminally-active substance abusers?
My answer to this is no, and a large fraction of the article (and the book) is devoted to making the case that exactly such randomized trials really are the gold standard for the kind of knowledge that is required to make reliable, non-obvious predictions that rationally outweigh settled practice and even common sense. The major caveat to the evaluation of this specific program (about which Kleiman is deeply expert) is whether or not the experiment has been replicated, as I also make the argument that replication is essential to drawing valid conclusions from such experiments – the principle that Arnold Kling called in a review of the article, “Don’t trust one-offs.”
Philosophy of science… some deep waters to wade into. You guys might want to get your scuba gear before getting involved in this debate.
— Console · Aug 3, 01:13 AM · #
Mark Kleiman writes:
“Without any social-scientific basis at all (unless you count Gary Becker’s speculations) we managed to expand incarceration by 500 percent between 1975 and the present.”
Nonsense. I read social scientist James Q. Wilson’s book “Thinking About Crime” in high school in 1975. Wilson laid out a sizable amount of social science evidence for the logical proposition that locking up more criminals for longer sentences will reduce how often those criminals prey on the general population. Wilson’s work was well-known and had much subsequent influence on the war on crime.
— Steve Sailer · Aug 3, 01:22 AM · #
Nassim Taleb is a big believer in trial and error and against the scientific method. He had some outrageous (but possibly true) factoid in one of his EconTalk interview that some outrageous (but possibly true) number of peer reviewed studies end up failing attempts at replication – I wish I could find the actual stat and the support.
IMHO, the point of trial and error is that you get different attempts. The US locks up 10 people in 1000, and France locks up 2 in 1000 (or whatever), and then you can discuss which system works. But if you announce to everyone that 10 in 1000 or 2 in 1000 is the right system, you never get to see whether the other system was better. Alternately, the US tries a stimulus, and Germany sits out. In both cases, you don’t have clean experimental data to say one is better than the other, even after to try them.
But if you announce that “science” best guesses that one is probably better so that’s the only one you try, then you’re stuck with that.
— J Mann · Aug 3, 01:47 PM · #
That would indeed be terrible, but if you announce that social science doesn’t really know anything and therefore we shouldn’t mess with the status quo so you won’t try anything new, then you’re just as stuck with that.
— Consumatopia · Aug 3, 03:51 PM · #
Consumatopia,
1) You’re right, and
2) Thanks for taking the time to parse that sentence, which I definitely could have written better.
You’re definitely right that if, for example, Gulliver tells the Lilliputians that acrobatic competitions are a lame way to pick cabinet officals, they would be ill-served to say “well, we arrived at our system through trial and error, so we reject your counterintuitive finding.”
But when Mark pretends not even to know what trial and error is, I find it a little goofy. Some communities lock up lots of prisoners. Some communities engage in early release, or release for good behavior. Not scientific, but not necessarily much less reliable than a regression someplace that shows that increasing incarceration rates, under some conditions, can increase crime.
Manzi’s proposal, if I understand it, is to design an experiment, then repeat it several times in different times and places. If you can do it ethically, it makes some sense.
— J Mann · Aug 3, 06:17 PM · #
Social policy experimentation has a long history. For example, Nixon’s chief domestic policy adviser in 1969-70, Daniel Patrick Moynihan favored Milton Friedman’s negative income tax for poor people idea, so a big experiment was done, but motivation to hold a job was lower in the families getting the negative income tax than in the control groups.
The social sciences have learned a lot over the years, just not what most social scientists hoped to learn.
— Steve Sailer · Aug 3, 06:28 PM · #
One half of my response can be gleaned from this 2007 essay found in ISJ.
The other half is a political cynicism that supervenes on everything else we know.
— KVS · Aug 3, 08:27 PM · #
KVS,
I’m just starting to work my way through that paper – any hints you want to toss out about it?
— Jim Manzi · Aug 3, 09:30 PM · #
Jim, the Quinean prototype metaphor maps onto the polis in a theoretically robust way (also onto subcomponents like agencies, criminal codes, and tax regimes). For your book I would pay special attention to the exploratory/experimental/evolutionary trichotomy. As a heuristic, this tripartite division is a useful if fundamentally arbitrary way of looking at things. Because the Quinean prototype is such a robust metaphor for a nationstate, this paper — an IS application of sophisticated epistemology — is especially germane to your discussion.
Without having read your book I can’t say how the paper intersects it, but I hope it helps (and think it might).
— KVS · Aug 3, 10:25 PM · #
Jim,
education is a field that does lots of studies. When I was in ed school we had a textbook of “best practices” that gave the best practices—concrete teaching techniqes— for each areas of study and classroom management. Have you looked at education studies, specifically those dealing with concrete techniques? If you have, what have you found?
Off the top of my head I would expect that education is a more succesful area for experimentation because schools, by there very nature, collect uniform populations into controled environments. Sort of like animal testing. (ha ha).
— cw · Aug 3, 11:40 PM · #
Education schools are notorious as perhaps the most anti-scientific institutions in the modern world, precisely because real social science has discovered facts like “IQ matters” and “race matters.” Since the dominant national goal for education is closing the racial gaps, and that is as unlikely as inventing a perpetual motion machine, pretty much any kind of nonsense is encouraged in the Ed School biz.
— Steve Sailer · Aug 3, 11:47 PM · #
God bless you, Steve! Every time I start to think I’m lost in my own little world and seeing everything through my own twisted prism, you come bounding and letting me know that how every deep in the weeds I am, at least I can still see the fairway!
— Tony Comstock · Aug 4, 12:45 AM · #
Steve Sailer,
I am pretty dang sure that most of the studies that you use to conclude that there are racial differences in IQ have exactly the flaws that Jim Manzi points out in his article: extreme casual density and use of regression analysis to control for variables.
— cw · Aug 4, 01:09 AM · #
Pause, while Steve looks up “regression analysis” on Wikipedia.
— Pause Fill · Aug 4, 03:25 AM · #
“I am pretty dang sure that most of the studies that you use to conclude that there are racial differences in IQ have exactly the flaws that Jim Manzi points out in his article: extreme casual density and use of regression analysis to control for variables.”
Can you figure out the obvious logical error in this statement?
— Steve Sailer · Aug 4, 04:23 AM · #
I am terrible at puzzles. I need a hint. Is this logical error a rhetorical problem or is it something that makes the statement untrue?
— cw · Aug 4, 05:13 AM · #
Whether or not “there are racial differences in IQ” is not a causal question, it’s an empirical question.
This is an extremely common mistake that says a lot about the unspoken assumptions of many of the people making the mistake.
The most comprehensive investigation of the empirical question was carried out by Philip L. Roth of Clemson and colleagues in a 2001 article, “Ethnic Group Differences in Cognitive Ability in Employment and Educational Settings: A Meta-Analysis,” in the academic journal Personnel Psychology.
They looked at 105 different studies covering 6,246,729 individuals and found an overall average difference between whites and blacks of 16.5 IQ points, or 1.1 standard deviations. The 95 percent confidence interval runs merely from 1.06 to 1.15 standard deviations (in other words, there is strong agreement among the 105 studies).
These 105 studies summarized by Roth aren’t regression analyses wrestling with “extreme causal density.” These are simple averages and variances.
As for what are the causes of these racial differences in IQ, well, we’re going to find out, sooner or later. But, in the meantime, we can’t just wish them away, or mandate that we will (somehow) abolish by 2014, as President Bush’s and Senator Kennedy’s NCLB law states.
— Steve Sailer · Aug 4, 05:48 AM · #
I like your style so much, I am your honest reader…
— designer clothes · Aug 4, 08:10 AM · #
cw:
K-12 schooling is, in theory, an almost ideal candidate for testing. Unfortunately until about 10 years ago, very little had been done (Cook at Northwestern identified exactly 3 RFTs throughout the 60’s and 70s), and as recently as 2002, could identify no RFTs executed against most of the key methodological disputes in the field. During this era many more experiments were done in welfare economics and criminology. This is plausibly due to the ideology of educational institutions, combined with the greater political power of teachers and parents than of indigents and criminals.
In the past decade, the DOE has established the IES (Institute for Education Sciences) as its key research arm, and it has focused heavily on RFTs as the gold standard of evaluation. Mostly what these have shown so far, consistent with the introduction of randomized experiments to therapeutics, business, criminology and social welfare, is that the vast majority of widely-touted pedagogical and other programs don’t work.
Quasi-RFTs, as is well-known, have been run in New York, Washington, DC, Dayton, Milwaukee, and Charlotte because of lotteries for admissions to charter and voucher schools. While these have randomization, they are not true experiments, and many seemingly-minor differences between these procedures and actual experimental procedures have a big impact on how to interpret them.
— Jim Manzi · Aug 4, 09:35 AM · #
@KVS,
Okay, because you said it was worthwhile, I read the whole damn thing. But I gotta say, that was a long damn walk for “Lawyers make for quarrelsome, meddlesome clients.”
(Standard practice in my line is to charge lawyers 2.5 times what you’d charge anyone else for the same work; and even then it’s not worth it half the time.)
— Tony Comstock · Aug 4, 12:41 PM · #
Jim Manzi,
Thanks.
Steve Sailer,
You got me there. I should have said, “…studies you use to conclude that racial diffenrences in IQs are the result of genetic differences between the races are flawed….”
— cw · Aug 4, 02:28 PM · #
“I should have said, “…studies you use to conclude that racial diffenrences in IQs are the result of genetic differences between the races are flawed….”
But I haven’t concluded that. What I have concluded is that racial differences in IQ are extraordinarily pervasive and stable, so public policies, such as Bush-Kennedy’s NCLB, that assume they can be made to go away Real Soon Now are likely to fail.
I actually have thought about the unthinkable a lot and hold much more sophisticated ideas than you have exposed to, so you could save yourself a lot of embarrassment by reading what I’ve actually written. A good place to start is my IQ FAQ:
http://www.vdare.com/sailer/071203_iq.htm
— Steve Sailer · Aug 4, 07:47 PM · #
Steve, I read your FAQ and some of your other writings as well. Are you really describing your ideas as “unthinkable”? Maybe it’s my lack of sophistication, but history seems littered with ideas so similar as to be indistinguishable from yours.
— Tony Comstock · Aug 5, 12:56 PM · #