Just a quick thing before this week’s post. If you’ve preordered THE STRANGEST IDEA IN THE WORLD, I’d like to send you a little something extra in the mail. Submit your preorder receipt and your mailing address here. This is only available to folks with US or UK addresses—apologies to those who live elsewhere! In the US, everyone with a valid entry will get one; in the UK they’ll pick winners randomly. The link will be open until December 11, and we’ll start mailing things out after that. I won’t spoil the surprise, but I will say that a) it’s very silly, and b) I can’t wait to get mine.
(For the US: US Residents, 18+. Ends December 11, 2026. See terms at https://sites.prh.com/strangestidea-preorder)
Anyway, onto the post!
Before anybody enacts a major social policy or launches any philanthropic programs, they should sacrifice a goat. There ought to be an entire liturgy of grotesqueries, in fact—pricked fingers, names written in blood, robes, candles, chanting, obsidian blades, still-beating hearts, the whole shebang.
The point of this ritual is not to signal our obeisance to the gods (although if any of them want to stop by and help us out, that would be cool), but to remind ourselves that we know nothing. This is procedurally-induced humility, which is desperately necessary, because an alarming number of our best-intentioned interventions do not work.
For example:
In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.
Universities that hired a Chief Diversity Officer between 2001 and 2019 did not end up with more diverse students, faculties, or administrators.
The American government spent $280 million to help Afghan women get jobs, and now they can’t really say whether any women got any jobs.1
Remember 20 years ago, when giving laptops to kids in developing countries seemed like the most important thing to do? Well, a 10-year RCT of 531 rural schools in Peru found zero effects of “One Laptop Per Child” on academic performance.2
Researchers developed four different messages that were supposed to convince parents to vaccinate their kids. None of the messages worked, and one of them accidentally made parents more likely to believe that vaccines cause autism.
Most of those are left-coded failures, so here are some right-coded failures:
When Arkansas added a work requirement to Medicaid, the number of Arkansans with health insurance went down without any increase in employment.
In 2002, the Bush administration launched the Building Strong Families Program, which provided counseling to low-income, unmarried parents with the goal of strengthening their relationships, keeping them both in their child’s life, and ultimately getting them married. Three years later, an independent evaluation found that the program did none of those things.
And here is the effect of four different abstinence-only sex ed programs that run across the United States:
In the social sciences, nothing ever dies; it merely becomes embarrassing. We can’t say that it’s impossible for anything like these programs to ever succeed, given enough time, flimsy theorizing, and generous definitions of “success”. The point is that these effects should be large, and instead they are nonexistent. Seems to me like it’s time to get the goats in here.
HEAVY METAL
It’s inevitable that some of our interventions are going to fail. (If they don’t, we should either be suspicious of the results, or we should be more ambitious with our programs.) What we really want to know is: when we try to improve the world through some policy or program, how often do we succeed?
We don’t have a good answer to this question, but we do have a good joke about it.
In 1987, the sociologist Peter Rossi wrote a tongue-in-cheek paper that accidentally became his intellectual legacy. He would later refer to it as “far from being one of my favorites” and “the source of considerable embarrassment”, and yet it’s the only thing of his that I’ve ever read or even heard of.3 It was called “The Iron Law of Evaluation and Other Metallic Rules“, and it went like this, lightly paraphrased:
The Iron Law of Evaluation: The net expected impact of a large social program is zero.
The Stainless Steel Law: Better-designed evaluations are more likely to find null effects.
The Brass Law: The more a program is intended to change people, the more likely the impact of that program will be zero.
The Zinc Law: The programs most likely to be evaluated are those most likely to fail.
Rossi tried to walk back his laws shortly before he died in 2006, insisting that there is now a “large number of well conducted impact assessments that yield statistically and substantively significant effect sizes”. Still, even after supposedly enduring 16 years of embarrassment from his one satirical paper, he concluded, “we really do not know whether programs over a wide variety of substantive areas are worth anything”.
Rossi did not do us the favor of actually naming any of those “statistically and substantively significant effect sizes”, and I know why, because I have gone looking for them, and they are not easy to find. The Earned Income Tax Credit, which is designed to benefit low- and middle-income families, was once considered a slam dunk; now a new analysis suggests that it might not do anything. Plain packaging—where you strip cigarette boxes of their attractive branding—seems like a no brainer, but in Australia, it looks like the policy caused smokers to switch to cheaper brands and to smoke more. There’s a reading and math program that has done well in India, but as soon as other researchers tried to implement it elsewhere, the effects shrank.
You can always find a one-off study showing that an intervention works, but if we’ve learned anything from the past 20 years of social science, it’s that one-off studies ain’t worth much. It’s so easy to juice your stats and journals are so hungry for positive results that entire literatures can be suspect, and decades of research can culminate in a shrug. “Sometimes it works and sometimes it doesn’t, it depends on all sorts of factors that we can’t really specify, so uh, good luck!”
Even if we take Rossi at his word and there are as-yet-unnamed programs out there that are reliable and effective, the question still remains: why aren’t there more of them? Why do so many of our best-laid plans end up like this:
OJ A-OK
The answer lies, I think, in a 50-year-old study that is almost certainly false.
One of my favorite genres of psychology experiment is called something like: “the study is probably bullshit but the effect is probably real”. At the top of that heap is The Illusion of Control, a classic from 1975 and a favorite of Psych 101 classes everywhere. Usually, this study gets summarized as “people value their lottery tickets more when you let them pick the numbers vs. when you pick the numbers for them.” It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4
I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5
Unsurprisingly, well-powered, preregistered replications with actual lotteries have failed to find any difference in picking vs. not-picking. So that’s that: there’s no such thing as the illusion of control.
Except...obviously there’s an illusion of control! Look around! People will blow on their dice at the craps table, they’ll don their lucky undies when the Knicks go to the finals, they’ll scream “look out!” to the hapless protagonists of a horror movie. When I was a kid, my parents used to send me off to play arcade games without any quarters, knowing that I would simply move the joystick in synchrony with the demo playing on the screen, convincing myself that I was causing the characters to move, when in fact I was merely mirroring them.
Put it this way: why do you think The Secret has sold over 30 million copies? It ain’t the beautiful prose or the sterling scientific evidence. It’s because when Rhonda Byrne tells you that “your thoughts are the primary cause of everything,” it feels kind of true. After all, can’t you jinx yourself by thinking the wrong thing? Haven’t you ever been like “whatever you do, don’t spill this soup” and then the next thing you know you’ve got a shirtful of bisque? And if brainwaves are powerful enough to spill soup, who knows what else they might do!
So no, people don’t believe they can magically improve their chances of winning the lottery by picking the numbers themselves. Still, our intentions feel more potent than they should, and this shows up not only in our harmless superstitions, but also in the way we keep meddling in each other’s lives despite having no idea what we’re doing.6
How do I know my preferred policies will have their intended results? Well, because I intend them. That’s not really my answer to the question; it’s the reason I don’t even think of the question in the first place.
GEORGE W. BUSH, PROPHET OF OUR TIME
Here’s the real evidence of our illusion of control: we’re so certain our interventions will work that we forget to check whether they do at all.
For instance, the National Science Foundation funds over 1,000 STEM education projects, which are aimed at improving curricula, training, student outcomes, etc. According to an analysis by Stuart Buck, 76% of the time, the researchers running those programs don’t even attempt to evaluate them with actual experiments. Apparently if you have a bright idea about how to teach kindergarteners to code, you don’t need to bother assessing it. It turns out George W. Bush was actually right when he said:
Speaking of the government, according to a report from the Government Accountability Office, many federal programs don’t even define their goals, let alone collect data on their progress toward those goals. For instance, the Department of Homeland Security is supposed to stop people from producing and distributing illicit fentanyl, and so they do things like search people’s cars and walk around with drug-sniffing dogs. When DHS was asked, “How do we know any of this is working? What would success even look like?”, they were like “hmm great questions”. They hadn’t set any goals or collected any data, despite being required to do so by law. As the GAO puts it with characteristic mildness: “As a result, DHS’s ability to fully understand the effectiveness of these efforts is limited.”
Maybe it’s too much to expect that the Department of Homeland Security, especially in its current incarnation, would try to figure out whether it’s doing things that actually work. But it’s certainly not too much to expect of private philanthropists, who choose to give away their money of their own free will. Certainly they would care whether their dollars do anything. Right?
Wrong. In 2018, Kenneth Langone, the Home Depot billionaire, donated $100 million to eliminate tuition fees at New York University’s main medical school. This was supposed to encourage med students to pick lower-paying specialties like pediatrics and primary care, and it was supposed to attract lower-income students to medical school in the first place. But neither of those things happened. The makeup of the student body hasn’t changed, and they’re picking the same specialities they always did.
In this case, it should have been easy to foresee that those beaucoup bucks were going to do bupkis. First, medical school is already a good investment, regardless of income: yes, you take on a big pile of debt, but you also make a much bigger pile of money, even in the lowest-compensated specialties. Second, wiping out that debt does almost nothing to reduce the disparities between specialties: the average orthopedic surgeon earns $564,000 per year, while the average family practitioner earns about $281,000. If both of them work for 30 years, the surgeon earns about $17 million, while the family doc earns about $8 million. Meanwhile, the average medical school debt is about $200k, which is nothing compared to an extra $9 million over the course of your lifetime.
And remember: in this situation, both the surgeon and the family doc attend for free, regardless of how much money they had when they arrived, or how much money they make after they leave. So there’s no reason why free tuition should encourage anybody to do anything differently—except to apply to NYU in the first place. And indeed, applications are up 47%.7
To be clear, giving away dunderheaded donations is better than a lot of things that billionaires could be doing with their money, like buying a second superyacht or digging an additional doomsday bunker so they have a place to summer after the apocalypse. I’m glad that the plutocrats are trying to do something nice. I would just like them to succeed.
THE GREAT BARRIER, GRIEF
When people see our big heap of failed interventions, they tend to go through three stages of grief.
Bargaining
Okay, fine, these high-minded, utopian interventions don’t work, so what we really need is Common Sense and Obviously Good Ideas. For example, if you want to make poor people better off, you should give them money. That’s just Common Sense! That’s an Obviously Good Idea!
Unfortunately, large, well-designed randomized controlled trials keep finding null effects of unconditional cash transfers. In one study, when low-income participants got $2000 with no strings attached, it made them less happy compared to people who got $500 or nothing at all. Psychologically, receiving 20 Benjamins was about 25% as bad as losing your job during the pandemic. That finding might be a fluke, but it also showed up in a study where researchers wiped out a bunch of people’s medical debt and it maybe made them more depressed.
These programs do succeed in sense that when you give people more money, they have more money, and they spend it on things that they need. It’s just surprisingly difficult to find any evidence that having more money makes a difference. (Though note that results are more promising in low- and middle- income countries.)
Denial
This is where people typically raise the no true Scotsman defense: if we get bad results from a good idea, it must be because someone screwed something up, not because the good ideas were actually bad ideas. Maybe the measures were wrong, the implementation was shoddy, or the people who benefitted didn’t fill out the followup survey for some reason.
This response isn’t totally irrational; programs go pear-shaped all the time. Maybe your One Laptop Per Child program only failed because all the laptops got stolen out of the back of the truck. Case in point: in that medical debt study, it seems like 76% of people didn’t even realize their medical debt had been wiped out. (But don’t get too excited: if anything, calling people and informing them of this boon made them slightly more depressed.)
The real problem with no-true-Scotsman-ing these studies is that you must subscribe to the dubious belief that some tiny tweaks will somehow unlock giant effects. This is a holdover from pre-replication-crisis psychology, back when we thought you could, say, subliminally flash an American flag at someone and make them more likely to vote for a Republican two weeks later. It now seems like rinky-dink interventions tend to have rinky-dink effects, or none at all. So yes, we could always pick more appropriate measures, and the facilitators delivering the intervention could always have a bit more pep in their step. That might be enough to make a program slightly more effective, but it’s not enough to turn a null result into a blockbuster.
Acceptance
Acceptance sounds good, but it isn’t. Once we realize how hard it is to improve people’s lives, it’s tempting to believe that doing so is simply impossible, and so we are excused from even trying.
This is a bit like finding out that bloodletting doesn’t work and then concluding that no medicine can ever work. (“Well, We’ve Tried Everything”, Says Society That Has Only Tried One or Two Things.) The social sciences are still in their alchemical stage. We do not understand the fundamental workings of the human mind, let alone what happens when those minds combine in societies and economies. Imagine trying to cure cancer using Galenic medicine or trying to land on the moon using Aristotelian physics—that’s what we’re doing when we roll out programs designed to change human behavior.
FULL THROTTLE ARISTOTLE
That doesn’t mean our ill-informed attempts can never succeed. Both Galen and Aristotle were right about some important things: it was Galen who figured out that urine is produced by the kidneys, rather than the bladder (as was commonly assumed), and it was Aristotle who successfully described how objects move in fluids, even if he attributed it to the wrong forces.
But if we want to solve real, Hydra-headed problems, we need more than a handful of facts. (“Doc, I’m all itchy and covered in spots! What do I do?” “Hmm...did you know that urine is produced in the kidneys?”) Until psychology and its neighboring disciplines grow into scientific maturity, all we can do is guess and check. There’s nothing shameful in that, but there is something shameful in pretending otherwise.
This is hard for us to internalize, because by default we assume that we were born near the end of history. We understand, of course, that we still have a few things to figure out, but emphasis on the few. Just look at our beautiful universities, our begowned scholars, our numerous and inscrutable papers! How can you say we know nothing when we have so many TED talks?
We misunderstand what it feels like to be in the middle of things. Almost no one who read Galen or Aristotle ever thought to themselves, “oh, obviously this is only version 0.1 of our knowledge, all of this is tentative and incomplete, and one day it will be overturned and replaced with something better, something that actually works.” No, they thought to themselves, “Thank u Mr. Aristotle for your many true and useful facts.”
That’s why the worst possible reaction to a failed intervention is resignation. We may not know enough to improve our lot right now, but perhaps we could figure it out, if we tried. Maybe that means another RCT, or maybe it means another century of futzing around in the fundamentals before we can do anything useful.
(This does put a ceiling on the utopia we can expect to achieve in the near future. If you believe that we can perfect human nature by reorganizing society to your specifications, you would do well to remember that, as of now, we can’t even get people to go to the gym.)
That may seem unfair—what a bummer to be born in medias res, when so many of our problems remain unsolved! But if it is an affront, every human before us has been similarly affronted, and some of them chose to begin the work anyway. Whenever we pluck a low-hanging fruit today, it’s only because someone else planted a seed that they never got to harvest. So why should we expect to pluck without planting?
As Seneca put it nearly 2,000 years ago:
I revere the discoveries of wisdom and their discoverers [...] For me those things were acquired, for me they were labored for. [...] Let us make larger what we have received. Let this heritage pass from me to posterity increased.
I know some people don’t feel this way. Some believe that the answers are simple and known and merely need to be enacted by force. Others think the answers are unknowable and will remain so forever. For those folks, I offer an invitation: the next time we conduct the ritual before undertaking some new policy, they may have the honor of playing the part of the goat.
Of course, that sum is negligible compared to $4-6 trillion we spent trying and failing to turn Afghanistan and Iraq into thriving democracies.
How times have changed: I know parents who spend hours commuting every day just so their kids can attend a school that won’t issue them a laptop.
Other times an academic made a joke that went on to become their most prominent work:
In 2009, four neuroscientists published a poster—not even a paper!—showing that, with fMRI techniques that were standard at the time, it was possible to detect activity in the brain of a dead salmon. As far as I can tell, this is one of the most-cited publications, if not the most-cited publication, for every single author.
Andre Geim won the Nobel Prize in 2010 for his work on graphene, but when people stop him at conferences they want to talk about that time he used diamagnetism to make a frog float.
Stigler’s Law of Eponymy: “No scientific discovery is named after its original discoverer.” Named after Stephen Stigler (and by Stephen Stigler), originally discovered by Robert Merton. Cited more than most of Stiglers other papers.
Goodhart’s Law was originally a “jocular aside“ in a paper about monetary policy, and is now the claim for which he is best known.
$68.09 and $15.39 today, respectively.
There are five other studies in the paper that purport to show similar effects, but none of them inspire much confidence. Most of these studies have like 15 participants in each condition, which is not enough to detect big honking effects like “people have more lapses in attention when you deprive them of sleep for 24 hours” (required N = ~28 well-rested people, 28 sleep-deprived people).
In this way, I think the illusion of control may be another triple-decker fallacy, much like the one I described in Bamboozle Me, Daddy:
There’s the illusion of control: people assume they have more influence over the course of events than they really do
An illusion of the illusion of control: The original demonstrations of the illusion of control were probably flukes or fraud
And an illusion of the illusion of the illusion of control: Nevertheless, people still seem to believe they have more influence of the course of events than they really do, and so if you believe there is no such thing as the illusion of control, you are probably subject to an illusion yourself
And after this resounding non-success, Langone donated another $200 million to also eliminate tuition at NYU’s separate Long Island School of Medicine.