Science is a strong-link problem

OR: How to eat fewer asparagus beetles

Photo cred: my dad

There are two kinds of problems in the world: strong-link problems and weak-link problems.¹

Weak-link problems are problems where the overall quality depends on how good the worst stuff is. You fix weak-link problems by making the weakest links stronger, or by eliminating them entirely.

Food safety, for example, is a weak-link problem. You don’t want to eat anything that will kill you. That’s why it makes sense for the Food and Drug Administration to inspect processing plants, to set standards, and to ban dangerous foods. The upside is that, for example, any frozen asparagus you buy can only have “10% by count of spears or pieces infested with 6 or more attached asparagus beetle eggs and/or sacs.” The downside is that you don’t get to eat the supposedly delicious casu marzu, a Sardinian cheese with live maggots inside it.

It would be a big mistake for the FDA to instead focus on making the safest foods safer, or to throw the gates wide open so that we have a marketplace filled with a mix of extremely dangerous and extremely safe foods. In a weak-link problem like this, the right move is to minimize the number of asparagus beetle egg sacs.

Weak-link problems are everywhere. A car engine is a weak-link problem: it doesn’t matter how great your spark plugs are if your transmission is busted. Nuclear proliferation is a weak-link problem: it would be great if, say, France locked up their nukes even tighter, but the real danger is some rogue nation blowing up the world. Putting on too-tight pants is a weak-link problem: they’re gonna split at the seams.

It’s easy to assume that all problems are like this, but they’re not. Some problems are strong-link problems: overall quality depends on how good the best stuff is, and the bad stuff barely matters. Like music, for instance. You listen to the stuff you like the most and ignore the rest. When your favorite band releases a new album, you go “yippee!” When a band you’ve never heard of and wouldn’t like anyway releases a new album, you go…nothing at all, you don’t even know it’s happened. At worst, bad music makes it a little harder for you to find good music, or it annoys you by being played on the radio in the grocery store while you’re trying to buy your beetle-free asparagus.

Because music is a strong-link problem, it would be a big mistake to have an FDA for music. Imagine if you could only upload a song to Spotify after you got a degree in musicology, or memorized all the sharps in the key of A-sharp minor, or demonstrated competence with the oboe. Imagine if government inspectors showed up at music studios to ensure that no one was playing out of tune. You’d wipe out most of the great stuff and replace it with a bunch of music that checks all the boxes but doesn’t stir your soul, and gosh darn it, souls must be stirred.

Strong-link problems are everywhere; they’re just harder to spot. Winning the Olympics is a strong-link problem: all that matters is how good your country’s best athletes are. Friendships are a strong-link problem: you wouldn’t trade your ride-or-dies for better acquaintances. Venture capital is a strong-link problem: it’s fine to invest in a bunch of startups that go bust as long as one of them goes to a billion.

Figuring out whether a problem is strong-link or weak-link is important because the way you solve them is totally different:

When you’re looking to find a doctor for a routine procedure, you’re in a weak-link problem. It would be great to find the best doctor on the planet, of course, but an average doctor is fine—you just want to avoid someone who’s going to prescribe you snake oil or botch your wart removal. For you, it’s great to live in a world where doctors have to get medical degrees and maintain their licenses, and where drugs are thoroughly checked for side effects.

But if you’re diagnosed with a terminal disease, you’re suddenly in a strong-link problem. An average doctor won’t cut it for you anymore, because average means you die. You need a miracle, and you’re furious at anyone who would stop that from happening: the government for banning drugs that might help you, doctors who refuse to do risky treatments, and a medical establishment that’s more worried about preventing quacks than allowing the best healers to do as they please.

REST IN PEACE LIL SPERM BOYS

Science is a strong-link problem.

In the long run, the best stuff is basically all that matters, and the bad stuff doesn’t matter at all. The history of science is littered with the skulls of dead theories. No more phlogiston nor phlegm, no more luminiferous ether, no more geocentrism, no more measuring someone’s character by the bumps on their head, no more barnacles magically turning into geese, no more invisible rays shooting out of people’s eyes, no more plum pudding, and, perhaps saddest of all, no more little dudes curled up inside sperm cells:

Goodby lil dudes, we hardly knew you

Our current scientific beliefs are not a random mix of the dumbest and smartest ideas from all of human history, and that’s because the smarter ideas stuck around while the dumber ones kind of went nowhere, on average—the hallmark of a strong-link problem. That doesn’t mean better ideas win immediately. Worse ideas can soak up resources and waste our time, and frauds can mislead us temporarily. It can take longer than a human lifetime to figure out which ideas are better, and sometimes progress only happens when old scientists die. But when a theory does a better job of explaining the world, it tends to stick around.

(Science being a strong-link problem doesn’t mean that science is currently strong. I think we’re still living in the Dark Ages, just less dark than before.)

SCIENTIFIC KRYPTONITE

Here’s the crazy thing: most people treat science like it’s a weak-link problem.

Peer reviewing publications and grant proposals, for example, is a massive weak-link intervention. We spend ~15,000 collective years of effort every year trying to prevent bad research from being published. We force scientists to spend huge chunks of time filling out grant applications—most of which will be unsuccessful—because we want to make sure we aren’t wasting our money. 

These policies, like all forms of gatekeeping, are potentially terrific solutions for weak-link problems because they can stamp out the worst research. But they’re terrible solutions for strong-link problems because they can stamp out the best research, too. Reviewers are less likely to greenlight papers and grants if they’re novel, risky, or interdisciplinary. When you’re trying to solve a strong-link problem, this is like swallowing a big lump of kryptonite.

(Peer review also does a pretty bad job at stamping out bad research too, oops.)

Giant replication projects—like this one, this one, this one, this one, and this one—also only make sense for weak-link problems. There’s no point in picking some studies that are convenient to replicate, doing ‘em over, and reporting “only 36% of them replicate!” In a strong-link situation, most studies don’t matter. To borrow the words of a wise colleague: “What do I care if it happened a second time? I didn’t care when it happened the first time!”

This is kind of like walking through a Barnes & Noble, grabbing whichever novels catch your eye, and reviewing them. “Only 36% of novels are any good!” you report. That’s fine! Novels are a strong-link problem: you read the best ones, and the worst ones merely take up shelf space. Most novels are written by Danielle Steel anyway.

(See also: Psychology might be a big stinkin’ load of hogwash and that’s just fine.)

CHEATERS SOMETIMES WIN AND THAT’S OKAY

I think there are two reasons why scientists act like science is a weak-link problem.

The first reason is fear. Competition for academic jobs, grants, and space in prestigious journals is more cutthroat than ever. When a single member of a grant panel, hiring committee, or editorial board can tank your career, you better stick to low-risk ideas. That’s fine when we’re trying to keep beetles out of asparagus, but it’s not fine when we’re trying to discover fundamental truths about the world.

(See also: Grant funding is broken. Here’s how to fix it.)

The second reason is status. I’ve talked to a lot of folks since I published The rise and fall of peer review and got a lot of comments, and I’ve realized that when scientists tell me, “We need to prevent bad research from being published!” they often mean, “We need to prevent people from gaining academic status that they don’t deserve!” That is, to them, the problem with bad research isn’t really that it distorts the scientific record. The problem with bad research is that it’s cheating.

I get that. It’s maddening to watch someone get ahead using shady tactics, and it might seem like the solution is to tighten the rules so we catch more of the cheaters. But that’s weak-link thinking. The real solution is to care less about the hierarchy. If you spend your life yelling at bad scientists, you’ll make yourself hoarse. If you spend your life trying to do great science, you might forever change the world for the better, which seems like a better use of time.

THE MISSING STRONG LINKS

Here’s our reward for a generation of weak-link thinking.

The US government spends ~10x more on science today than it did in 1956, adjusted for inflation. We’ve got loads more scientists, and they publish way more papers. And yet science is less disruptive than ever, scientific productivity has been falling for decades, and scientists rate the discoveries of decades ago as worthier than the discoveries of today. (Reminder, if you want to blame this on ideas getting harder to find, I will fight you.)

We should have seen this coming, because the folks doing the strongest-link research have been warning us about it for a long time. One of my favorite genres is “Nobel Prize winner explains how it would be impossible to do their Nobel Prize-winning work today.” For instance, here’s Peter Higgs (Nobel Prize in Physics, 2013):

Today I wouldn’t get an academic job. It’s as simple as that. I don’t think I would be regarded as productive enough.

Sydney Brenner (Nobel Prize in Physiology or Medicine, 2002) on Frederick Sanger (Nobel Prize in Chemistry, 1958 & 1980):

A Fred Sanger would not survive today's world of science. With continuous reporting and appraisals, some committee would note that he published little of import between insulin in 1952 and his first paper on RNA sequencing in 1967 with another long gap until DNA sequencing in 1977. He would be labeled as unproductive, and his modest personal support would be denied. We no longer have a culture that allows individuals to embark on long-term—and what would be considered today extremely risky—projects.

Carol Greider (Nobel Prize in Physiology or Medicine, 2009):

“I’m not sure in the current climate we have for research funding that I would have received funding to be able to do the work that led to the Nobel Prize,” Greider said at a National Institutes of Health (NIH) event last month, adding that her early work on enzymes and cell biology was well outside the mainstream.

John Sulston (Nobel Prize in Physiology or Medicine, 2002):

I wandered along to the chemistry labs, more or less on the rebound, and asked about becoming a research student. It was the 60s, a time of university expansion: the doors were open and a 2:1 [roughly equivalent to a B] was good enough to get me in. I couldn’t have done it now.

Jeffrey C. Hall (Nobel Prize in Physiology or Medicine, 2017):

I admit that I resent running out of research money. [...] In my day you could get a faculty job with zero post-doc papers, as in the case of yours truly; but now the CV of a successful applicant looks like that of a newly minted full Professor from olden times. [...] US institutions (possibly also those in other countries) behave as though they and their PIs are entitled to research funding, which will magically materialize from elsewhere: ‘Get a grant, serf! If you can't do it quickly, or have trouble for some years — or if your funding doesn't get renewed, despite continuing productivity — forget it!’ But what if there are so many applicants (as there are nowadays) that even a meritorious proposal gets the supplicant nowhere or causes a research group to grind prematurely to a halt? [...] Thus, as I say ‘so long,’ one component of my last-gasp disquiet stems from pompously worrying about biologists who are starting out or are in mid-career.

It goes on and on like this. When the people doing the best work are saying “hey there’s no way you could do work like this anymore,” maybe we should listen to them.

WHAT TO DO WHEN YOU STINK

I’ve got a hunch that science isn’t the only strong-link problem we’ve mistakenly diagnosed as a weak-link problem. It’s easy to get your knickers in a pinch about weak links—look at these bad things!! They’re so bad!! Can you believe how bad they are?? 

It’s even easier to never think about the strong links that were prevented from existing. The terrible study that gets published sounds like nails on a chalkboard, but the terrific study that never got funded sounds like nothing at all. Purge all the terrible at the cost of the terrific, and all you’re left with is the mediocre.

Of course, it’s also easy to make the opposite mistake, to think you’re facing a strong-link problem when in fact you’ve got a weak-link problem on your hands. It doesn’t really matter how rich the richest are when the poorest are starving. Issuing parking tickets is pointless when people are getting mugged on the sidewalk. Upgrading your wardrobe is a waste when you stink like a big fart.

Whether we realize it or not, we’re always making calls like this. Whenever we demand certificates, credentials, inspections, professionalism, standards, and regulations, we are saying: “this is a weak-link problem; we must prevent the bad!” 

Whenever we demand laissez-faire, the cutting of red tape, the letting of a thousand flowers bloom, we are saying: “this is a strong-link problem; we must promote the good!”

When we get this right, we fill the world with good things and rid the world of bad things. When we don’t, we end up stunting science for a generation. Or we end up eating a lot of asparagus beetles.

1. I originally heard these terms on this podcast discussing this book.