The dance of the naked emperors

A followup to "The rise and fall of peer review”

photo cred: my dad

Last month I published a paper by uploading a PDF to the internet and people were like “nice paper, here are some thoughts!” 

Two weeks ago I wrote a post saying peer review is a failed experiment and that one alternative is to upload PDFs to the internet and some people were like “HOLD ON THERE, BUSTER!” 

A tenured professor hinted she might try to get me fired. A person with a PhD accused me of “cynical metacognitive polywaffle,” which a good name for a postmodern noise band. I got some weird and vaguely threatening emails, including one that had a screenshot of my personal website with my improv experience highlighted, proof that I am literally a clown. Which is, I guess, true.

(People said nice things too.)

PREVIOUSLY ON EXPERIMENTAL HISTORY

To recap, I argued in my last post that:

1. We've published science lots of different ways for a long time, and universal pre-publication peer review is both pretty new and historically strange.

2. That system doesn’t seem to accomplish the goals that it claims to or that we wish it would.

3. It’s worthwhile to try other things. 

At its core, this is an argument against scientific monoculture. Why should everyone publish the same way? You’d have to be extremely certain that way was better than all other ways—and that it was better for every single person!—and that amount of certainty seems pretty loony to me. Uploading a PDF to the internet worked for me, but there are lots of other ways people could communicate their findings, and I hope they try them out.

This generated, in addition to the threats, some great comments. Thanks to everyone who wrote! Here are some responses.

FEAR OF THE MASSES PART ONE; OR, WHAT HAPPENS WHEN YOU CAN PRINT NOBEL PRIZES AT HOME?

First, some people were worried about what might happen in a world where everyone chose to upload PDFs to the internet instead of publishing in journals. For instance, Annon writes:

I don't know what the solution is. Full disclosure, I've been working in journal publishing for 15 years.

You successfully self-published an article. If everyone who wrote a paper did that, there would be 100s of manuscripts uploaded weekly with zero quality control and zero discoverability unless like a self-publishing fiction author you work your ass off at social media to get noticed.

I understand this fear, but I think it’s got the wrong model of the world.

Imagine that the Nobel Committee decides to stop picking Nobel Prize winners. “Everybody can print out their own Nobel Prize certificate at home!” they announce.

It would be silly to worry that people would spend all day printing out Nobel Prizes—there’s no reason to, because they’re now worthless. If you show up to the lab and say, “Look everybody, I have four thousand Nobel Prizes!” everyone is going to laugh at you.

I think it would work the same in publishing papers. Right now you get credit for each paper you publish in a journal (with more credit for more prestigious journals), so you want to publish as many as you can. But if “publishing” is just “uploading a PDF to the internet,” you get no credit for the act of publishing itself, so publishing lots of papers just for the sake of publishing them would only make you look dumb.

I don’t have to imagine what this would be like, because it’s exactly the situation I’m in right now. When I work on projects that are intended to be published in journals, my brain goes, “PLAY THE GAME, PRODUCE LOTS OF PAPERS, GET IT OVER WITH.” 

But when I work on projects intended to be published on Experimental History, my brain goes “make sure you do a good job! Be honest, interesting, and kind!” Without a journal to vouch for me, all I have is my reputation and the quality of my work. I could try lots of desperate tactics to get people’s attention—“12 AMAZING PSYCHOLOGICAL FINDINGS THAT DOCTORS DON’T WANT YOU TO KNOW ABOUT”—but if people click on my work and think it’s crap, they won't come back again, they won’t tell anyone else about it, and pretty soon I’ll be talking to myself.

That’s also why I’m not worried about an onslaught of terrible papers—we’ve already got an onslaught of terrible papers. You learn very quickly to ignore them, just like you learn to ignore junk mail, stupid Netflix shows, and spam calls. Again, science is a strong-link problem. I’m not worried about how many bad papers there are; I’m worried about how many good papers there are.

FEAR OF THE MASSES PART TWO; OR, WHAT HAPPENS WHEN EVERYONE IS ALLOWED TO SPEAK?

A variant of this same fear is: won’t we have anarchy if anyone can say anything?

Jerry D. Harris: 

I SO look forward to the onslaught of poorly written but publicly elevated, freely available papers about the veracity of creationism and intelligent design, and the falsity of anthropogenic climate change, and that vaccines don't work, written by "scientists" in a world without peer-review as a barrier. 

I understand this worry, too, but I think it also has the wrong model of the world.

There’s nothing stopping creationists or anybody else from starting their own peer-reviewed journal. In fact, creationists already did this. It’s called The Journal of Creation, and they just published a new issue this month.

So peer-review isn’t a barrier preventing creationists from publishing their papers. What, then, is stopping creationism from being the dominant theory of where humans come from?

It’s the fact that creationists have worse arguments. It’s tough to get someone to believe that the Earth is 7,000 years old. You have to convince them that you have a book of God’s actual words, that the other supposed books of God’s words are imposters, that the words are meant to be interpreted in this specific way and no other, and that all of the archeologists, paleontologists, biologists, etc., who claim to have encountered evidence to the contrary are in fact mistaken or have been sent by the Devil to mislead you.

It isn’t impossible to get people to believe all this, of course, which is why The Journal of Creation exists. It’s just not a winning argument, so it doesn’t win as much. 

That’s good! If something is actually true, then it should be able to defeat any falsehood in a fair fight. And if we’re serious about seeking truth, we should want that fight to be fair.

PEER REVIEW WORKS FOR ME / NO IT DOESN'T

Some people shared their bad experiences with peer review, like Martina Pugliese (author of Doodling Data Cards), mardin56, and Grey Coupland. Richard Smith, the former editor of the British Medical Journal, commented:

It's fascinating to me that a process at the heart of science is faith not evidence based. Indeed, believing in peer review is less scientific than believing in God because we have lots of evidence that peer review doesn't work, whereas we lack evidence that God doesn't exist.

On the other hand, some people shared their good experiences with peer review, like Jake, Nuno Henrique Franco, and John Vidale. In Jake’s words:

I'm an early career researcher in the humanities. I've often found peer review really improved my work prior to publication and resulted in better research (though I know some people are not so lucky with their peer reviewers).

I’m glad this works for some folks. Here are five things I would add:

1. When people spend a collective 15,000 years of labor every year commenting on each other’s work, I would certainly hope that some of it is helpful. That’s about the lowest bar it could clear.

2. Pre-publication peer review is one of many ways you could get feedback. People naturally comment on stuff they care about, and their comments are way better when given out of interest instead of obligation. I’ve gotten more useful criticism on my “Things Could Be Better” paper than anything else I’ve published. If I had put that paper out there and everyone had just ignored it, that would be useful feedback too: “Turns out nobody cares! Either I need to do a better job, or do something else."

3. Publishing this way means a paper stops improving once it’s published. (You can issue corrections, but otherwise it’s supposed to be final.) But why would you stop listening to comments and making your paper better just because it’s now publicly accessible?

4. If this way works for you, I say: keep doing it.

5. If it doesn’t work for you, I say: try something else.

TWEAKIN’

Lots of folks had ideas about how peer review could work better.

Transitory (author of Transitory):

An idea: research papers should have the reviewers names listed same way as the authors

Ben Sahlmueller (author of Benerkenswert):

I would prefer a decentralized system with open reviews, open access, more accountability etc. […]

Josh McGrath:

[W]hat if Arxiv had an openreview-esque comments section? So that communications were centralized. Or something like a reddit-esque community governed forum?

Stan Jastrzebski:

So what if publishing was more like the website Rotten Tomatoes is for movie and TV reviews? 

The face in the mirror (author of The face in the mirror):

Would a combination of narrowly tailored and broad LLMs (large language models) work as automated peer review within specialized sub-fields?

If you’ve got a hunch that something would work, try it! If you’re an editor, run an experiment on your editorial process. If you’re an engineer, build those LLMs and see what they can do. If you’re mad about bad scientific papers, start posting reviews online. There’s almost certainly something you can do, no matter who you are.

But there is no Emperor of Science who can make changes by decree; it’s up to you. If that sounds like a lot of work and you’d rather not bother, that’s okay. There are lots of problems in the world, and you can’t work on all of them.

A FEW OTHER GOOD COMMENTS

Noah M:

As little as 25 years ago, you needed journals to physically publish and distribute your work. […] Today, the logistical and material cost of reading a paper is virtually zero. We convinced ourselves that journals were providing more of a service than simply being the paper boy because we needed them for that, at minimum. We then built an entire incentive and ladder climbing system based on that silly hang up.

This is a great point. I can feel the status quo bias weighing down all of our imaginations.

Ruz:

I have had people get very defensive when I question anything, and I have had other people amazed and astonished that they actually get relevant feedback beyond really basic stuff! It is uncomfortable, and boring to do peer reviews.

The word boring really stuck with me. Reviewing should be interesting. It should matter whether the paper’s claims turn out to be true or not, and the only reason to review it is that you care about those claims. The fact that we find it boring suggests that part of us, deep down, believes that the paper in front of us doesn’t actually deserve our attention.

Akshat Mahajan:

[W]e *have* truly open, zero moderation platforms (e.g. vixra.org). They have failed to produce the intended effect of better science, for many foundational reasons. 

I agree with Mahajan's first point: we’ve got all the infrastructure we need, but people aren’t using it to experiment. They just post their PDFs on a website before trying to get them published in a journal, so whatever they produce is still intended to pass peer review. It’s like everyone has a Jeep that can go off-road, and yet they only ever drive on the highway. 

I disagree with Mahajan’s second point: these platforms haven’t failed. If you give everyone a Jeep hoping that they’ll drive it into the wilderness and nobody does, don’t fix the Jeeps; fix the drivers. You need to make them less afraid to leave the highway, convince them that there’s something worth seeing out there, and gas up their tanks.

This is why, as much as I would love to see people try out all the alternative platforms they suggested, I don’t think we’ll revolutionize science by building the perfect website. We have to a) free minds, and b) fund them. I’m working on (a) right now, and I’ve got plans in the works for (b).

PLEASE DON’T TIP THIS LADDER OVER, WE’RE TRYING TO CLIMB IT

I feel very grateful for the smart folks who read this blog and make good points in the comments. I always learn something, even when I don’t agree. But damn, some of the comments I got here and on Twitter—often from scientists themselves—were just plain nasty.

I know the risks of writing words on the internet, and I know I’m not the first person to get yelled at. I just didn’t think it would be about this. Why did it get people so riled up?

I can think of two reasons.

First: the third-person effect, which is people’s tendency to think that other people are susceptible to persuasion. I am a savvy consumer; you are a knucklehead who can be duped into buying Budweiser by a pair of boobs. I evaluate arguments rationally; you listen to whoever is shouting the loudest. I won’t be swayed by a blog post; you will.

This is a stupid and condescending thing to think, and yet we all think it, and perhaps scientists especially so. So it’s no wonder that we get upset when we hear arguments we dislike—we foolishly fear that those arguments will win over all the dummies who populate our big dumb world. 

And second: social dominance. Scientists may think they're egalitarian because they don’t believe in hierarchies based on race, sex, wealth, and so on. But some of them believe very strongly in hierarchy based on prestige. In their eyes, it is right and good for people with more degrees, bigger grants, and fancier academic positions to be above people who have fewer of those things. They don’t even think of this as hierarchy, exactly, because that sounds like a bad word. To them, it's just the natural order of things.

(To see this in action, watch what happens when two academic scientists meet. The first things they’ll want to know about each are are 1) career stage—grad student, postdoc, professor, etc., and 2) institution. These are the X and Y coordinates that allow you to place someone in the hierarchy: professor at elite institution gets lots of status, grad student at no-name institution gets none. Older-looking graduate students sometimes have the experience of being mistaken for professors, and professors will chat to them amiably until they realize their mistake, at which point they will, horrified, high-tail it out of the conversation.)

People who are all-in on a hierarchy don’t like it when you question its central assumptions. If peer review doesn’t work or is even harmful to science, it suggests the people at the top of the hierarchy might be naked emperors, and that's upsetting not just to the naked emperors themselves, but also the people who are diligently disrobing in the hopes of becoming one. In fact, it’s more than upsetting—it’s dangerous, because it could tip over a ladder that has many people on it.

But look, it should be pretty easy to defend a system that’s supposedly based on evidence. You should be able to count the costs and benefits of doing things the way we do them, and it should be clear that the benefits outweigh the costs. You should be able to lay out the data without embarrassment, you should be able to answer any good-faith objections, and you should be both honest about and comfortable with the limits of your knowledge. You definitely shouldn’t have to resort to invoking prestige or threatening to get me fired. If that’s all you got, you ain’t got much.

SEND IN THE CLOWNS

Arguing on the internet is all well and good. It helps you trim the flabby parts of your reasoning, like cognitive liposuction. It brings all the claims into the daylight so people can read them and come to their own conclusions. And it can be pretty fun, except the part where people yell.

But this is just sparring, just sending electrons through fiber-optic cables and moving pixels around on screens. The real battle is out there. Ideas don’t succeed by getting people to agree; they succeed by inspiring people to do stuff.

“What if you try something and it doesn’t work? Then you’ll look foolish!” My friend, have you ever run an experiment? If there’s no chance of failure, then there’s no chance of learning anything. 

So now is the time to act. May the best ideas win. And if I end up looking foolish, that’s all right. I am, after all, literally a clown.