364 Comments
Dec 13, 2022·edited Dec 13, 2022Liked by Adam Mastroianni

I think your key connection here, which I, a military historian, had not made until now, is that peer review came out of those heady days of the early Cold War when THE EXPERTS arrived at the Pentagon. The most famous of these was MacNamara. I leave the Vietnam analogy in your capable hands. From my perspective, this is the moment when scientists were being asked to think unthinkable things and question every set of assumptions until they had exhausted all cognitive powers, for the stakes were so very, very high. We may forgive the motivations and still see the shortcomings here.

I should also note that this all fits with Max Weber's theory that specialist classes make informational gatekeeping processes into safeguards for their own status. Terry Shinn wrote about it studying L'Ecole Polytechnique and I have encountered it in my own study of the French naval engineering school. If you want to add this to the social history of how entire professions can fail together, for decades, there is historiography.

Expand full comment
author

Interesting! It does seem to fit into a broader moment of standardization. The gatekeeping angle makes a lot of sense to me––in the 19th century, a lot of science was done by dabblers. Once science becomes a profession, it has to erect a moat around itself, and journals will do that nicely.

Expand full comment
Dec 14, 2022·edited Dec 14, 2022Liked by Adam Mastroianni

Standardization is intimately connected to military revolution throughout the modern period. Everyone argues what that is, but for me, it starts in 1494, when the first effective siege artillery appeared. Cannons require intimate knowledge and care. By 1600, Europeans were loading paper or canvas cartridges with measured amounts of gunpowder into cannons, which is pretty much how the modern howitzer works. Too much diversity in an artillery park proved to be a bad thing; better to have a few standard barrel calibers. Napoleon could have grand batteries because they all fired the same shot, simplifying his logistics. I could go on at length about this topic, and sometimes do. My point here is that military necessity has often created new bureaucratic approaches within professions, and that failures arise out of the groupthink of those professions defending their own existence.

EDIT: BuOrd and the Mark-14 torpedo in 1942 is another example.

Expand full comment

That's very interesting. Why do you think it's the military revolution specifically, and not industrialization as a whole that enforced standardization?

Expand full comment

The standards emerged within industries that were responding to military necessity during a period of rapid change. Chicken or egg? State centralization and other questions started before the Westphalian peace, when almost all the means of production related to the military revolution was in private hands.

Take cannonballs. You want them to be exactly the right size to fit the barrel and roll down it easily. If you have 12 different cannon calibers, and I have 6, then you are not actually winning, because your supply chain must provide twice as many sizes of shot. The cannon will need to be test-fired with a double load of powder to ensure safety. To aim the cannon, you will need a level, and if you have a lot of gunners using levels, you will want them to all be the same. Cannon-makers tried to make every cannon tube meet the same standards, allowing any gunner to use any cannon of the same make. The gunpowder had to be produced in the right size grains and powder makers perfected the right balance of ingredients. And on and on and on.

So now imagine you are in a country whose language you don't speak, that kings and parliaments have declared is at war, and you must procure cannonballs of exactly the right diameter. "Je voudrais un boulet de canon de trois poids en livres s'il vous plaît. Dix mille"

Vauban looked at the disasters of the Thirty Years War and recognized the logistical realities of defending France. He built artillery fortresses on the frontiers and connected them with roads and bridges so that he could march armies on them quickly. Every road and bridge had to be strong enough for an army to cross it with cannons. Behold the invention of highway safety standards. The school created to improve and maintain them, L'Ecole des Ponts et Chausses, was the first modern engineering school. It still exists.

Expand full comment

Thank you for your in-depth answer, I learned a lot!

Expand full comment

The Church placed Copernicus under house arrest in the 17th Century until he capitulated his theory on Heliocentrism. and later punished Galileo for allowing the theory out into the light of day (pardon the play on words).

Now, here in the 21st Century, we have replaced the Catholic Church with a Politically Charged Peer Process (Read Government Controlled Bureaucracy) that speaks Power to Truth and like the Church, prevents said truth from reaching the sanitizing rays of sunshine.

This all supported by a Legacy Media "Ministry of Truth" that manages the Bureaucratic Newspeak and uses information suppression as its greatest instrument of Propaganda.

Expand full comment

I'm not a historian but I read a different spin on the Galileo incident, namely that the church was proud of Galileo and did not want to take any action against him, but was pressured to do so by Galileo's fellow scientists who were jealous of his fame and accomplishments. Would not be surprised if the accepted history re Galileo is a result of peer reviewed spin.

Expand full comment

Did you happen to find this narrative in “Against Method”?

Expand full comment

I saw it online but perhaps it was someone referring to that book - don't remember, honestly.

Expand full comment
Dec 20, 2022·edited Dec 20, 2022

Computer Science in its early years was awash with informational gatekeepers. They gave credence to the saying that knowledge is power. People made whole careers out of it. Interestingly, the Internet brought them all crashing down as the information that mattered had changed and was now universally available.

Expand full comment

The gatekeeper thing is probably the most important element of this. It's what keeps the system going. Science gatekeeping has pervaded our whole society, to the point where many say that if you're not an academic scientist, you have no right to an *opinion* about scientific matters.

I can (and will, in another comment) come up with a better way to do peer review. But until this gatekeeper incentive changes or is worked with somehow, nothing substantive will change. And that I don't know how to do.

Expand full comment

I would like a more adversarial system. Not that scientists should become lawyers, but the strength of an adversarial system of evidence is that both litigants are expected to present the best possible case for and against, respectively. So, here is the article, here is the damning critique, here is the weak argument. Or the article, a critique that doesn't score many points, a strong argument from strong evidence. Or maybe both sides score important valid points with the reader. Facts and law both matter in court, but I guess here it would be scientific law, and the reader gets to judge the merits according to both fact and science. Not perfect, but a starting place perhaps.

Gatekeeping, like peer review, starts with the best of intentions and becomes a problem as the profession begins to defend its own existence.

Expand full comment

I agree entirely, but the problem is modifying or eliminating the incentive structure that incentivizes gatekeeping. It's easy to say what we want things to look like; the hard problem is figuring out how to get there.

Expand full comment

The problem is that given the system we have, most people who aren't academic scientists really don't have anything useful to contribute. Their views are based on religion, political ideology, or just mental illness because if you are interested in actually learning about something you normally go through academia.

Expand full comment

That is true of most people who are not academic scientists, yes. But that leaves a very large number of people.

Expand full comment

Let me try again, as that was not clear:

Yes, most people who aren't academic scientists really don't have anything useful to contribute.

But that leaves a very large number of people who are not academic scientists who *do* have something useful to contribute.

Expand full comment

Agreed. There are quite a few people out there who have PhDs but do not work in academia. In my field (software engineering) it is common to encounter CS PhDs, Math PhDs, Physics PhDs, etc. Meaning, they have all the training to qualify for academia, but they work on practical projects.

If we are talking about life sciences, I'd bet on the average actuary, civil engineer or economist in a debate against the average biology professor on statistics. Life science researchers are quite weak on mathematics.

Expand full comment

Hi Matt, I'm very curious about Max Weber's theory of gatekeeping that you mentioned! Where is this from? I would love to read more about it!

Expand full comment

Weber's ideas are the foundation of "elite theory," a whole body of sociopolitical study. "Protestant Ethic" and "Theory of Economic and Social Organization" may be his most famous titles but if you're not a philosopher, maybe start with Gianfranco Poggi's book about Weber.

Expand full comment

Fascinating Weber (para-)quote. I ran into it yesterday in one of Fukuyama's books (The Origins of Political Order).

Expand full comment

Hi Matt, as a naval architect working for a French marine company, i would be very interested in your study of french naval engineering. do you have links pls?

Expand full comment

I also recommend the magazine articles of French naval historian Luc Feron, Terry Shinn's book about L'Ecole Polytechnque, Theodore Ropp's "Development of a Modern Navy," and Arne Roksund's book about the Jeune Ecole. Primary sources include the works of Honore Sebastian Vial du Clairbois and Pieere-Charles Francois Dupin. You'll see a distinct French aesthetic developed around the severe tumblehome hull shapes of early steel warships.

Expand full comment

"...the social history of how entire professions can fail together, for decades, there is historiography."

Who needs it? Just look at global warming.

Expand full comment

I wish we could get some remotely definitive account of that.

Expand full comment

Just read the peer-reviewed review articles.

Expand full comment

…which all indicate that the profession is doing great, and is definitely right?

Expand full comment

You aren't supposed to question it.

Expand full comment

I find a lot of the climate scientists are quite open about the limitations and uncertainties that accompany climate modeling. The activists and politicians are the ones who view it as a religion that cannot be questioned.

Expand full comment

I don't think it failed perhaps as much as it stopped working, especially as the number of scientists exploded. A good idea becomes a bureaucratic nightmare with scale.

Also the weak link point is excellent!

Expand full comment

The more centralized an entity is, the more likely it is to be the target of corruption. This system is corrupted. We need to decentralize science, the FDA and more.

Like this:

https://joshketry.substack.com/p/embrace-decentralized-systems-fear

Expand full comment

I'm much more sanguine. Some things ought to be centralised, and some not. All systems gets somehwat corrupted over time unless they evolve, centralised or no.

Expand full comment

I would say that decentralized systems become corrupt by becoming centralized.

Expand full comment

Simply putting it on chain would solve a lot of the issues. Listen to Balaji Srinivasan talk about it here: https://fs.blog/knowledge-project-podcast/balaji-srinivasan-2/

Expand full comment

Decentralizing a system greatly increases problems of fostering excellence, finding and collaborating with peers, and actually doing the critical work of building knowledge - by first understanding and synthesizing what has been done, which requires some centralization or indexing, then adding to it in a way that others who work in the area can find and recognize.

Yes, corruption pressures are critical, but very frequently, corrupted metrics are still an improvement on no measurements at all. https://mpra.ub.uni-muenchen.de/98288/

Expand full comment

This is a classic misunderstanding of systems. You can absolutely have organized, collaborating decentralized systems if that is your goal. There are hybrid systems too.

https://joshketry.substack.com/p/embrace-decentralized-systems-fear

Expand full comment

None of the examples in that post seem to work as well as the corresponding centralized ones. They are certainly less liable to be corrupted, but they don't really fit any of the criteria I noted - building knowledge, or enhancing collaboration, or fostering good solutions.

Any you ignored the central point, which was about metrics, which are central to the reason peer review and similar systems exist.

Expand full comment

I mean, what is a market other than an organized, collaborative decentralized system?

Expand full comment

...but markets themselves are almost never decentralized. In fact, they aggregate prices only if they are centralized or standardized, and otherwise they tend to be fragmented and inefficient. Thankfully, we in the west have regulators and system designers that (centrally) insist on standards and rules so that markets work. Which is why essentially no-one is trying to have an IPO or have their shares traded anywhere other than in liberal democratic western country's markets.

Expand full comment

That's…not how this works. There are some centralized markets—monopolies or monopsonies—but it is just in those markets that pricing works poorly. Where prices are functioning well and fluidly, you see decentralization—many buyers, many sellers.

But of course the decentralization of markets goes far, far beyond that. Read this (it's short): https://fee.org/resources/i-pencil/

As for regulation and standards, many quite successful companies were started before those things were the norm.

Expand full comment

Simply putting it on chain would solve a lot of the issues. Listen to Balaji Srinivasan talk about it here: https://fs.blog/knowledge-project-podcast/balaji-srinivasan-2/

Expand full comment

Thanks for pointing me to that podcast. That guy seems pretty brilliant.

Expand full comment

Agreed 100%—but *how*?

Expand full comment

Simply putting it on chain would solve a lot of the issues. Listen to Balaji Srinivasan talk about it here: https://fs.blog/knowledge-project-podcast/balaji-srinivasan-2/

Expand full comment

Well, yes, but on the other hand, peer review became widespread in large part as a response to the expansion of science in the postwar era. Journal editors simply couldn't keep up, so they relied more and more on external reviewers.

Expand full comment

Yes that's part of what I meant

Expand full comment
Dec 14, 2022Liked by Adam Mastroianni

The one thing I’m surprised didn’t make it into this amazing article is how the digital age transformed how we view science. As little as 25 years ago, you needed journals to physically publish and distribute your work. And no one was going to get your paper delivered without an established organization approving it first.

Today, the logistical and material cost of reading a paper is virtually zero. We convinced ourselves that journals were providing more of a service than simply being the paper boy because we needed them for that, at minimum. We then built an entire incentive and ladder climbing system based on that silly hang up.

No longer.

Expand full comment
author

Yes, that's a great point!

Expand full comment
Dec 14, 2022Liked by Adam Mastroianni

It seems to me that 'believe science' mostly means 'leave it to us' as a kind of argument from authority. I know someone (a scientist) who is viscerally angry that non-scientific people question the fitness of the scientific 'system' in surfacing materially objective truths about complex things without being inevitably distorted by incentives.

Maybe they're just afraid to lose elite status and peer review is simply a way to circle the wagons.

Expand full comment
author

I totally agree. There's an idea in psychology called "social dominance orientation," which is basically the extent to which you agree that some groups in society should dominate other groups. People usually think of this in terms of race, and of course all academics recoil at that. But they are generally on board with elites dominating non-elites. The changes to scientific publishing over the last sixty years are a very effective way of ensuring that nobody outside the system gets to call themselves a scientist.

Expand full comment
Dec 14, 2022Liked by Adam Mastroianni

Thanks for that, Adam - it's a new concept to me but instantly slots into personal experience.

Expand full comment

The difference is that elites actually have some idea what they're doing and the general public doesn't, which is not applicable to race if for no other reason than that biologically there's no such thing as race.

Expand full comment

I cant imagine anyone saying that out loud with a straight face. I have to wonder how you define genes as not biological.

Expand full comment

Of course ancestry exists and people get genes from their ancestors and those ancestors lived in particular places. But racial categories like "Black", "White", "Asian", and "mixed" are arbitrary and cannot be rigorously defined in an objectively correct way. This fact leads to obviously unjust absurdities whenever they are written into laws, as with trials for miscegenation, or more recent disputes over whether someone qualifies as a member of a racial group for affirmative action purposes.

Expand full comment
Jun 15, 2023·edited Jun 15, 2023

They aren't arbitrary if they can be tested for and identified via DNA. It isn't a coincidence that those groups we see are the same ones we can identify with DNA. All you are doing is parroting long debunked Gouldian lies. Race exists and has consequences. No one ever thought otherwise until jews began committing academic fraud and used their influence to shut down the research.

Expand full comment

An idea: research papers should have the reviewers names listed same way as the authors. Thus, they bear the responsibility for their review. It is also a way to check if the paper is the product of a groupthink.

Expand full comment
author

Yes, if we're going to act like something has been vetted, someone's reputation should be on the line. For what it's worth, that was the point behind early scientific societies voting on what to publish––if it puts out rubbish, the whole society looks bad.

Expand full comment

this even further disincentivizes publishing work that goes against the consensus

Expand full comment

And also a way to judge whether adverse competition between those working in the same fields might have been a factor.

Expand full comment
Dec 14, 2022Liked by Adam Mastroianni

Okay, I am going to step into a dangerous area here.

First, I am a retired accountant, not trained in the sciences. I have acquaintances at the PhD level (including in the sciences) who have gone through the "necessity" of publishing. Okay, I have stated my (non) credentials and am now ready for your attack.

It is my observation that a large part of the idea of the scientific process absolutely involves publishing of knowledge gleaned and then surviving challenge by those with contrary views. All of those presenting views are to be accorded courtesy and treated honorably. At least, as I naively see the construct.

It is my further observation that in recent years there were two very important issues that absolutely did not see this process extended. I speak of those with "unusual" [a deliberate obfuscation, you might say] views on climate change and COVID. Both those topics, it is my (further, further?) observation are not treated as being subject to courteous and honorable scientific discourse but rather as religion where someone who questions the assertions is vilified and subject to dishonor.

I offer to you an analogy. As an accountant, it is my training to seek balance, i.e. the debits must equal the credits. I therefore suggest that you folks in the scientific community have a larger problem than Mr. Mastroianni has identified. I do not deny that the accepted "religion" of climate change or COVID is perhaps correct but why is no one looking at the contrary possibilities and why do you as scientists allow the vilification, dishonor, defunding, de-platforming, etc. of those who want to explore alternative possibilities?

Oh, and as to the article itself? It seems to my non-scientific way of thinking to be a very good analysis, so, Bravo, Mr. Mastroianni!

Expand full comment
author

Thanks for posting, Ken! I don't know if I would agree with your views on climate change and covid, but I certainly agree that censorship isn't the way to solve scientific disputes. In a fair fight, the truth prevails eventually. Anyone who wants the deck stacked in favor of their idea must, deep down, doubt whether it's really true.

I think people also have a very strong conviction that it's not enough for *most* people to share their beliefs––it must be 100%. For me, anyway, I'd love my point of view to prevail, but I would be *afraid* if 100% of people thought the way I did. What if we're all wrong? That's why what I want most is not for my ideas to be enforced by fiat, but for the rules to be fair to all ideas.

Expand full comment

“I wholly disapprove of what you say—and will defend to the death your right to say it.” (Voltaire)

Expand full comment

One of the dumbest sentences ever uttered. This is saying people should have the right to commit fraud, lie in public office, lie about science, commit perjury, etc.

Expand full comment

It was already like that. Most people are just unaware, because most have not found themselves on the "wrong side" of the accepted narrative until Covid.

Expand full comment
Dec 14, 2022Liked by Adam Mastroianni

Good comment, on a good article. Possibly biased, as I too am an accountant!

Expand full comment

It's interesting you cite COVID because in the first few months of the pandemic, thousands of scientific papers were published, read and cited without a proper peer-review process. It was an all hand on deck and a lot of studies got discredited afterwards but were important in an emergency. What you get through the media or governmental sources is either biased or so heavily summarized that it might have appeared not objective towards Science. But it was, just no one bother to read or understand the scientific literature objectively.

Expand full comment
Dec 14, 2022Liked by Adam Mastroianni

Loved the article! Now convince universities to drop the publish-or-perish requirement. How many y of my life have I spent reading rubbish in abominable APA style whose only function was to add a line to a CV!

PS: The alchemy link was great.

Expand full comment
author

If you succeed in making a Philosopher's Stone, let me know!

Expand full comment
Dec 16, 2022·edited Dec 16, 2022

You would have to convince grant agencies to take the "contributions to science" out of proposal requirements. People are publishing, because they have to get grants for institutions. It all comes back to $$ not the science.

I was thinking that scientific data (mostly academia) should have solid quality control and quality assurance data that gets uploaded into a database everyone can use, instead of being lost in these publications the public does not have access to. They should provide standardized SOPs for all their data.

I did read this interesting article about creating algorithms to run quick quality control checks on the data before the paper goes to reviewers, seems like a good start.

https://www.nature.com/articles/d41586-022-03791-5

Expand full comment

I like your suggestion of a public database. And thanks for the link. Inconsistencies between test statistics and p-values could indeed be caught by such a program. But lurking behind statistical inference lies the assumption that the sample data were randomly selected from the population of interest. Except when the population is small and well defined, this is impractical or impossible. The problem of external validity can't be solved with statistics, but since everyone is in the same boat, we ignore it.

Expand full comment

Except this also exists in the humanities, and in fields where there are no grants.

Expand full comment
Dec 15, 2022Liked by Adam Mastroianni

Lots of good points here and I find myself nodding in agreement. A couple more thoughts.

One is that a system that relies on volunteer labor should generally be suspect. I got zero dollars for being an associate editor of a journal, having to assign papers to peer reviewers and then synthesize their reviews, which generally were cursory. The reviewers got zero dollars too. Oh, I know that reviewing papers is considered a 'professional obligation,' but that's the point: the peer-review system has extraordinary responsibilities for which it is willing to pay nothing. Only journals make money.

The second point is that transparency seems like a far better approach to quality control than peer review. Having to post one's data and code is highly disciplining because there's always the chance that another researcher might use them to show that you estimated a parameter or interpreted your estimates incorrectly. It's rare for someone to put in that effort, but the discipline comes from the fact that it might happen. Maybe worth an experiment.

Expand full comment

“ One is that a system that relies on volunteer labor should generally be suspect.”

Precisely why I distrust open source software.

Expand full comment

I imagine the difference is whether unpaid labor is truly voluntary. Reviewing papers without pay is effectively compulsory in academia. Is contributing to open source software effectively compulsory in order to have a career in tech?

Expand full comment

Late reply, but absolutely not.

Not only is Tchebeycheff's comment a misunderstanding of open source software, it's also just wrong.

Open Source Software (OSS) just means that the source code for the software is open for anyone to view, and according to license, edit, copy and redistribute.

OSS doesn't inherently mean "free" software: neither that the developers were unpaid, nor that the final product is free for anyone to use. Many OSS developers do so under payroll, the largest tech companies all have big stakes in OSS projects and developers working in part or exclusively in OSS.

A lot of OSS software has a license that allows any user to copy and modify the code, and that if they intend to monetize it, then they must also make their source code open.

A lot more "for pay" software than people think heavily relies on open source software.

Furthermore, you could view the fact that the source is open to review by anyone as a system that has the possible positives of a peer review system (scrutiny, improvement, refinement, critique) with none of the drawbacks (publish or perish, pay to get reviewed, gatekeeping information and elitism).

Any amateur can write or contribute to OSS if they so desire.

There is also closed source software that is free, which is freeware/shareware.

Another thought, one might argue that the rapid pace of development in computer software both in terms of tools, programs, paradigms, etc. can be attributed to not being stifled by peer review, (computer science and software development are not the same) only that the end result works. There is certainly a lot of critique to be made for software development practices.

Expand full comment

“Any amateur can write or contribute to OSS if they so desire.”

Also why I distrust it.

Expand full comment

Contributions are still subject to approval from the owners/maintainers of the project. You can't just come in and change anything you want.

Expand full comment

Scientific publishing houses have a 37% profit margin*. They perpetuate this "professional obligation" narrative because the key to 37% is not to pay your (otherwise very expensive) laborers. When I'm asked to review, I ask how much I will be paid. Fair question - paying for skilled labor is non-controversial everywhere in our society. Strangely, I never get an answer (or a follow up request). * https://infrastructure.sparcopen.org/landscape-analysis/elsevier#:~:text=Elsevier%20operates%20at%20a%2037,operates%20at%20a%2023%25%20margin.

Expand full comment

Adam, I really love all of your writing on scientific publishing. Also, this made me go read your Things Could Be Better paper again, which might be the first time that I read a full paper in full *twice*, *for fun*.

Expand full comment
author

Thanks, Étienne! Your review on Making Nature was formative background reading for this post.

Expand full comment

Also just to note that the reliance of university tenure and promotion standards on measuring the publication of a certain number of "peer reviewed" articles also maintains the rigidity of this system

Expand full comment
Dec 15, 2022Liked by Adam Mastroianni

Ever tried questioning what the results are while doing a peer review? Ever asked about underlying assumptions? Most peer reviews are done by people that have the same background, want the same results, and are under the same impetus to publish, publish, publish. They know that if they are critical, then when it is their turn the others will be critical. I have had people get very defensive when I question anything, and I have had other people amazed and astonished that they actually get relevant feedback beyond really basic stuff! It is uncomfortable, and boring to do peer reviews. All the incentives for it are backwards.

Expand full comment

Or they are competing peers and rivals for status and grant money in their field.

Expand full comment

Loud applause! I’ve published only 2 papers in the past 4 years and by “published” I mean uploaded to ReaearchGate. I’m very happy with the readership and the constructive discussions and feedback that has allowed me to further refine my theories as the field progresses.

Given the abject rubbish appearing in even the most prestigious journals, I feel no urge to submit my stuff or pay their fees.

Expand full comment
author

Pay to publish your article, pay to read your article––it really is a beautiful scam.

Expand full comment

The business brilliance of Robert Maxwell created the monster. Ever increasing numbers of journals, paid submissions and subscriptions and the complicity of institutions that also want beauracitize systems add plenty of fuel to the problem.

Expand full comment

We need to think about the systems that govern over science. If we make them transparent and decentralize them, we can fix everything. Put everything on chain so other scientists can build off each other.

Try to find Balaji Srinivasan talking about this on The Knowledge Project podcast:

Start this at 26:10

https://www.bing.com/videos/search?q=the+kowledge+project+balaji&view=detail&mid=35DE2BA7D38A7A3B41C635DE2BA7D38A7A3B41C6&FORM=VIRE

Also read this

https://joshketry.substack.com/p/embrace-decentralized-systems-fear

Expand full comment
Dec 16, 2022Liked by Adam Mastroianni

A wonderful article. I've been arguing something similar for many years: https://breast-cancer-research.biomedcentral.com/articles/10.1186/bcr2742 It's fascinating to me that a process at the heart of science is faith not evidence based. Indeed, believing in peer review is less scientific than believing in God because we have lots of evidence that peer review doesn't work, whereas we lack evidence that God doesn't exist. Why does the juggernaut roll on? Because there are lots of jobs, profit, and reputations that depend on continuing to believe in it?

Expand full comment
author

Hi, Richard! I didn't realize at first that you are the same Richard Smith I was reading as I wrote this article. Thanks for all your work on this subject!

Expand full comment
Dec 13, 2022Liked by Adam Mastroianni

I agree with your conclusion about the current system being suboptimal. However, I'm curious if you think there's anything more efficient than chaos? For example, what if Arxiv had an openreview-esque comments section? So that communications were centralized. Or something like a reddit-esque community governed forum?

Expand full comment
author

I think progress requires a healthy dose of chaos. Those kinds of systems would be better than the one we have now, but I would always want there to be some kind of outlet for weird stuff.

Expand full comment

We need to think about the systems that govern over science. If we make them transparent and decentralize them, we can fix everything. Put everything on chain so other scientists can build off each other.

Try to find Balaji Srinivasan talking about this on The Knowledge Project podcast:

Start this at 26:10

https://www.bing.com/videos/search?q=the+kowledge+project+balaji&view=detail&mid=35DE2BA7D38A7A3B41C635DE2BA7D38A7A3B41C6&FORM=VIRE

Also read this

https://joshketry.substack.com/p/embrace-decentralized-systems-fear

Expand full comment
Dec 13, 2022Liked by Adam Mastroianni

Thanks. Another great observation and article.

I wonder if we made a) reviews optional b) showed the reviews c) let the authors respond to the reviews and show those too i.e. the whole conversation (like in any blog comments section!) d) give the reviewers that contributed most thoughtfully almost as much prominence as the authors themselves (in the author's opinion? or perhaps a legitimate curator role who narrates the whole 'story'?) e) linked to follow up work that supported or contradicted (again the web makes this easy) we could start to build better mechanisms for encouraging, thinking about and testing our ideas.

Expand full comment
author

I think that would be a huge improvement on the current system. I still worry that if it becomes centralized or mandatory it will eventually get captured just like this system did. But I'd certainly be interested in trying it.

Expand full comment

You’ve heard of The Seeds of Science...?

https://www.theseedsofscience.org/about

Expand full comment
author

Yep, I recommend it on Substack! I prefer a direct-to-consumer model, but I am in favor of all forms of experimentation.

Expand full comment
Dec 14, 2022Liked by Adam Mastroianni

Yes there is a risk of capture - perhaps in any situation where groups of humans 'collaborate' there is a risk of groupthink or even partial groupthink (is that a thing?). So we need the groups, that naturally form through shared interests, remain as open and dynamic as possible. To mitigate against noise or even sabotage would remain a challenge but one we can figure out, starting with verifiable identities.

Another interesting idea might be to allow the traditional paper to be deconstructed so hypotheses can be proposed, experiments to test them proposed and reported, conclusions drawn and avenues of further exploration as separate units. This might speed up the progress by allowing more people to participate in the conversation before a full paper is published.

Expand full comment
Dec 16, 2022Liked by Adam Mastroianni

I don't know what the solution is. Full disclosure, I've been working in journal publishing for 15 years.

You successfully self-published an article. If everyone who wrote a paper did that, there would be 100s of manuscripts uploaded weekly with zero quality control and zero discoverability unless like a self-publishing fiction author you work your ass off at social media to get noticed.

You talk about pre-1950s academic publishing, but the world was a much smaller place then and with a lot less research happening. You didn't have China pushing their academics to publish - many of them being unable graduate until they've been published.

Perhaps is this the world shaped by peer review.

Obviously, there are issues with peer review and with for-profit journal publishing. I'm certainly not 100% comfortable with the way things are going, especially since OA became the drive for profit.

I don't think my industry will survive as it is. I know other colleagues feel this way. However, I don't know what the future is.

Expand full comment
author

I don't know what the future is either. I agree that discoverability is a problem, but I don't think peer review and journals solve it, and they in part create it. Here are three reasons:

1) For knowledge to be discoverable, it has to be public, and journals prevent that by paywalling knowledge or rejecting it in the first place. Publication bias is well-known to be a huge problem that distorts our view of the truth, and the cause of publication bias is publication.

2) Discoverable knowledge should be easily digestible. Journals encourage sprawling, jargony articles that nobody actually reads from beginning to end, diluting any useful knowledge so much that it takes a long time to concentrate it again.

3) Most research isn't good and should be ignored anyway. Nobody knows exactly how many articles are never cited after they're published (https://blogs.lse.ac.uk/impactofsocialsciences/2014/04/23/academic-papers-citation-rates-remler/), but it seems to be a pretty large chunk. If you could also filter out papers that are never cited again except by their own authors, that chunk would probably grow. So if we had a morass of unread papers, I think that's both perfectly fine and in fact what we already have, except right now we waste additional time reviewing those papers first before they're ignored forever.

Expand full comment

The solution is to decentralize science in a transparent system.

As Balaji likes to say: "Science" is simply the ability to be independently repeatable. If you have some prestigious study, the chances of it being centralized are great. And if it is centralized, then it is likely corrupted.

The only thing that matters is if the experiment can be independently replicated.

We need to decentralize science with technology.

https://joshketry.substack.com/p/embrace-decentralized-systems-fear

Expand full comment

Well, Balaji has no clue.

There is only scientific method - positing falsifiable hypothesis and refuting it. There is no such thing as “Science” the noun.

Expand full comment

"You successfully self-published an article. If everyone who wrote a paper did that, there would be 100s of manuscripts uploaded weekly with zero quality control and zero discoverability unless like a self-publishing fiction author you work your ass off at social media to get noticed." The big word in that passage is "if." As you point out, uploading 100s of manuscripts every week would not be worthwhile — which strongly suggests that people won't do it. They would probably only upload things that seem worthwhile.

We would need an experiment to determine what actually would happen without peer review. (We already know what happens *with* peer review, and it's not exactly an unalloyed good: unreproducible results, exaggerations of significance, outright lies (e.g., fake data), 100s of papers published weekly...)

I think the alternative deserves a try, though I'm not sure how to handle it. Perhaps a specific discipline could drop peer review to see what actually happens.

Expand full comment