Dec 13, 2022·edited Dec 13, 2022Liked by Adam Mastroianni
I think your key connection here, which I, a military historian, had not made until now, is that peer review came out of those heady days of the early Cold War when THE EXPERTS arrived at the Pentagon. The most famous of these was MacNamara. I leave the Vietnam analogy in your capable hands. From my perspective, this is the moment when scientists were being asked to think unthinkable things and question every set of assumptions until they had exhausted all cognitive powers, for the stakes were so very, very high. We may forgive the motivations and still see the shortcomings here.
I should also note that this all fits with Max Weber's theory that specialist classes make informational gatekeeping processes into safeguards for their own status. Terry Shinn wrote about it studying L'Ecole Polytechnique and I have encountered it in my own study of the French naval engineering school. If you want to add this to the social history of how entire professions can fail together, for decades, there is historiography.
I don't think it failed perhaps as much as it stopped working, especially as the number of scientists exploded. A good idea becomes a bureaucratic nightmare with scale.
The one thing I’m surprised didn’t make it into this amazing article is how the digital age transformed how we view science. As little as 25 years ago, you needed journals to physically publish and distribute your work. And no one was going to get your paper delivered without an established organization approving it first.
Today, the logistical and material cost of reading a paper is virtually zero. We convinced ourselves that journals were providing more of a service than simply being the paper boy because we needed them for that, at minimum. We then built an entire incentive and ladder climbing system based on that silly hang up.
It seems to me that 'believe science' mostly means 'leave it to us' as a kind of argument from authority. I know someone (a scientist) who is viscerally angry that non-scientific people question the fitness of the scientific 'system' in surfacing materially objective truths about complex things without being inevitably distorted by incentives.
Maybe they're just afraid to lose elite status and peer review is simply a way to circle the wagons.
An idea: research papers should have the reviewers names listed same way as the authors. Thus, they bear the responsibility for their review. It is also a way to check if the paper is the product of a groupthink.
Okay, I am going to step into a dangerous area here.
First, I am a retired accountant, not trained in the sciences. I have acquaintances at the PhD level (including in the sciences) who have gone through the "necessity" of publishing. Okay, I have stated my (non) credentials and am now ready for your attack.
It is my observation that a large part of the idea of the scientific process absolutely involves publishing of knowledge gleaned and then surviving challenge by those with contrary views. All of those presenting views are to be accorded courtesy and treated honorably. At least, as I naively see the construct.
It is my further observation that in recent years there were two very important issues that absolutely did not see this process extended. I speak of those with "unusual" [a deliberate obfuscation, you might say] views on climate change and COVID. Both those topics, it is my (further, further?) observation are not treated as being subject to courteous and honorable scientific discourse but rather as religion where someone who questions the assertions is vilified and subject to dishonor.
I offer to you an analogy. As an accountant, it is my training to seek balance, i.e. the debits must equal the credits. I therefore suggest that you folks in the scientific community have a larger problem than Mr. Mastroianni has identified. I do not deny that the accepted "religion" of climate change or COVID is perhaps correct but why is no one looking at the contrary possibilities and why do you as scientists allow the vilification, dishonor, defunding, de-platforming, etc. of those who want to explore alternative possibilities?
Oh, and as to the article itself? It seems to my non-scientific way of thinking to be a very good analysis, so, Bravo, Mr. Mastroianni!
Loved the article! Now convince universities to drop the publish-or-perish requirement. How many y of my life have I spent reading rubbish in abominable APA style whose only function was to add a line to a CV!
Lots of good points here and I find myself nodding in agreement. A couple more thoughts.
One is that a system that relies on volunteer labor should generally be suspect. I got zero dollars for being an associate editor of a journal, having to assign papers to peer reviewers and then synthesize their reviews, which generally were cursory. The reviewers got zero dollars too. Oh, I know that reviewing papers is considered a 'professional obligation,' but that's the point: the peer-review system has extraordinary responsibilities for which it is willing to pay nothing. Only journals make money.
The second point is that transparency seems like a far better approach to quality control than peer review. Having to post one's data and code is highly disciplining because there's always the chance that another researcher might use them to show that you estimated a parameter or interpreted your estimates incorrectly. It's rare for someone to put in that effort, but the discipline comes from the fact that it might happen. Maybe worth an experiment.
Adam, I really love all of your writing on scientific publishing. Also, this made me go read your Things Could Be Better paper again, which might be the first time that I read a full paper in full *twice*, *for fun*.
Also just to note that the reliance of university tenure and promotion standards on measuring the publication of a certain number of "peer reviewed" articles also maintains the rigidity of this system
Ever tried questioning what the results are while doing a peer review? Ever asked about underlying assumptions? Most peer reviews are done by people that have the same background, want the same results, and are under the same impetus to publish, publish, publish. They know that if they are critical, then when it is their turn the others will be critical. I have had people get very defensive when I question anything, and I have had other people amazed and astonished that they actually get relevant feedback beyond really basic stuff! It is uncomfortable, and boring to do peer reviews. All the incentives for it are backwards.
Loud applause! I’ve published only 2 papers in the past 4 years and by “published” I mean uploaded to ReaearchGate. I’m very happy with the readership and the constructive discussions and feedback that has allowed me to further refine my theories as the field progresses.
Given the abject rubbish appearing in even the most prestigious journals, I feel no urge to submit my stuff or pay their fees.
I wonder if we made a) reviews optional b) showed the reviews c) let the authors respond to the reviews and show those too i.e. the whole conversation (like in any blog comments section!) d) give the reviewers that contributed most thoughtfully almost as much prominence as the authors themselves (in the author's opinion? or perhaps a legitimate curator role who narrates the whole 'story'?) e) linked to follow up work that supported or contradicted (again the web makes this easy) we could start to build better mechanisms for encouraging, thinking about and testing our ideas.
A wonderful article. I've been arguing something similar for many years: https://breast-cancer-research.biomedcentral.com/articles/10.1186/bcr2742 It's fascinating to me that a process at the heart of science is faith not evidence based. Indeed, believing in peer review is less scientific than believing in God because we have lots of evidence that peer review doesn't work, whereas we lack evidence that God doesn't exist. Why does the juggernaut roll on? Because there are lots of jobs, profit, and reputations that depend on continuing to believe in it?
I don't know what the solution is. Full disclosure, I've been working in journal publishing for 15 years.
You successfully self-published an article. If everyone who wrote a paper did that, there would be 100s of manuscripts uploaded weekly with zero quality control and zero discoverability unless like a self-publishing fiction author you work your ass off at social media to get noticed.
You talk about pre-1950s academic publishing, but the world was a much smaller place then and with a lot less research happening. You didn't have China pushing their academics to publish - many of them being unable graduate until they've been published.
Perhaps is this the world shaped by peer review.
Obviously, there are issues with peer review and with for-profit journal publishing. I'm certainly not 100% comfortable with the way things are going, especially since OA became the drive for profit.
I don't think my industry will survive as it is. I know other colleagues feel this way. However, I don't know what the future is.
I agree with your conclusion about the current system being suboptimal. However, I'm curious if you think there's anything more efficient than chaos? For example, what if Arxiv had an openreview-esque comments section? So that communications were centralized. Or something like a reddit-esque community governed forum?
I think your key connection here, which I, a military historian, had not made until now, is that peer review came out of those heady days of the early Cold War when THE EXPERTS arrived at the Pentagon. The most famous of these was MacNamara. I leave the Vietnam analogy in your capable hands. From my perspective, this is the moment when scientists were being asked to think unthinkable things and question every set of assumptions until they had exhausted all cognitive powers, for the stakes were so very, very high. We may forgive the motivations and still see the shortcomings here.
I should also note that this all fits with Max Weber's theory that specialist classes make informational gatekeeping processes into safeguards for their own status. Terry Shinn wrote about it studying L'Ecole Polytechnique and I have encountered it in my own study of the French naval engineering school. If you want to add this to the social history of how entire professions can fail together, for decades, there is historiography.
I don't think it failed perhaps as much as it stopped working, especially as the number of scientists exploded. A good idea becomes a bureaucratic nightmare with scale.
Also the weak link point is excellent!
The one thing I’m surprised didn’t make it into this amazing article is how the digital age transformed how we view science. As little as 25 years ago, you needed journals to physically publish and distribute your work. And no one was going to get your paper delivered without an established organization approving it first.
Today, the logistical and material cost of reading a paper is virtually zero. We convinced ourselves that journals were providing more of a service than simply being the paper boy because we needed them for that, at minimum. We then built an entire incentive and ladder climbing system based on that silly hang up.
No longer.
It seems to me that 'believe science' mostly means 'leave it to us' as a kind of argument from authority. I know someone (a scientist) who is viscerally angry that non-scientific people question the fitness of the scientific 'system' in surfacing materially objective truths about complex things without being inevitably distorted by incentives.
Maybe they're just afraid to lose elite status and peer review is simply a way to circle the wagons.
An idea: research papers should have the reviewers names listed same way as the authors. Thus, they bear the responsibility for their review. It is also a way to check if the paper is the product of a groupthink.
Okay, I am going to step into a dangerous area here.
First, I am a retired accountant, not trained in the sciences. I have acquaintances at the PhD level (including in the sciences) who have gone through the "necessity" of publishing. Okay, I have stated my (non) credentials and am now ready for your attack.
It is my observation that a large part of the idea of the scientific process absolutely involves publishing of knowledge gleaned and then surviving challenge by those with contrary views. All of those presenting views are to be accorded courtesy and treated honorably. At least, as I naively see the construct.
It is my further observation that in recent years there were two very important issues that absolutely did not see this process extended. I speak of those with "unusual" [a deliberate obfuscation, you might say] views on climate change and COVID. Both those topics, it is my (further, further?) observation are not treated as being subject to courteous and honorable scientific discourse but rather as religion where someone who questions the assertions is vilified and subject to dishonor.
I offer to you an analogy. As an accountant, it is my training to seek balance, i.e. the debits must equal the credits. I therefore suggest that you folks in the scientific community have a larger problem than Mr. Mastroianni has identified. I do not deny that the accepted "religion" of climate change or COVID is perhaps correct but why is no one looking at the contrary possibilities and why do you as scientists allow the vilification, dishonor, defunding, de-platforming, etc. of those who want to explore alternative possibilities?
Oh, and as to the article itself? It seems to my non-scientific way of thinking to be a very good analysis, so, Bravo, Mr. Mastroianni!
Loved the article! Now convince universities to drop the publish-or-perish requirement. How many y of my life have I spent reading rubbish in abominable APA style whose only function was to add a line to a CV!
PS: The alchemy link was great.
Lots of good points here and I find myself nodding in agreement. A couple more thoughts.
One is that a system that relies on volunteer labor should generally be suspect. I got zero dollars for being an associate editor of a journal, having to assign papers to peer reviewers and then synthesize their reviews, which generally were cursory. The reviewers got zero dollars too. Oh, I know that reviewing papers is considered a 'professional obligation,' but that's the point: the peer-review system has extraordinary responsibilities for which it is willing to pay nothing. Only journals make money.
The second point is that transparency seems like a far better approach to quality control than peer review. Having to post one's data and code is highly disciplining because there's always the chance that another researcher might use them to show that you estimated a parameter or interpreted your estimates incorrectly. It's rare for someone to put in that effort, but the discipline comes from the fact that it might happen. Maybe worth an experiment.
Adam, I really love all of your writing on scientific publishing. Also, this made me go read your Things Could Be Better paper again, which might be the first time that I read a full paper in full *twice*, *for fun*.
Also just to note that the reliance of university tenure and promotion standards on measuring the publication of a certain number of "peer reviewed" articles also maintains the rigidity of this system
Ever tried questioning what the results are while doing a peer review? Ever asked about underlying assumptions? Most peer reviews are done by people that have the same background, want the same results, and are under the same impetus to publish, publish, publish. They know that if they are critical, then when it is their turn the others will be critical. I have had people get very defensive when I question anything, and I have had other people amazed and astonished that they actually get relevant feedback beyond really basic stuff! It is uncomfortable, and boring to do peer reviews. All the incentives for it are backwards.
Loud applause! I’ve published only 2 papers in the past 4 years and by “published” I mean uploaded to ReaearchGate. I’m very happy with the readership and the constructive discussions and feedback that has allowed me to further refine my theories as the field progresses.
Given the abject rubbish appearing in even the most prestigious journals, I feel no urge to submit my stuff or pay their fees.
Thanks. Another great observation and article.
I wonder if we made a) reviews optional b) showed the reviews c) let the authors respond to the reviews and show those too i.e. the whole conversation (like in any blog comments section!) d) give the reviewers that contributed most thoughtfully almost as much prominence as the authors themselves (in the author's opinion? or perhaps a legitimate curator role who narrates the whole 'story'?) e) linked to follow up work that supported or contradicted (again the web makes this easy) we could start to build better mechanisms for encouraging, thinking about and testing our ideas.
A wonderful article. I've been arguing something similar for many years: https://breast-cancer-research.biomedcentral.com/articles/10.1186/bcr2742 It's fascinating to me that a process at the heart of science is faith not evidence based. Indeed, believing in peer review is less scientific than believing in God because we have lots of evidence that peer review doesn't work, whereas we lack evidence that God doesn't exist. Why does the juggernaut roll on? Because there are lots of jobs, profit, and reputations that depend on continuing to believe in it?
I don't know what the solution is. Full disclosure, I've been working in journal publishing for 15 years.
You successfully self-published an article. If everyone who wrote a paper did that, there would be 100s of manuscripts uploaded weekly with zero quality control and zero discoverability unless like a self-publishing fiction author you work your ass off at social media to get noticed.
You talk about pre-1950s academic publishing, but the world was a much smaller place then and with a lot less research happening. You didn't have China pushing their academics to publish - many of them being unable graduate until they've been published.
Perhaps is this the world shaped by peer review.
Obviously, there are issues with peer review and with for-profit journal publishing. I'm certainly not 100% comfortable with the way things are going, especially since OA became the drive for profit.
I don't think my industry will survive as it is. I know other colleagues feel this way. However, I don't know what the future is.
I agree with your conclusion about the current system being suboptimal. However, I'm curious if you think there's anything more efficient than chaos? For example, what if Arxiv had an openreview-esque comments section? So that communications were centralized. Or something like a reddit-esque community governed forum?