Aug 29, 2023·edited Aug 29, 2023Liked by Adam Mastroianni
The post section "How Much Does a Thought Weigh" makes me think of something from a paper by Tal Yarkoni, "The generalizability crisis" (https://doi.org/10.1017/S0140525X20001685). In his conclusion, he writes:
"Closer examination reveals that the inferential statistics reported in psychology articles typically have only a tenuous correspondence to the verbal claims they are intended to support. The overarching conclusion is that many fields of psychology currently operate under a kind of collective self-deception, using a thin sheen of quantitative rigor to mask inferences that remain, at their core, almost entirely qualitative."
Your excursus on the study of leadership is a great riff on Yarkoni's concern. Learning more about how the psychological sausage gets made has made me enormously skeptical of any claimed results from the discipline; the apparent fact that there is no core of understanding makes me wonder further just what the point of psychological science is.
Wild theory, that literally occurred to me 30 seconds ago, so be gentle...
Maybe there is only all this energy fuelling 'hogwash' science of this kind because creeping technocracy creates massive demand for understanding people, so that they can be 'managed'. That, plus marketing.
This is your best essay yet. Or perhaps I should say instead that it emphatically confirms my biases. I blame tenure and promotion practices: It's unpleasant to critically evaluate a colleague's work. It's much easier to assume that the editorial boards of academic journals are up to the job and that the length of one's CV is a useful index.
I'm sympathetic to the idea that science is at its best when it as it's oriented towards a concrete goal. Educational psychology seems to not be a steaming pile of garbage, probably because we all have a decent grasp of what success would look like -- more successful learning and teaching. I have no idea what the goals of social psychology are. I know there have been examples of trying to scale up nudges or whatever for policy uses, and those have mostly not worked out, but I'd think that "policy psych" or something would be a more productive field. In general, I wonder if the key moving forward is to hew closely to big real-world stuff that needs improvement.
I enjoyed this. Reminds me of the concluding paragraph of Gigerenzer's great 1998 paper, "Surrogates for theories."
"Several years ago, I spent a day and a night in a library reading through issues of the Journal of Experimental Psychology from the 1920s and 1930s. This was professionally a most depressing experience. Not because these articles were methodologically mediocre. On the contrary, many of them make today’s research pale in comparison to their diversity of methods and statistics, their detailed reporting of single-case data rather than mere averages, and their careful selection of trained subjects. And many topics—such as the influence of the gender of the experimenter on the performance of the participants—were of interest then as now. What depressed me was that almost all of this work is forgotten; it does not seem to have left a trace in the collective memory of our profession. It struck me that most of it involved collecting data without substantive theory. Data without theory are like a baby without a parent: their life expectancy is low."
Great post. Fwiw, I think the most pernicious paradigm lies at the intersection of the “cognitive biases/nudge” mania and “pick-a-noun”. I’ll call it “create a noun”...the perverse joy people get from naming a phenomenon (usually a cognitive bias) because it boosts their currency and citation index. After all, everybody would love to be Dunning...or maybe Kruger.
I would say that this article is a nail in psychology's coffin but then again, I also share Adam's sentiment that none of it really matters: Gino, Ariely, Stapel, whoever... the person doesn't matter because there are way too many people invested in the current paradigm of producing papers that don't advance our understanding of humans in any meaningful way. They are just designed to look that way, with phrases such as “Nevertheless, this is a promising finding which suggests...” and
“It is interesting to note that...”, and elaborate-looking statistics (without any real underlying theory), and a bunch of made up words that are rephrasings or slight adjustments of the words that we already use.
Worse yet, this gets perpetuated, as Adam wrote, into infinity because of vested interest -- just look at how Baumeister tries to protect his baby (https://lupinepublishers.com/psychology-behavioral-science-journal/pdf/SJPBS.MS.ID.000234.pdf), ego depletion, even after the field has collectively been forced to conclude that no, these two words are not useful fictions anymore, e.g. https://pubmed.ncbi.nlm.nih.gov/34520296/. -- and the sheer simplicity of modern psychological research (which is basically just generating a bunch of numbers on a spreadsheet through an online survey platform like mTurk, Qualtrics, whatever.... Something compels me to think that no groundbreaking discoveries will ever occur via a bunch of people in a half-zombie mode clicking at a bunch of seemingly "valid" and "reliable" statements. It's just not the platform, honestly.) And now, apparently, the paradigm gets perpetuated because of lawsuits as well, and criticism gets quashed underneath the lofty foot of the legal system.
As a fresh psychology graduate, this makes me feel incredibly aimless. Imagine studying for 5 (or sometimes even longer) years only to gradually learn that whatever you studied is a bunch of made up stuff that people sort of collectively agreed on, with limited (or at least very wonky) real-world impact. I feel like my two options are either to abandon the field and keep my sanity and honor (whatever that is in our age); or conform, and be a tiny cog in the machine that produces meaningless vapor.
It seems psychology has been in its infancy since forever. Lots of researchers gathering observations; lots of therapists with varying degrees of success. But no coherent big picture to tie it all together. Where is psychology's Newton, Lavoisier, or Darwin to give it a robust, unifying paradigm -- one that both has explanatory/predictive power and is concrete enough to be falsifiable?
"But there's no world-changing insight like relativity, evolution, or DNA, nor any smaller-but-still-very-cool discoveries like polymerase chain reaction, CRISPR, or Higgs bosons."
This isn't true. There is an extremely important, novel (when it was discovered), and (of course) replicable insight that came out of psychology: IQ. That an abstract concept like intelligence can be readily quantified and measured, that it has predictive power across almost every major field of human endeavor, and that it is, as a first order approximation, a unitary construct, are all valuable and unintuitive ideas. Psychologists just don't like the results of IQ research.
on the point about "psychology is too young to have a paradigm", my intuition is actually the opposite: that academic psychology is running into these issues because, as a field (and as a culture more broadly), we've come to a pretty sophisticated understanding of ourselves and our behaviors, and we're running up against the limit of what more there is to learn if we're limiting ourselves to just the methods used in traditional psychology (statistical analysis of survey responses, etc etc.).
the fact that we even have concepts like personality, emotion, the conscious/unconscious distinction, the concept of mental illness/health, etc etc. are indicative of a fairly evolved paradigm of what a human mind is like and how it works, more so than we had in the past. as Scott Alexander points out, "Statements like “my abuse gave me a lot of baggage that I’m still working through” involves a theory-of-mind that would have been incomprehensible a few centuries ago."
from the standpoint of further scientific discovery, I think the real fertile ground is in bridging the gap between the physical and the psychological, to get a deeper and more fine-grained understanding of how exactly the brain produces behavior & experience. to the extent that psychology limits itself to _only_ studying externally measurable actions, and tries to come up with generalizable causal laws that predict those actions, I think it will remain in a rut, because human behavior is fundamentally chaotic/complex, and there's only so much you can learn without taking a peek inside the brain itself.
interesting read. I wanted to ask in general whether you think, and to what degree do you think, these are accidental to psychology, or essential? My own view is that so much of these ongoing 'problems' follow from a deep epistemic mistake, i.e. assuming psychology is more like biology or chemistry, than it is to history or literature, and correlatively assuming psychology's methods consequently lead to objective facts of some sort, rather than interesting food for thought, questions, and interpretations that remain relative and subject to interesting alternative points of view. People read popular psychology or take psychology courses because they are interesting [or should be] not because they are discoveries. And thats how it should be. And, the significance remains as long as one is engaged critically and reflectively about whatever is being discussed, hopefully leading to insight or at least a deepening appreciation of the human condition. As we find when we read history, or literature, or philosophy. There might be progress and discoveries of sorts, but neever in the same way one expects from more objectively defined sciences. I think it is possible that this epistemic mistake in self understanding of psychology sets the stage to the professional game--- no one really takes the research that seriously. the emphasis is on professional standing and the external goods that go with the territory And, once one makes this the sole point, why not fake it or distort it?
So what if we thought of psychologists as existing in the same “space” as fiction writers and we judged their studies (now called simply “stories”) by how much they moved us or helped us understand ourselves and others? How would Gino and Ariely ... indeed, how would anyone ... fare then? I enjoyed your essay.
Good blog. I've worked in psychology for almost 40 years, and it's accurate. Most studies don't really matter, because they don't actually connect to any broader theories or nomothetic networks in meaningful ways.
The weird thing about contemporary psychology is that the most solid, best-replicated findings tend to be in the marginalized, demonized, poorly-funded fields such as intelligence research, behavior genetics, evolutionary psychology, and personality trait research. These are the fields that challenge the Blank Slate view, that make progressive liberals sad, and that don't offer any easy fixes to 'social problems'.
Whereas, the field that's at the epicenter of the replication crisis -- social psychology -- has basically been a bastion of leftist political activism (disguised as science) since the 1950s. All the excitement about 'the power of the situation', 'nudges', 'priming', etc was trying to show that 'improving' human behavior can be done easily, quickly, and cheaply, and that this validates leftist social policy interventions.
So, before psychology can sort itself out, it really needs to re-examine its fundamental _political_ biases, and the ways that political ideologies (especially the Blank Slate dogma) have undermined its scientific integrity for many decades/
The post section "How Much Does a Thought Weigh" makes me think of something from a paper by Tal Yarkoni, "The generalizability crisis" (https://doi.org/10.1017/S0140525X20001685). In his conclusion, he writes:
"Closer examination reveals that the inferential statistics reported in psychology articles typically have only a tenuous correspondence to the verbal claims they are intended to support. The overarching conclusion is that many fields of psychology currently operate under a kind of collective self-deception, using a thin sheen of quantitative rigor to mask inferences that remain, at their core, almost entirely qualitative."
Your excursus on the study of leadership is a great riff on Yarkoni's concern. Learning more about how the psychological sausage gets made has made me enormously skeptical of any claimed results from the discipline; the apparent fact that there is no core of understanding makes me wonder further just what the point of psychological science is.
Wild theory, that literally occurred to me 30 seconds ago, so be gentle...
Maybe there is only all this energy fuelling 'hogwash' science of this kind because creeping technocracy creates massive demand for understanding people, so that they can be 'managed'. That, plus marketing.
> As a young psychologist, this chills me to my bones
Bunches of bees don’t have bones!
This is your best essay yet. Or perhaps I should say instead that it emphatically confirms my biases. I blame tenure and promotion practices: It's unpleasant to critically evaluate a colleague's work. It's much easier to assume that the editorial boards of academic journals are up to the job and that the length of one's CV is a useful index.
I'm sympathetic to the idea that science is at its best when it as it's oriented towards a concrete goal. Educational psychology seems to not be a steaming pile of garbage, probably because we all have a decent grasp of what success would look like -- more successful learning and teaching. I have no idea what the goals of social psychology are. I know there have been examples of trying to scale up nudges or whatever for policy uses, and those have mostly not worked out, but I'd think that "policy psych" or something would be a more productive field. In general, I wonder if the key moving forward is to hew closely to big real-world stuff that needs improvement.
I enjoyed this. Reminds me of the concluding paragraph of Gigerenzer's great 1998 paper, "Surrogates for theories."
"Several years ago, I spent a day and a night in a library reading through issues of the Journal of Experimental Psychology from the 1920s and 1930s. This was professionally a most depressing experience. Not because these articles were methodologically mediocre. On the contrary, many of them make today’s research pale in comparison to their diversity of methods and statistics, their detailed reporting of single-case data rather than mere averages, and their careful selection of trained subjects. And many topics—such as the influence of the gender of the experimenter on the performance of the participants—were of interest then as now. What depressed me was that almost all of this work is forgotten; it does not seem to have left a trace in the collective memory of our profession. It struck me that most of it involved collecting data without substantive theory. Data without theory are like a baby without a parent: their life expectancy is low."
If your find this post interesting I recommend looking into critical psychology (https://en.wikipedia.org/wiki/Critical_psychology).
These people though long and hard about these problems in the 1970s, 1980s and 1990s and there's a few people who still work within this tradition.
I have found these two books to be good introductions to the topic:
- "Psychology, Subjectivity, and Society: An Introduction to German Critical Psychology" by Charles Tolman
- "Psychology from the Standpoint of the Subject: Selected Writings of Klaus Holzkamp" edited by Ernst Schraube and Ute Osterkamp.
Great post. Fwiw, I think the most pernicious paradigm lies at the intersection of the “cognitive biases/nudge” mania and “pick-a-noun”. I’ll call it “create a noun”...the perverse joy people get from naming a phenomenon (usually a cognitive bias) because it boosts their currency and citation index. After all, everybody would love to be Dunning...or maybe Kruger.
Construal Level Theory is a new idea that I have found insightful, useful, and built upon theoretically.
I would say that this article is a nail in psychology's coffin but then again, I also share Adam's sentiment that none of it really matters: Gino, Ariely, Stapel, whoever... the person doesn't matter because there are way too many people invested in the current paradigm of producing papers that don't advance our understanding of humans in any meaningful way. They are just designed to look that way, with phrases such as “Nevertheless, this is a promising finding which suggests...” and
“It is interesting to note that...”, and elaborate-looking statistics (without any real underlying theory), and a bunch of made up words that are rephrasings or slight adjustments of the words that we already use.
Worse yet, this gets perpetuated, as Adam wrote, into infinity because of vested interest -- just look at how Baumeister tries to protect his baby (https://lupinepublishers.com/psychology-behavioral-science-journal/pdf/SJPBS.MS.ID.000234.pdf), ego depletion, even after the field has collectively been forced to conclude that no, these two words are not useful fictions anymore, e.g. https://pubmed.ncbi.nlm.nih.gov/34520296/. -- and the sheer simplicity of modern psychological research (which is basically just generating a bunch of numbers on a spreadsheet through an online survey platform like mTurk, Qualtrics, whatever.... Something compels me to think that no groundbreaking discoveries will ever occur via a bunch of people in a half-zombie mode clicking at a bunch of seemingly "valid" and "reliable" statements. It's just not the platform, honestly.) And now, apparently, the paradigm gets perpetuated because of lawsuits as well, and criticism gets quashed underneath the lofty foot of the legal system.
As a fresh psychology graduate, this makes me feel incredibly aimless. Imagine studying for 5 (or sometimes even longer) years only to gradually learn that whatever you studied is a bunch of made up stuff that people sort of collectively agreed on, with limited (or at least very wonky) real-world impact. I feel like my two options are either to abandon the field and keep my sanity and honor (whatever that is in our age); or conform, and be a tiny cog in the machine that produces meaningless vapor.
A great read as usual.
It seems psychology has been in its infancy since forever. Lots of researchers gathering observations; lots of therapists with varying degrees of success. But no coherent big picture to tie it all together. Where is psychology's Newton, Lavoisier, or Darwin to give it a robust, unifying paradigm -- one that both has explanatory/predictive power and is concrete enough to be falsifiable?
"But there's no world-changing insight like relativity, evolution, or DNA, nor any smaller-but-still-very-cool discoveries like polymerase chain reaction, CRISPR, or Higgs bosons."
This isn't true. There is an extremely important, novel (when it was discovered), and (of course) replicable insight that came out of psychology: IQ. That an abstract concept like intelligence can be readily quantified and measured, that it has predictive power across almost every major field of human endeavor, and that it is, as a first order approximation, a unitary construct, are all valuable and unintuitive ideas. Psychologists just don't like the results of IQ research.
on the point about "psychology is too young to have a paradigm", my intuition is actually the opposite: that academic psychology is running into these issues because, as a field (and as a culture more broadly), we've come to a pretty sophisticated understanding of ourselves and our behaviors, and we're running up against the limit of what more there is to learn if we're limiting ourselves to just the methods used in traditional psychology (statistical analysis of survey responses, etc etc.).
the fact that we even have concepts like personality, emotion, the conscious/unconscious distinction, the concept of mental illness/health, etc etc. are indicative of a fairly evolved paradigm of what a human mind is like and how it works, more so than we had in the past. as Scott Alexander points out, "Statements like “my abuse gave me a lot of baggage that I’m still working through” involves a theory-of-mind that would have been incomprehensible a few centuries ago."
from the standpoint of further scientific discovery, I think the real fertile ground is in bridging the gap between the physical and the psychological, to get a deeper and more fine-grained understanding of how exactly the brain produces behavior & experience. to the extent that psychology limits itself to _only_ studying externally measurable actions, and tries to come up with generalizable causal laws that predict those actions, I think it will remain in a rut, because human behavior is fundamentally chaotic/complex, and there's only so much you can learn without taking a peek inside the brain itself.
interesting read. I wanted to ask in general whether you think, and to what degree do you think, these are accidental to psychology, or essential? My own view is that so much of these ongoing 'problems' follow from a deep epistemic mistake, i.e. assuming psychology is more like biology or chemistry, than it is to history or literature, and correlatively assuming psychology's methods consequently lead to objective facts of some sort, rather than interesting food for thought, questions, and interpretations that remain relative and subject to interesting alternative points of view. People read popular psychology or take psychology courses because they are interesting [or should be] not because they are discoveries. And thats how it should be. And, the significance remains as long as one is engaged critically and reflectively about whatever is being discussed, hopefully leading to insight or at least a deepening appreciation of the human condition. As we find when we read history, or literature, or philosophy. There might be progress and discoveries of sorts, but neever in the same way one expects from more objectively defined sciences. I think it is possible that this epistemic mistake in self understanding of psychology sets the stage to the professional game--- no one really takes the research that seriously. the emphasis is on professional standing and the external goods that go with the territory And, once one makes this the sole point, why not fake it or distort it?
So what if we thought of psychologists as existing in the same “space” as fiction writers and we judged their studies (now called simply “stories”) by how much they moved us or helped us understand ourselves and others? How would Gino and Ariely ... indeed, how would anyone ... fare then? I enjoyed your essay.
Good blog. I've worked in psychology for almost 40 years, and it's accurate. Most studies don't really matter, because they don't actually connect to any broader theories or nomothetic networks in meaningful ways.
The weird thing about contemporary psychology is that the most solid, best-replicated findings tend to be in the marginalized, demonized, poorly-funded fields such as intelligence research, behavior genetics, evolutionary psychology, and personality trait research. These are the fields that challenge the Blank Slate view, that make progressive liberals sad, and that don't offer any easy fixes to 'social problems'.
Whereas, the field that's at the epicenter of the replication crisis -- social psychology -- has basically been a bastion of leftist political activism (disguised as science) since the 1950s. All the excitement about 'the power of the situation', 'nudges', 'priming', etc was trying to show that 'improving' human behavior can be done easily, quickly, and cheaply, and that this validates leftist social policy interventions.
So, before psychology can sort itself out, it really needs to re-examine its fundamental _political_ biases, and the ways that political ideologies (especially the Blank Slate dogma) have undermined its scientific integrity for many decades/