Aug 29, 2023·edited Aug 29, 2023Liked by Adam Mastroianni
The post section "How Much Does a Thought Weigh" makes me think of something from a paper by Tal Yarkoni, "The generalizability crisis" (https://doi.org/10.1017/S0140525X20001685). In his conclusion, he writes:
"Closer examination reveals that the inferential statistics reported in psychology articles typically have only a tenuous correspondence to the verbal claims they are intended to support. The overarching conclusion is that many fields of psychology currently operate under a kind of collective self-deception, using a thin sheen of quantitative rigor to mask inferences that remain, at their core, almost entirely qualitative."
Your excursus on the study of leadership is a great riff on Yarkoni's concern. Learning more about how the psychological sausage gets made has made me enormously skeptical of any claimed results from the discipline; the apparent fact that there is no core of understanding makes me wonder further just what the point of psychological science is.
the terms like "leadership" are useful terms to describe what we observe as phenomenon in a high-level context (where a human ego perceives social reality), but may not actually describe any phenomenon at the neurological/cognitive level.
perhaps the point should be to connect neuroscience to these high-level concepts.
Read Dean Radin's book Supernormal. It lists over a 1000 peer reviewed journal articles supporting psi phenomena. Then read The Nature of Personal Reality by Jane Roberts.
Wild theory, that literally occurred to me 30 seconds ago, so be gentle...
Maybe there is only all this energy fuelling 'hogwash' science of this kind because creeping technocracy creates massive demand for understanding people, so that they can be 'managed'. That, plus marketing.
I'm sympathetic to the idea that science is at its best when it as it's oriented towards a concrete goal. Educational psychology seems to not be a steaming pile of garbage, probably because we all have a decent grasp of what success would look like -- more successful learning and teaching. I have no idea what the goals of social psychology are. I know there have been examples of trying to scale up nudges or whatever for policy uses, and those have mostly not worked out, but I'd think that "policy psych" or something would be a more productive field. In general, I wonder if the key moving forward is to hew closely to big real-world stuff that needs improvement.
I'm sympathetic to this theoretically, but I think you always need people mucking around on basic questions to really push the envelope. When you're only trying to solve practical problems, you often don't have the freedom to consider weird ideas that might end up upending things.
This is your best essay yet. Or perhaps I should say instead that it emphatically confirms my biases. I blame tenure and promotion practices: It's unpleasant to critically evaluate a colleague's work. It's much easier to assume that the editorial boards of academic journals are up to the job and that the length of one's CV is a useful index.
It's very hard to publicly entertain that it's quite possible that nothing anybody is doing matters. But if only 99% doesn't matter, we can all secretly think we're in the 1%. 🙂
I enjoyed this. Reminds me of the concluding paragraph of Gigerenzer's great 1998 paper, "Surrogates for theories."
"Several years ago, I spent a day and a night in a library reading through issues of the Journal of Experimental Psychology from the 1920s and 1930s. This was professionally a most depressing experience. Not because these articles were methodologically mediocre. On the contrary, many of them make today’s research pale in comparison to their diversity of methods and statistics, their detailed reporting of single-case data rather than mere averages, and their careful selection of trained subjects. And many topics—such as the influence of the gender of the experimenter on the performance of the participants—were of interest then as now. What depressed me was that almost all of this work is forgotten; it does not seem to have left a trace in the collective memory of our profession. It struck me that most of it involved collecting data without substantive theory. Data without theory are like a baby without a parent: their life expectancy is low."
Great post. Fwiw, I think the most pernicious paradigm lies at the intersection of the “cognitive biases/nudge” mania and “pick-a-noun”. I’ll call it “create a noun”...the perverse joy people get from naming a phenomenon (usually a cognitive bias) because it boosts their currency and citation index. After all, everybody would love to be Dunning...or maybe Kruger.
Adam, congratulations for your beautiful and elegant post! I agree with @rabchi and I believe that you have ran into an intersection of psychology and philosophy, regarding the epistemology of the field. How can psychologists actually know what they are aiming for since the very notion of a "psychology" is a fiction like any of the other examples you have mentioned?
Of course I'm not the first to suggest this: my main reference is the work of Richard Rorty. In his book Philosphy and the Mirror of Nature, he departs from a single presupposition: that the idea of "mind" has guided science for a long time, but it should be reconsidered as something that resembles more of a metaphor (or a fiction) perpetrated by a cartesian tradition than the actual basis that should be used by scientific enquiries. He then proceeds to point problems with behaviorist or materialist methods to "identify" what could be these mental activities.
TLDR: philosophy could be an useful approach to redefine strong links for psychology.
Grammarly check for run-on sentences, and its known as a fused sentence or comma splice, occurs when two independent clauses are improperly joined without proper punctuation or a conjunction. Independent clauses, capable of standing alone as complete sentences, merge into a confusing and jarring read when incorrectly combined.
I would say that this article is a nail in psychology's coffin but then again, I also share Adam's sentiment that none of it really matters: Gino, Ariely, Stapel, whoever... the person doesn't matter because there are way too many people invested in the current paradigm of producing papers that don't advance our understanding of humans in any meaningful way. They are just designed to look that way, with phrases such as “Nevertheless, this is a promising finding which suggests...” and
“It is interesting to note that...”, and elaborate-looking statistics (without any real underlying theory), and a bunch of made up words that are rephrasings or slight adjustments of the words that we already use.
Worse yet, this gets perpetuated, as Adam wrote, into infinity because of vested interest -- just look at how Baumeister tries to protect his baby (https://lupinepublishers.com/psychology-behavioral-science-journal/pdf/SJPBS.MS.ID.000234.pdf), ego depletion, even after the field has collectively been forced to conclude that no, these two words are not useful fictions anymore, e.g. https://pubmed.ncbi.nlm.nih.gov/34520296/. -- and the sheer simplicity of modern psychological research (which is basically just generating a bunch of numbers on a spreadsheet through an online survey platform like mTurk, Qualtrics, whatever.... Something compels me to think that no groundbreaking discoveries will ever occur via a bunch of people in a half-zombie mode clicking at a bunch of seemingly "valid" and "reliable" statements. It's just not the platform, honestly.) And now, apparently, the paradigm gets perpetuated because of lawsuits as well, and criticism gets quashed underneath the lofty foot of the legal system.
As a fresh psychology graduate, this makes me feel incredibly aimless. Imagine studying for 5 (or sometimes even longer) years only to gradually learn that whatever you studied is a bunch of made up stuff that people sort of collectively agreed on, with limited (or at least very wonky) real-world impact. I feel like my two options are either to abandon the field and keep my sanity and honor (whatever that is in our age); or conform, and be a tiny cog in the machine that produces meaningless vapor.
I left, many years ago. For me, it was a good decision. The material you learned in Stat, as well as the determination you've shown in getting your "working papers" are both prized and transferable to *many* well respected occupations and professions.
Generally, I keep my Ψ degree, under my hat; all too many people worry that you're analyzing them. ....R O T F L
I feel the same. I'll be honest, until I checked the citations just now I thought it was a lot older than it was. I do think there's a basket of ideas like these in psychology––mid-level theories that involve empirical results but really are, at their core, a useful way of looking at things. They're not on the level of evolution, nor even a hundredth of that size, so it's hard to call them major. But they are important and useful.
I was thinking of Relational Frame Theory as another mid-level theory that appears to have some empirical support while being "a useful way of looking at things." It even might have some interesting overlap with Construal Level Theory in terms of the relationship between increasing abstraction and how we engage with reality. Maybe one way major theories emerge is not through some one-off visionary leap, but via cumulative insights from these kind of mid-level theories that slowly reveal complementary features.
It seems psychology has been in its infancy since forever. Lots of researchers gathering observations; lots of therapists with varying degrees of success. But no coherent big picture to tie it all together. Where is psychology's Newton, Lavoisier, or Darwin to give it a robust, unifying paradigm -- one that both has explanatory/predictive power and is concrete enough to be falsifiable?
on the point about "psychology is too young to have a paradigm", my intuition is actually the opposite: that academic psychology is running into these issues because, as a field (and as a culture more broadly), we've come to a pretty sophisticated understanding of ourselves and our behaviors, and we're running up against the limit of what more there is to learn if we're limiting ourselves to just the methods used in traditional psychology (statistical analysis of survey responses, etc etc.).
the fact that we even have concepts like personality, emotion, the conscious/unconscious distinction, the concept of mental illness/health, etc etc. are indicative of a fairly evolved paradigm of what a human mind is like and how it works, more so than we had in the past. as Scott Alexander points out, "Statements like “my abuse gave me a lot of baggage that I’m still working through” involves a theory-of-mind that would have been incomprehensible a few centuries ago."
from the standpoint of further scientific discovery, I think the real fertile ground is in bridging the gap between the physical and the psychological, to get a deeper and more fine-grained understanding of how exactly the brain produces behavior & experience. to the extent that psychology limits itself to _only_ studying externally measurable actions, and tries to come up with generalizable causal laws that predict those actions, I think it will remain in a rut, because human behavior is fundamentally chaotic/complex, and there's only so much you can learn without taking a peek inside the brain itself.
So, to say complexity and chaos, systemic thinking, neuropsychology, brain science, networks, AI. All these fields are paradigms of "pure" and maybe "hard" science against the self-fulfilling prophecy of questionnaires. Keeping of course in mind that Psychology must study humans in a humanitarian way.
"But there's no world-changing insight like relativity, evolution, or DNA, nor any smaller-but-still-very-cool discoveries like polymerase chain reaction, CRISPR, or Higgs bosons."
This isn't true. There is an extremely important, novel (when it was discovered), and (of course) replicable insight that came out of psychology: IQ. That an abstract concept like intelligence can be readily quantified and measured, that it has predictive power across almost every major field of human endeavor, and that it is, as a first order approximation, a unitary construct, are all valuable and unintuitive ideas. Psychologists just don't like the results of IQ research.
IQ is over 100 years old, so it's not quite recent. It's also a good example of the limitations of abstractions. It's an undeniably useful concept, but it also fools us into thinking we've mapped the whole space when in fact we've only mapped a sliver. https://www.experimental-history.com/p/why-arent-smart-people-happier
interesting read. I wanted to ask in general whether you think, and to what degree do you think, these are accidental to psychology, or essential? My own view is that so much of these ongoing 'problems' follow from a deep epistemic mistake, i.e. assuming psychology is more like biology or chemistry, than it is to history or literature, and correlatively assuming psychology's methods consequently lead to objective facts of some sort, rather than interesting food for thought, questions, and interpretations that remain relative and subject to interesting alternative points of view. People read popular psychology or take psychology courses because they are interesting [or should be] not because they are discoveries. And thats how it should be. And, the significance remains as long as one is engaged critically and reflectively about whatever is being discussed, hopefully leading to insight or at least a deepening appreciation of the human condition. As we find when we read history, or literature, or philosophy. There might be progress and discoveries of sorts, but neever in the same way one expects from more objectively defined sciences. I think it is possible that this epistemic mistake in self understanding of psychology sets the stage to the professional game--- no one really takes the research that seriously. the emphasis is on professional standing and the external goods that go with the territory And, once one makes this the sole point, why not fake it or distort it?
TL;DR You are correct, and the reason is that Psychology gave up on theories. But nothing is going to change.
Wall of text, who has time to read, etc.
When I started studying Psychology 20 years ago, I immediately had the same questions you have right now. It was the same situation back then as it is now. If so many studies turn out not to be replicable, what are we even doing here?
It took me very long to get a grip on what is going on, and why. But others wrote it before me. Psychology is unable to produce real theories (e.g., Gigerenzer, 1998). There is no Theoretical Psychology. Try finding a book on relevant theories in Psychology. There are none. Only introductions to Psychology, listing a bunch of experimental paradigms and results.
What are real theories? They are specifications of robust relationships, mostly causal relationships. Mathematically, I believe they are probably structures (a type of mathematical objects), but that’s another topic.
All science is inductive in the sense that it cannot guarantee the consequences if a set of premises is true, that is only possible in formal systems. However, F = m*a does a darn good job in many situations. It relates important measurable quantities and can tell us what the result will be. If you calculate a force knowing mass and acceleration, the result is valid in the sense that you can have faith in it. You can base decisions on it and will be correct. Many, perhaps most results produced in Psychology are not valid in the sense that you should not trust them. You should be very careful if you base a decision on such results.
I believe that since about the mid 1990s, Psychology has given up on producing theories that encode and abstract robust relationships. That was not always the case. If you read Human Error by James Reason (1991), you find a very different future of Psychology. It was full of formalized robust relationships that could make quantitative predictions for important real life problems. This future has never materialized.
Why not? Several reasons.
First, to specify psychological insights, you need to know math and logic. Like really. Math is not a part of our curriculum. „Statistics“ is just learning how to click buttons in SPSS, it has nothing to do with mathematical statistics. Psychologists are, by and large, incapable to abstract from empirical insights from specific studies. Formalization is not a part of psychological culture. You can’t abstract with words, you need formalization.
Second, demographics. Most people who study psychology do it because they are interested in people, not in things. They want to become therapists, counselors, but not ponder formalization of empirical results. There is not critical mass demanding to learn how to build and use reliable theories (or even methods). However, there are a lot of people who want to get a degree and go on with their lives. You don’t need math as a school counselor.
Third, Artificial Intelligence. Much of what 1980s Psychologists where sure was going to happen did not because they bet the house on purely symbolic AI. Most of the research program of cognitive architectures was born out of this view. As we know, this bet was lost. It’s mostly subsymbolic and computational statistics now. Psychology did not even try to keep up, it simply gave up.
That is not to say we never did anything right. Psychology made crucial methodological contributions, just two examples. Structual equation modelling is a direct attempt to specify causal relationships based on empirical results. One of the smartest minds ever helped building it, Judea Perl. You might know him as one of the inventors of Bayesian Networks (yes really), and he teamed up with statisticians and psychologists to do it. Another invaluable contribution was to (generalized) linear mixed models. The interpretation of fixed vs. random factors is really tricky, and much empirical work in (educational!!) Psychology helped to clarify that distinction.
A bit of bad luck is that sometimes, we produce robust relationships (i.e, trustworthy results), and the world ignores it. For example, we 100% predicted the Tesla accidents by showing over and over again that drivers cannot react quickly enough in time to silent failures. We predicted that 15 years ago, I conducted such experiments back then.
And we are experts at ignoring our own best results. One of the very few real attempts at a theory is a brilliant theory on causality by Patricia Cheng (1997). The more you know about how little we know about causality, the more in awe you’ll be about this paper. Another example is „Frontiers of Test Validity Theory“ (Markus & Borsboom, 2013). It’s not the best edited book, but it layes out the very specific problems we have when trying to (really!) measure latent variables. In many respects, that is trying to translate Kant into actionable statistics. A research program addressing causality and a latent variable theory could become the gold standard for many other fields, far beyond Psychology.
Alas, that is never going to happen. The status quo is too convenient for too many people. I’ve seen it too often that we produced insights of interest, just to be discarded by those who should be interested. For example, changes in traffic safety are largely only achievable by adapting the infrastructure. That is not something politicians want to hear. They want cheap, magical tricks like „Write your name on every page.“ It really, really pays to produce such papers. Next, again, demographics. I get it, what does a therapist need Linear Algebra and Predicate Logic for. But this is our critical mass. Thus, we have none.
I could go on and on, but let’s be real, who cares? We get paid, we can do experiments all day long and create a gripping narrative afterwards, tell everyone how our research will change the world, etc.
If you really want to move something, leave the field. You can’t win against the system. Yes I’m incredibly frustrated, but that’s the reality. Psychology as a paid profession will endure because there is a market for the activity, but it will not change because there is no impetus for a Kuhnian structural revolution. You can have only if there is some sort of validation criterion, if it pays to predict correct results. Unfortunately, that is not the case in today’s Psychology, as you have so eloquently shown.
If mathematical formalization was the answer, economics wouldn't have the same failure rates as psychology by most measures. You can lie with math too, especially to yourself.
I'm not 100% sure what you mean, I take it to mean „doing lots of math is not a sufficient condition for good science“. I agree, but I probably do not agree with your premises.
First, formalization comes in many shapes and forms. I think of it as agreements about which things exist, and how they are related. Think of predicate logic, ontologies, or even musical notes. Quantification can help to clarify such relations, but in general it probably plays a small role in understanding the world. I'm not an expert in economics, but I think they use lots of numbers simply because economies have lots of things you can count. At the end of the day, they struggle with the same issue: What do these numbers really mean? What latent structure do they refer to?
Second, I do not agree that replicability on its own is a good measure for scientific success. Pretty much every empirical science struggles with the precursor to replicability, i.e., reproducibility. Psychology is doing worse than some, but the natural sciences or engineering disciplines are not necessarily doing stellar. Just search for something like „reproducibility medicine biochemistry“ on Google Scholar. Fun times, I promise. And many empirical results *cannot* be reproduced. Ever tried to build a Hadron collider in your backyard? If you cannot reproduce empirical results, how do you decide whether they are replicable?
This brings me to my third point. Popper said it best: Science is what we can omit. It is exactly what I referred to as „uncovering robust relations“. Galilei threw things from the Tower of Pisa to show that things of different weight can fall at the same speed. You can easily reproduce that result in your backyard. You do not need to know the time of year Galilei did it, or at what angle the Tower of Pisa stood when Galilei did it, or which things he exactly used. Science is taking things away until we arrive at a firm core which cannot be altered further. To me, this is what formalization is all about.
My point is: If all studies are done extremely well but only investigate garbage, science will not have moved one inch forward. Without good theories, you do not know what you should study in the first place, and if it goes wrong you do not know why. Einstein (1949) said „Thinking without the positing of categories and concepts in general would be as impossible as breathing in a vacuum.“ You need a metastructure that produces these categories, which is a theory. Better theories will lead to replicable results because you know what you can leave out to reproduce a study. Is the time of the year important, the object to throw down and its angle? What are the irreducible relationships which let you repeat what others have done before you?
By blindly searching for „effects“ without the faintest idea of why they should occur we have no idea what to leave out. We then do not know whether we could not reproduce an existing effect because we got the setup wrong, or if the effect cannot be replicated because it most probably does not exist. This is the real tragedy, not whether some crooks faked their data.
TL;DR: successful replication isn't proof of scientific progress, and we need better theories.
Agreed. But the absence of replication success is definitely evidence that something is deeply wrong, regardless of what we should/shouldn't be doing with theories.
So formalization in economics hasn't proved a better approach than the hypothesis testing in psychology. Because the problem isn't primarily methodological, it's cultural.
So what if we thought of psychologists as existing in the same “space” as fiction writers and we judged their studies (now called simply “stories”) by how much they moved us or helped us understand ourselves and others? How would Gino and Ariely ... indeed, how would anyone ... fare then? I enjoyed your essay.
The post section "How Much Does a Thought Weigh" makes me think of something from a paper by Tal Yarkoni, "The generalizability crisis" (https://doi.org/10.1017/S0140525X20001685). In his conclusion, he writes:
"Closer examination reveals that the inferential statistics reported in psychology articles typically have only a tenuous correspondence to the verbal claims they are intended to support. The overarching conclusion is that many fields of psychology currently operate under a kind of collective self-deception, using a thin sheen of quantitative rigor to mask inferences that remain, at their core, almost entirely qualitative."
Your excursus on the study of leadership is a great riff on Yarkoni's concern. Learning more about how the psychological sausage gets made has made me enormously skeptical of any claimed results from the discipline; the apparent fact that there is no core of understanding makes me wonder further just what the point of psychological science is.
the terms like "leadership" are useful terms to describe what we observe as phenomenon in a high-level context (where a human ego perceives social reality), but may not actually describe any phenomenon at the neurological/cognitive level.
perhaps the point should be to connect neuroscience to these high-level concepts.
Read Dean Radin's book Supernormal. It lists over a 1000 peer reviewed journal articles supporting psi phenomena. Then read The Nature of Personal Reality by Jane Roberts.
Wild theory, that literally occurred to me 30 seconds ago, so be gentle...
Maybe there is only all this energy fuelling 'hogwash' science of this kind because creeping technocracy creates massive demand for understanding people, so that they can be 'managed'. That, plus marketing.
A large portion of social psychology now lives in marketing and management departments, so this fits
The psychology lab to airport bookshelf pipeline is well oiled
> As a young psychologist, this chills me to my bones
Bunches of bees don’t have bones!
Oh no whose bones are these then
I'm sympathetic to the idea that science is at its best when it as it's oriented towards a concrete goal. Educational psychology seems to not be a steaming pile of garbage, probably because we all have a decent grasp of what success would look like -- more successful learning and teaching. I have no idea what the goals of social psychology are. I know there have been examples of trying to scale up nudges or whatever for policy uses, and those have mostly not worked out, but I'd think that "policy psych" or something would be a more productive field. In general, I wonder if the key moving forward is to hew closely to big real-world stuff that needs improvement.
I'm sympathetic to this theoretically, but I think you always need people mucking around on basic questions to really push the envelope. When you're only trying to solve practical problems, you often don't have the freedom to consider weird ideas that might end up upending things.
Absolutely agree on this as well.
This is your best essay yet. Or perhaps I should say instead that it emphatically confirms my biases. I blame tenure and promotion practices: It's unpleasant to critically evaluate a colleague's work. It's much easier to assume that the editorial boards of academic journals are up to the job and that the length of one's CV is a useful index.
It's very hard to publicly entertain that it's quite possible that nothing anybody is doing matters. But if only 99% doesn't matter, we can all secretly think we're in the 1%. 🙂
I do! I do!
I enjoyed this. Reminds me of the concluding paragraph of Gigerenzer's great 1998 paper, "Surrogates for theories."
"Several years ago, I spent a day and a night in a library reading through issues of the Journal of Experimental Psychology from the 1920s and 1930s. This was professionally a most depressing experience. Not because these articles were methodologically mediocre. On the contrary, many of them make today’s research pale in comparison to their diversity of methods and statistics, their detailed reporting of single-case data rather than mere averages, and their careful selection of trained subjects. And many topics—such as the influence of the gender of the experimenter on the performance of the participants—were of interest then as now. What depressed me was that almost all of this work is forgotten; it does not seem to have left a trace in the collective memory of our profession. It struck me that most of it involved collecting data without substantive theory. Data without theory are like a baby without a parent: their life expectancy is low."
Great post. Fwiw, I think the most pernicious paradigm lies at the intersection of the “cognitive biases/nudge” mania and “pick-a-noun”. I’ll call it “create a noun”...the perverse joy people get from naming a phenomenon (usually a cognitive bias) because it boosts their currency and citation index. After all, everybody would love to be Dunning...or maybe Kruger.
This is a good point. When we can create new nouns, we aren't even limited by the dictionary.
Nounification! Now waiting for the citations to start rolling in.
If your find this post interesting I recommend looking into critical psychology (https://en.wikipedia.org/wiki/Critical_psychology).
These people though long and hard about these problems in the 1970s, 1980s and 1990s and there's a few people who still work within this tradition.
I have found these two books to be good introductions to the topic:
- "Psychology, Subjectivity, and Society: An Introduction to German Critical Psychology" by Charles Tolman
- "Psychology from the Standpoint of the Subject: Selected Writings of Klaus Holzkamp" edited by Ernst Schraube and Ute Osterkamp.
Thanks!
Adam, congratulations for your beautiful and elegant post! I agree with @rabchi and I believe that you have ran into an intersection of psychology and philosophy, regarding the epistemology of the field. How can psychologists actually know what they are aiming for since the very notion of a "psychology" is a fiction like any of the other examples you have mentioned?
Of course I'm not the first to suggest this: my main reference is the work of Richard Rorty. In his book Philosphy and the Mirror of Nature, he departs from a single presupposition: that the idea of "mind" has guided science for a long time, but it should be reconsidered as something that resembles more of a metaphor (or a fiction) perpetrated by a cartesian tradition than the actual basis that should be used by scientific enquiries. He then proceeds to point problems with behaviorist or materialist methods to "identify" what could be these mental activities.
TLDR: philosophy could be an useful approach to redefine strong links for psychology.
Grammarly check for run-on sentences, and its known as a fused sentence or comma splice, occurs when two independent clauses are improperly joined without proper punctuation or a conjunction. Independent clauses, capable of standing alone as complete sentences, merge into a confusing and jarring read when incorrectly combined.
https://runonchecker.io/
I would say that this article is a nail in psychology's coffin but then again, I also share Adam's sentiment that none of it really matters: Gino, Ariely, Stapel, whoever... the person doesn't matter because there are way too many people invested in the current paradigm of producing papers that don't advance our understanding of humans in any meaningful way. They are just designed to look that way, with phrases such as “Nevertheless, this is a promising finding which suggests...” and
“It is interesting to note that...”, and elaborate-looking statistics (without any real underlying theory), and a bunch of made up words that are rephrasings or slight adjustments of the words that we already use.
Worse yet, this gets perpetuated, as Adam wrote, into infinity because of vested interest -- just look at how Baumeister tries to protect his baby (https://lupinepublishers.com/psychology-behavioral-science-journal/pdf/SJPBS.MS.ID.000234.pdf), ego depletion, even after the field has collectively been forced to conclude that no, these two words are not useful fictions anymore, e.g. https://pubmed.ncbi.nlm.nih.gov/34520296/. -- and the sheer simplicity of modern psychological research (which is basically just generating a bunch of numbers on a spreadsheet through an online survey platform like mTurk, Qualtrics, whatever.... Something compels me to think that no groundbreaking discoveries will ever occur via a bunch of people in a half-zombie mode clicking at a bunch of seemingly "valid" and "reliable" statements. It's just not the platform, honestly.) And now, apparently, the paradigm gets perpetuated because of lawsuits as well, and criticism gets quashed underneath the lofty foot of the legal system.
As a fresh psychology graduate, this makes me feel incredibly aimless. Imagine studying for 5 (or sometimes even longer) years only to gradually learn that whatever you studied is a bunch of made up stuff that people sort of collectively agreed on, with limited (or at least very wonky) real-world impact. I feel like my two options are either to abandon the field and keep my sanity and honor (whatever that is in our age); or conform, and be a tiny cog in the machine that produces meaningless vapor.
I left, many years ago. For me, it was a good decision. The material you learned in Stat, as well as the determination you've shown in getting your "working papers" are both prized and transferable to *many* well respected occupations and professions.
Generally, I keep my Ψ degree, under my hat; all too many people worry that you're analyzing them. ....R O T F L
Construal Level Theory is a new idea that I have found insightful, useful, and built upon theoretically.
I feel the same. I'll be honest, until I checked the citations just now I thought it was a lot older than it was. I do think there's a basket of ideas like these in psychology––mid-level theories that involve empirical results but really are, at their core, a useful way of looking at things. They're not on the level of evolution, nor even a hundredth of that size, so it's hard to call them major. But they are important and useful.
I was thinking of Relational Frame Theory as another mid-level theory that appears to have some empirical support while being "a useful way of looking at things." It even might have some interesting overlap with Construal Level Theory in terms of the relationship between increasing abstraction and how we engage with reality. Maybe one way major theories emerge is not through some one-off visionary leap, but via cumulative insights from these kind of mid-level theories that slowly reveal complementary features.
A great read as usual.
It seems psychology has been in its infancy since forever. Lots of researchers gathering observations; lots of therapists with varying degrees of success. But no coherent big picture to tie it all together. Where is psychology's Newton, Lavoisier, or Darwin to give it a robust, unifying paradigm -- one that both has explanatory/predictive power and is concrete enough to be falsifiable?
Iain McGilchrist might be the man with his theory of right/left brain sidedness. His problem solving is Darwinian - creative and simple and profound.
on the point about "psychology is too young to have a paradigm", my intuition is actually the opposite: that academic psychology is running into these issues because, as a field (and as a culture more broadly), we've come to a pretty sophisticated understanding of ourselves and our behaviors, and we're running up against the limit of what more there is to learn if we're limiting ourselves to just the methods used in traditional psychology (statistical analysis of survey responses, etc etc.).
the fact that we even have concepts like personality, emotion, the conscious/unconscious distinction, the concept of mental illness/health, etc etc. are indicative of a fairly evolved paradigm of what a human mind is like and how it works, more so than we had in the past. as Scott Alexander points out, "Statements like “my abuse gave me a lot of baggage that I’m still working through” involves a theory-of-mind that would have been incomprehensible a few centuries ago."
from the standpoint of further scientific discovery, I think the real fertile ground is in bridging the gap between the physical and the psychological, to get a deeper and more fine-grained understanding of how exactly the brain produces behavior & experience. to the extent that psychology limits itself to _only_ studying externally measurable actions, and tries to come up with generalizable causal laws that predict those actions, I think it will remain in a rut, because human behavior is fundamentally chaotic/complex, and there's only so much you can learn without taking a peek inside the brain itself.
So, to say complexity and chaos, systemic thinking, neuropsychology, brain science, networks, AI. All these fields are paradigms of "pure" and maybe "hard" science against the self-fulfilling prophecy of questionnaires. Keeping of course in mind that Psychology must study humans in a humanitarian way.
I recommend you read the book 'the world behind the world' which just came out
"But there's no world-changing insight like relativity, evolution, or DNA, nor any smaller-but-still-very-cool discoveries like polymerase chain reaction, CRISPR, or Higgs bosons."
This isn't true. There is an extremely important, novel (when it was discovered), and (of course) replicable insight that came out of psychology: IQ. That an abstract concept like intelligence can be readily quantified and measured, that it has predictive power across almost every major field of human endeavor, and that it is, as a first order approximation, a unitary construct, are all valuable and unintuitive ideas. Psychologists just don't like the results of IQ research.
IQ is over 100 years old, so it's not quite recent. It's also a good example of the limitations of abstractions. It's an undeniably useful concept, but it also fools us into thinking we've mapped the whole space when in fact we've only mapped a sliver. https://www.experimental-history.com/p/why-arent-smart-people-happier
https://www.theintrinsicperspective.com/p/your-iq-isnt-160-no-ones-is
interesting read. I wanted to ask in general whether you think, and to what degree do you think, these are accidental to psychology, or essential? My own view is that so much of these ongoing 'problems' follow from a deep epistemic mistake, i.e. assuming psychology is more like biology or chemistry, than it is to history or literature, and correlatively assuming psychology's methods consequently lead to objective facts of some sort, rather than interesting food for thought, questions, and interpretations that remain relative and subject to interesting alternative points of view. People read popular psychology or take psychology courses because they are interesting [or should be] not because they are discoveries. And thats how it should be. And, the significance remains as long as one is engaged critically and reflectively about whatever is being discussed, hopefully leading to insight or at least a deepening appreciation of the human condition. As we find when we read history, or literature, or philosophy. There might be progress and discoveries of sorts, but neever in the same way one expects from more objectively defined sciences. I think it is possible that this epistemic mistake in self understanding of psychology sets the stage to the professional game--- no one really takes the research that seriously. the emphasis is on professional standing and the external goods that go with the territory And, once one makes this the sole point, why not fake it or distort it?
TL;DR You are correct, and the reason is that Psychology gave up on theories. But nothing is going to change.
Wall of text, who has time to read, etc.
When I started studying Psychology 20 years ago, I immediately had the same questions you have right now. It was the same situation back then as it is now. If so many studies turn out not to be replicable, what are we even doing here?
It took me very long to get a grip on what is going on, and why. But others wrote it before me. Psychology is unable to produce real theories (e.g., Gigerenzer, 1998). There is no Theoretical Psychology. Try finding a book on relevant theories in Psychology. There are none. Only introductions to Psychology, listing a bunch of experimental paradigms and results.
What are real theories? They are specifications of robust relationships, mostly causal relationships. Mathematically, I believe they are probably structures (a type of mathematical objects), but that’s another topic.
All science is inductive in the sense that it cannot guarantee the consequences if a set of premises is true, that is only possible in formal systems. However, F = m*a does a darn good job in many situations. It relates important measurable quantities and can tell us what the result will be. If you calculate a force knowing mass and acceleration, the result is valid in the sense that you can have faith in it. You can base decisions on it and will be correct. Many, perhaps most results produced in Psychology are not valid in the sense that you should not trust them. You should be very careful if you base a decision on such results.
I believe that since about the mid 1990s, Psychology has given up on producing theories that encode and abstract robust relationships. That was not always the case. If you read Human Error by James Reason (1991), you find a very different future of Psychology. It was full of formalized robust relationships that could make quantitative predictions for important real life problems. This future has never materialized.
Why not? Several reasons.
First, to specify psychological insights, you need to know math and logic. Like really. Math is not a part of our curriculum. „Statistics“ is just learning how to click buttons in SPSS, it has nothing to do with mathematical statistics. Psychologists are, by and large, incapable to abstract from empirical insights from specific studies. Formalization is not a part of psychological culture. You can’t abstract with words, you need formalization.
Second, demographics. Most people who study psychology do it because they are interested in people, not in things. They want to become therapists, counselors, but not ponder formalization of empirical results. There is not critical mass demanding to learn how to build and use reliable theories (or even methods). However, there are a lot of people who want to get a degree and go on with their lives. You don’t need math as a school counselor.
Third, Artificial Intelligence. Much of what 1980s Psychologists where sure was going to happen did not because they bet the house on purely symbolic AI. Most of the research program of cognitive architectures was born out of this view. As we know, this bet was lost. It’s mostly subsymbolic and computational statistics now. Psychology did not even try to keep up, it simply gave up.
That is not to say we never did anything right. Psychology made crucial methodological contributions, just two examples. Structual equation modelling is a direct attempt to specify causal relationships based on empirical results. One of the smartest minds ever helped building it, Judea Perl. You might know him as one of the inventors of Bayesian Networks (yes really), and he teamed up with statisticians and psychologists to do it. Another invaluable contribution was to (generalized) linear mixed models. The interpretation of fixed vs. random factors is really tricky, and much empirical work in (educational!!) Psychology helped to clarify that distinction.
A bit of bad luck is that sometimes, we produce robust relationships (i.e, trustworthy results), and the world ignores it. For example, we 100% predicted the Tesla accidents by showing over and over again that drivers cannot react quickly enough in time to silent failures. We predicted that 15 years ago, I conducted such experiments back then.
And we are experts at ignoring our own best results. One of the very few real attempts at a theory is a brilliant theory on causality by Patricia Cheng (1997). The more you know about how little we know about causality, the more in awe you’ll be about this paper. Another example is „Frontiers of Test Validity Theory“ (Markus & Borsboom, 2013). It’s not the best edited book, but it layes out the very specific problems we have when trying to (really!) measure latent variables. In many respects, that is trying to translate Kant into actionable statistics. A research program addressing causality and a latent variable theory could become the gold standard for many other fields, far beyond Psychology.
Alas, that is never going to happen. The status quo is too convenient for too many people. I’ve seen it too often that we produced insights of interest, just to be discarded by those who should be interested. For example, changes in traffic safety are largely only achievable by adapting the infrastructure. That is not something politicians want to hear. They want cheap, magical tricks like „Write your name on every page.“ It really, really pays to produce such papers. Next, again, demographics. I get it, what does a therapist need Linear Algebra and Predicate Logic for. But this is our critical mass. Thus, we have none.
I could go on and on, but let’s be real, who cares? We get paid, we can do experiments all day long and create a gripping narrative afterwards, tell everyone how our research will change the world, etc.
If you really want to move something, leave the field. You can’t win against the system. Yes I’m incredibly frustrated, but that’s the reality. Psychology as a paid profession will endure because there is a market for the activity, but it will not change because there is no impetus for a Kuhnian structural revolution. You can have only if there is some sort of validation criterion, if it pays to predict correct results. Unfortunately, that is not the case in today’s Psychology, as you have so eloquently shown.
If mathematical formalization was the answer, economics wouldn't have the same failure rates as psychology by most measures. You can lie with math too, especially to yourself.
I'm not 100% sure what you mean, I take it to mean „doing lots of math is not a sufficient condition for good science“. I agree, but I probably do not agree with your premises.
First, formalization comes in many shapes and forms. I think of it as agreements about which things exist, and how they are related. Think of predicate logic, ontologies, or even musical notes. Quantification can help to clarify such relations, but in general it probably plays a small role in understanding the world. I'm not an expert in economics, but I think they use lots of numbers simply because economies have lots of things you can count. At the end of the day, they struggle with the same issue: What do these numbers really mean? What latent structure do they refer to?
Second, I do not agree that replicability on its own is a good measure for scientific success. Pretty much every empirical science struggles with the precursor to replicability, i.e., reproducibility. Psychology is doing worse than some, but the natural sciences or engineering disciplines are not necessarily doing stellar. Just search for something like „reproducibility medicine biochemistry“ on Google Scholar. Fun times, I promise. And many empirical results *cannot* be reproduced. Ever tried to build a Hadron collider in your backyard? If you cannot reproduce empirical results, how do you decide whether they are replicable?
This brings me to my third point. Popper said it best: Science is what we can omit. It is exactly what I referred to as „uncovering robust relations“. Galilei threw things from the Tower of Pisa to show that things of different weight can fall at the same speed. You can easily reproduce that result in your backyard. You do not need to know the time of year Galilei did it, or at what angle the Tower of Pisa stood when Galilei did it, or which things he exactly used. Science is taking things away until we arrive at a firm core which cannot be altered further. To me, this is what formalization is all about.
My point is: If all studies are done extremely well but only investigate garbage, science will not have moved one inch forward. Without good theories, you do not know what you should study in the first place, and if it goes wrong you do not know why. Einstein (1949) said „Thinking without the positing of categories and concepts in general would be as impossible as breathing in a vacuum.“ You need a metastructure that produces these categories, which is a theory. Better theories will lead to replicable results because you know what you can leave out to reproduce a study. Is the time of the year important, the object to throw down and its angle? What are the irreducible relationships which let you repeat what others have done before you?
By blindly searching for „effects“ without the faintest idea of why they should occur we have no idea what to leave out. We then do not know whether we could not reproduce an existing effect because we got the setup wrong, or if the effect cannot be replicated because it most probably does not exist. This is the real tragedy, not whether some crooks faked their data.
TL;DR: successful replication isn't proof of scientific progress, and we need better theories.
Agreed. But the absence of replication success is definitely evidence that something is deeply wrong, regardless of what we should/shouldn't be doing with theories.
So formalization in economics hasn't proved a better approach than the hypothesis testing in psychology. Because the problem isn't primarily methodological, it's cultural.
So what if we thought of psychologists as existing in the same “space” as fiction writers and we judged their studies (now called simply “stories”) by how much they moved us or helped us understand ourselves and others? How would Gino and Ariely ... indeed, how would anyone ... fare then? I enjoyed your essay.