101 Comments
Aug 29, 2023·edited Aug 29, 2023Liked by Adam Mastroianni

The post section "How Much Does a Thought Weigh" makes me think of something from a paper by Tal Yarkoni, "The generalizability crisis" (https://doi.org/10.1017/S0140525X20001685). In his conclusion, he writes:

"Closer examination reveals that the inferential statistics reported in psychology articles typically have only a tenuous correspondence to the verbal claims they are intended to support. The overarching conclusion is that many fields of psychology currently operate under a kind of collective self-deception, using a thin sheen of quantitative rigor to mask inferences that remain, at their core, almost entirely qualitative."

Your excursus on the study of leadership is a great riff on Yarkoni's concern. Learning more about how the psychological sausage gets made has made me enormously skeptical of any claimed results from the discipline; the apparent fact that there is no core of understanding makes me wonder further just what the point of psychological science is.

Expand full comment
Aug 29, 2023Liked by Adam Mastroianni

Wild theory, that literally occurred to me 30 seconds ago, so be gentle...

Maybe there is only all this energy fuelling 'hogwash' science of this kind because creeping technocracy creates massive demand for understanding people, so that they can be 'managed'. That, plus marketing.

Expand full comment
Aug 30, 2023Liked by Adam Mastroianni

> As a young psychologist, this chills me to my bones

Bunches of bees don’t have bones!

Expand full comment
Aug 29, 2023Liked by Adam Mastroianni

This is your best essay yet. Or perhaps I should say instead that it emphatically confirms my biases. I blame tenure and promotion practices: It's unpleasant to critically evaluate a colleague's work. It's much easier to assume that the editorial boards of academic journals are up to the job and that the length of one's CV is a useful index.

Expand full comment

I'm sympathetic to the idea that science is at its best when it as it's oriented towards a concrete goal. Educational psychology seems to not be a steaming pile of garbage, probably because we all have a decent grasp of what success would look like -- more successful learning and teaching. I have no idea what the goals of social psychology are. I know there have been examples of trying to scale up nudges or whatever for policy uses, and those have mostly not worked out, but I'd think that "policy psych" or something would be a more productive field. In general, I wonder if the key moving forward is to hew closely to big real-world stuff that needs improvement.

Expand full comment

I enjoyed this. Reminds me of the concluding paragraph of Gigerenzer's great 1998 paper, "Surrogates for theories."

"Several years ago, I spent a day and a night in a library reading through issues of the Journal of Experimental Psychology from the 1920s and 1930s. This was professionally a most depressing experience. Not because these articles were methodologically mediocre. On the contrary, many of them make today’s research pale in comparison to their diversity of methods and statistics, their detailed reporting of single-case data rather than mere averages, and their careful selection of trained subjects. And many topics—such as the influence of the gender of the experimenter on the performance of the participants—were of interest then as now. What depressed me was that almost all of this work is forgotten; it does not seem to have left a trace in the collective memory of our profession. It struck me that most of it involved collecting data without substantive theory. Data without theory are like a baby without a parent: their life expectancy is low."

Expand full comment
Aug 30, 2023·edited Aug 30, 2023Liked by Adam Mastroianni

If your find this post interesting I recommend looking into critical psychology (https://en.wikipedia.org/wiki/Critical_psychology).

These people though long and hard about these problems in the 1970s, 1980s and 1990s and there's a few people who still work within this tradition.

I have found these two books to be good introductions to the topic:

- "Psychology, Subjectivity, and Society: An Introduction to German Critical Psychology" by Charles Tolman

- "Psychology from the Standpoint of the Subject: Selected Writings of Klaus Holzkamp" edited by Ernst Schraube and Ute Osterkamp.

Expand full comment
Aug 30, 2023Liked by Adam Mastroianni

Great post. Fwiw, I think the most pernicious paradigm lies at the intersection of the “cognitive biases/nudge” mania and “pick-a-noun”. I’ll call it “create a noun”...the perverse joy people get from naming a phenomenon (usually a cognitive bias) because it boosts their currency and citation index. After all, everybody would love to be Dunning...or maybe Kruger.

Expand full comment

Construal Level Theory is a new idea that I have found insightful, useful, and built upon theoretically.

Expand full comment

I would say that this article is a nail in psychology's coffin but then again, I also share Adam's sentiment that none of it really matters: Gino, Ariely, Stapel, whoever... the person doesn't matter because there are way too many people invested in the current paradigm of producing papers that don't advance our understanding of humans in any meaningful way. They are just designed to look that way, with phrases such as “Nevertheless, this is a promising finding which suggests...” and

“It is interesting to note that...”, and elaborate-looking statistics (without any real underlying theory), and a bunch of made up words that are rephrasings or slight adjustments of the words that we already use.

Worse yet, this gets perpetuated, as Adam wrote, into infinity because of vested interest -- just look at how Baumeister tries to protect his baby (https://lupinepublishers.com/psychology-behavioral-science-journal/pdf/SJPBS.MS.ID.000234.pdf), ego depletion, even after the field has collectively been forced to conclude that no, these two words are not useful fictions anymore, e.g. https://pubmed.ncbi.nlm.nih.gov/34520296/. -- and the sheer simplicity of modern psychological research (which is basically just generating a bunch of numbers on a spreadsheet through an online survey platform like mTurk, Qualtrics, whatever.... Something compels me to think that no groundbreaking discoveries will ever occur via a bunch of people in a half-zombie mode clicking at a bunch of seemingly "valid" and "reliable" statements. It's just not the platform, honestly.) And now, apparently, the paradigm gets perpetuated because of lawsuits as well, and criticism gets quashed underneath the lofty foot of the legal system.

As a fresh psychology graduate, this makes me feel incredibly aimless. Imagine studying for 5 (or sometimes even longer) years only to gradually learn that whatever you studied is a bunch of made up stuff that people sort of collectively agreed on, with limited (or at least very wonky) real-world impact. I feel like my two options are either to abandon the field and keep my sanity and honor (whatever that is in our age); or conform, and be a tiny cog in the machine that produces meaningless vapor.

Expand full comment

A great read as usual.

It seems psychology has been in its infancy since forever. Lots of researchers gathering observations; lots of therapists with varying degrees of success. But no coherent big picture to tie it all together. Where is psychology's Newton, Lavoisier, or Darwin to give it a robust, unifying paradigm -- one that both has explanatory/predictive power and is concrete enough to be falsifiable?

Expand full comment

"But there's no world-changing insight like relativity, evolution, or DNA, nor any smaller-but-still-very-cool discoveries like polymerase chain reaction, CRISPR, or Higgs bosons."

This isn't true. There is an extremely important, novel (when it was discovered), and (of course) replicable insight that came out of psychology: IQ. That an abstract concept like intelligence can be readily quantified and measured, that it has predictive power across almost every major field of human endeavor, and that it is, as a first order approximation, a unitary construct, are all valuable and unintuitive ideas. Psychologists just don't like the results of IQ research.

Expand full comment

on the point about "psychology is too young to have a paradigm", my intuition is actually the opposite: that academic psychology is running into these issues because, as a field (and as a culture more broadly), we've come to a pretty sophisticated understanding of ourselves and our behaviors, and we're running up against the limit of what more there is to learn if we're limiting ourselves to just the methods used in traditional psychology (statistical analysis of survey responses, etc etc.).

the fact that we even have concepts like personality, emotion, the conscious/unconscious distinction, the concept of mental illness/health, etc etc. are indicative of a fairly evolved paradigm of what a human mind is like and how it works, more so than we had in the past. as Scott Alexander points out, "Statements like “my abuse gave me a lot of baggage that I’m still working through” involves a theory-of-mind that would have been incomprehensible a few centuries ago."

from the standpoint of further scientific discovery, I think the real fertile ground is in bridging the gap between the physical and the psychological, to get a deeper and more fine-grained understanding of how exactly the brain produces behavior & experience. to the extent that psychology limits itself to _only_ studying externally measurable actions, and tries to come up with generalizable causal laws that predict those actions, I think it will remain in a rut, because human behavior is fundamentally chaotic/complex, and there's only so much you can learn without taking a peek inside the brain itself.

Expand full comment

So what if we thought of psychologists as existing in the same “space” as fiction writers and we judged their studies (now called simply “stories”) by how much they moved us or helped us understand ourselves and others? How would Gino and Ariely ... indeed, how would anyone ... fare then? I enjoyed your essay.

Expand full comment

interesting read. I wanted to ask in general whether you think, and to what degree do you think, these are accidental to psychology, or essential? My own view is that so much of these ongoing 'problems' follow from a deep epistemic mistake, i.e. assuming psychology is more like biology or chemistry, than it is to history or literature, and correlatively assuming psychology's methods consequently lead to objective facts of some sort, rather than interesting food for thought, questions, and interpretations that remain relative and subject to interesting alternative points of view. People read popular psychology or take psychology courses because they are interesting [or should be] not because they are discoveries. And thats how it should be. And, the significance remains as long as one is engaged critically and reflectively about whatever is being discussed, hopefully leading to insight or at least a deepening appreciation of the human condition. As we find when we read history, or literature, or philosophy. There might be progress and discoveries of sorts, but neever in the same way one expects from more objectively defined sciences. I think it is possible that this epistemic mistake in self understanding of psychology sets the stage to the professional game--- no one really takes the research that seriously. the emphasis is on professional standing and the external goods that go with the territory And, once one makes this the sole point, why not fake it or distort it?

Expand full comment

TL;DR You are correct, and the reason is that Psychology gave up on theories. But nothing is going to change.

Wall of text, who has time to read, etc.

When I started studying Psychology 20 years ago, I immediately had the same questions you have right now. It was the same situation back then as it is now. If so many studies turn out not to be replicable, what are we even doing here?

It took me very long to get a grip on what is going on, and why. But others wrote it before me. Psychology is unable to produce real theories (e.g., Gigerenzer, 1998). There is no Theoretical Psychology. Try finding a book on relevant theories in Psychology. There are none. Only introductions to Psychology, listing a bunch of experimental paradigms and results.

What are real theories? They are specifications of robust relationships, mostly causal relationships. Mathematically, I believe they are probably structures (a type of mathematical objects), but that’s another topic.

All science is inductive in the sense that it cannot guarantee the consequences if a set of premises is true, that is only possible in formal systems. However, F = m*a does a darn good job in many situations. It relates important measurable quantities and can tell us what the result will be. If you calculate a force knowing mass and acceleration, the result is valid in the sense that you can have faith in it. You can base decisions on it and will be correct. Many, perhaps most results produced in Psychology are not valid in the sense that you should not trust them. You should be very careful if you base a decision on such results.

I believe that since about the mid 1990s, Psychology has given up on producing theories that encode and abstract robust relationships. That was not always the case. If you read Human Error by James Reason (1991), you find a very different future of Psychology. It was full of formalized robust relationships that could make quantitative predictions for important real life problems. This future has never materialized.

Why not? Several reasons.

First, to specify psychological insights, you need to know math and logic. Like really. Math is not a part of our curriculum. „Statistics“ is just learning how to click buttons in SPSS, it has nothing to do with mathematical statistics. Psychologists are, by and large, incapable to abstract from empirical insights from specific studies. Formalization is not a part of psychological culture. You can’t abstract with words, you need formalization.

Second, demographics. Most people who study psychology do it because they are interested in people, not in things. They want to become therapists, counselors, but not ponder formalization of empirical results. There is not critical mass demanding to learn how to build and use reliable theories (or even methods). However, there are a lot of people who want to get a degree and go on with their lives. You don’t need math as a school counselor.

Third, Artificial Intelligence. Much of what 1980s Psychologists where sure was going to happen did not because they bet the house on purely symbolic AI. Most of the research program of cognitive architectures was born out of this view. As we know, this bet was lost. It’s mostly subsymbolic and computational statistics now. Psychology did not even try to keep up, it simply gave up.

That is not to say we never did anything right. Psychology made crucial methodological contributions, just two examples. Structual equation modelling is a direct attempt to specify causal relationships based on empirical results. One of the smartest minds ever helped building it, Judea Perl. You might know him as one of the inventors of Bayesian Networks (yes really), and he teamed up with statisticians and psychologists to do it. Another invaluable contribution was to (generalized) linear mixed models. The interpretation of fixed vs. random factors is really tricky, and much empirical work in (educational!!) Psychology helped to clarify that distinction.

A bit of bad luck is that sometimes, we produce robust relationships (i.e, trustworthy results), and the world ignores it. For example, we 100% predicted the Tesla accidents by showing over and over again that drivers cannot react quickly enough in time to silent failures. We predicted that 15 years ago, I conducted such experiments back then.

And we are experts at ignoring our own best results. One of the very few real attempts at a theory is a brilliant theory on causality by Patricia Cheng (1997). The more you know about how little we know about causality, the more in awe you’ll be about this paper. Another example is „Frontiers of Test Validity Theory“ (Markus & Borsboom, 2013). It’s not the best edited book, but it layes out the very specific problems we have when trying to (really!) measure latent variables. In many respects, that is trying to translate Kant into actionable statistics. A research program addressing causality and a latent variable theory could become the gold standard for many other fields, far beyond Psychology.

Alas, that is never going to happen. The status quo is too convenient for too many people. I’ve seen it too often that we produced insights of interest, just to be discarded by those who should be interested. For example, changes in traffic safety are largely only achievable by adapting the infrastructure. That is not something politicians want to hear. They want cheap, magical tricks like „Write your name on every page.“ It really, really pays to produce such papers. Next, again, demographics. I get it, what does a therapist need Linear Algebra and Predicate Logic for. But this is our critical mass. Thus, we have none.

I could go on and on, but let’s be real, who cares? We get paid, we can do experiments all day long and create a gripping narrative afterwards, tell everyone how our research will change the world, etc.

If you really want to move something, leave the field. You can’t win against the system. Yes I’m incredibly frustrated, but that’s the reality. Psychology as a paid profession will endure because there is a market for the activity, but it will not change because there is no impetus for a Kuhnian structural revolution. You can have only if there is some sort of validation criterion, if it pays to predict correct results. Unfortunately, that is not the case in today’s Psychology, as you have so eloquently shown.

Expand full comment