In undergrad statistics, the prof began the first class by saying, "Within a narrow margin of error, on average everyone has one tit and one testicle". He then spent the semester demonstrating how we could do better.
Interestingly, some of psychology's most replicable findings were generated by experimenters who studied behavior at the level of the individual organism (Pavlov, Piaget, Skinner, Ebbinghaus), treating their data graphically, or with quantitative models far removed from statistical inference. Many of their findings are so reliable that they are used as class demonstrations in undergraduate labs.
“Personality changes following heart transplantation, which have been reported for decades, include accounts of recipients acquiring the personality characteristics of their donor. Four categories of personality changes are discussed in this article: (1) changes in preferences, (2) alterations in emotions/temperament, (3) modifications of identity, and (4) memories from the donor’s life.”
This was both insightful and hilarious goddammit! I think your point about how statistics as a tool in the combination of noisy signals can be a source of false hope is very important! I'm a first year PhD-student and my enthusiasm for statistics hasn't died yet, but I agree that one has to be aware of the risks. Statistics is one tool for understanding, not the only tool for understanding.
Still I wonder if not some swaths of medicine/biology are counterexamples to the inevitability of stagnation for "sometimes science". A lot of medicine/biology are dominated by effects that are noisy enough that you need statistics to detect them. Of course there are replication issues in medicine too, but I'm pretty confident it's also moving forward at a steady pace. I think maybe the difference is in how the gradual theory construction has happened, where even the noisiest parts of medicine is always able to partially lean on our more solid understanding of biology and chemistry to constrain it. Psychology on the other hand can't really lean on neuroscience, as neuroscience is in an equally messy place.
You may or may not be comforted by psychology not being the only field that is afflicted by indiscriminate number crunching. Wild life research has become a routine of gathering big data sets, tipping them into a general linear "modelling" bucket and stirring it with an R stick until the required result floats to the top.
It is possibly useful to point out that Down's correlates to maternal age. This could have been discovered without knowing the underlying mechanism of the syndrome. Even knowing mechanisms, we study to find out more about phenomena, because we don't always know what trips the mechanisms.
Great article. Have you read "The Cult of Statistical Significance" by Ziliak and McCloskey? It's not opposed to statistics as such, but to the modern obsession with p-values.
I have heard statistics described as "a quantification of ignorance", which I think is apt.
The challenge with "ignorance signals" in science (especially the academy) is that if everyone thinks they know something, and you think that, "well, we probably don't know that thing," then lots of people think you're stupid. If your advisor thinks you're stupid, you don't get a PhD. Best to follow the other academic lemmings and hope you're not running close to a cliff. https://www.youtube.com/watch?v=YNZ_K14iT-Q
Dude, we don't even like doing disprovable science in the "hard" sciences. It is much easier to create some unreproducible experiment on a custom fixture that measures a material to be 3% stronger than the last strongest material. You can make a career doing that.
I tried showing how everyone's measurements were wrong because they weren't measuring an important deformation - a disprovable assertion that could cut measurement error by over 100% in some cases. I was told my work wasn't "scientifically relevant." My advisor threatened to have my work retracted if I published if I published it on my own. This happened in engineering with highly quantitative research. It doesn't matter which field you're in, mathematics is more often used as a tool of obfuscation.
Reading this a day after an Advanced Econometrics test which ended the last straw of patience I had for this entire semester at grad school, which prompted me and my best friend to go on a random 1am walk to complain about the quantitative struggle of Social Scientists. I had a fantastic laugh, thank you.
It seems to me that the key to switching from sometimes to always is finding the right denominator. E.g., in your example of the invisible triangle illusion, the unspoken denominator is “people who aren’t blind or functionally blind.” If we didn’t know that people could be blind, we’d be stuck putting optical illusions in the very unsatisfactory “sometimes” category. Finding that denominator can be a reasonable use of stats, IMO.
Interestingly, trying to ground psychology in neuroscience ("thoughts, attitudes, memories, language, all that stuff—is created by the nervous system") seems likely to flounder, as neuroscience itself is floundering on the hard problem of consciousness (check out The World Behind the World, written by neuroscientist Erik Hoel). It might be more productive to engage with the stuff of psychology as some kind of fundamental of reality that has an uncertain relationship with matter.
They're not science. Sorry - not sorry. Science has the hallmarks of reproducibility and predictability.
2-"Well, here’s one: everything we consider human psychology—thoughts, attitudes, memories, language, all that stuff—is created by the nervous system."
Not so fast. There is disagreement whether all that stuff is created by the nervous system or just mediated by the nervous system (with creation elsewhere).
3-What you wished the Den of Hate guy had told you? Excellent! Good luck applying that to the social "sciences"!
In undergrad statistics, the prof began the first class by saying, "Within a narrow margin of error, on average everyone has one tit and one testicle". He then spent the semester demonstrating how we could do better.
“If your experiment needs statistics, you ought to have done a better experiment.” -Ernest Rutherford.
I’m not sure if he was joking at the time.
Dr. Down: I HATE YOU
Dr. Up: GO DIE
Literally laughing out loud.
It's funny 'cuz it's true.
This article was great! Thank you. I laughed out loud quite a few times. Who knew articles about statistics could be funny?!?
Interestingly, some of psychology's most replicable findings were generated by experimenters who studied behavior at the level of the individual organism (Pavlov, Piaget, Skinner, Ebbinghaus), treating their data graphically, or with quantitative models far removed from statistical inference. Many of their findings are so reliable that they are used as class demonstrations in undergraduate labs.
Regarding you heart transplant example, Aristotle might have been onto something (https://www.sciencedirect.com/science/article/abs/pii/S0306987719307145):
“Personality changes following heart transplantation, which have been reported for decades, include accounts of recipients acquiring the personality characteristics of their donor. Four categories of personality changes are discussed in this article: (1) changes in preferences, (2) alterations in emotions/temperament, (3) modifications of identity, and (4) memories from the donor’s life.”
This was both insightful and hilarious goddammit! I think your point about how statistics as a tool in the combination of noisy signals can be a source of false hope is very important! I'm a first year PhD-student and my enthusiasm for statistics hasn't died yet, but I agree that one has to be aware of the risks. Statistics is one tool for understanding, not the only tool for understanding.
Still I wonder if not some swaths of medicine/biology are counterexamples to the inevitability of stagnation for "sometimes science". A lot of medicine/biology are dominated by effects that are noisy enough that you need statistics to detect them. Of course there are replication issues in medicine too, but I'm pretty confident it's also moving forward at a steady pace. I think maybe the difference is in how the gradual theory construction has happened, where even the noisiest parts of medicine is always able to partially lean on our more solid understanding of biology and chemistry to constrain it. Psychology on the other hand can't really lean on neuroscience, as neuroscience is in an equally messy place.
You may or may not be comforted by psychology not being the only field that is afflicted by indiscriminate number crunching. Wild life research has become a routine of gathering big data sets, tipping them into a general linear "modelling" bucket and stirring it with an R stick until the required result floats to the top.
It is possibly useful to point out that Down's correlates to maternal age. This could have been discovered without knowing the underlying mechanism of the syndrome. Even knowing mechanisms, we study to find out more about phenomena, because we don't always know what trips the mechanisms.
Great article. Have you read "The Cult of Statistical Significance" by Ziliak and McCloskey? It's not opposed to statistics as such, but to the modern obsession with p-values.
I have heard statistics described as "a quantification of ignorance", which I think is apt.
The challenge with "ignorance signals" in science (especially the academy) is that if everyone thinks they know something, and you think that, "well, we probably don't know that thing," then lots of people think you're stupid. If your advisor thinks you're stupid, you don't get a PhD. Best to follow the other academic lemmings and hope you're not running close to a cliff. https://www.youtube.com/watch?v=YNZ_K14iT-Q
Dude, we don't even like doing disprovable science in the "hard" sciences. It is much easier to create some unreproducible experiment on a custom fixture that measures a material to be 3% stronger than the last strongest material. You can make a career doing that.
I tried showing how everyone's measurements were wrong because they weren't measuring an important deformation - a disprovable assertion that could cut measurement error by over 100% in some cases. I was told my work wasn't "scientifically relevant." My advisor threatened to have my work retracted if I published if I published it on my own. This happened in engineering with highly quantitative research. It doesn't matter which field you're in, mathematics is more often used as a tool of obfuscation.
Reading this a day after an Advanced Econometrics test which ended the last straw of patience I had for this entire semester at grad school, which prompted me and my best friend to go on a random 1am walk to complain about the quantitative struggle of Social Scientists. I had a fantastic laugh, thank you.
It seems to me that the key to switching from sometimes to always is finding the right denominator. E.g., in your example of the invisible triangle illusion, the unspoken denominator is “people who aren’t blind or functionally blind.” If we didn’t know that people could be blind, we’d be stuck putting optical illusions in the very unsatisfactory “sometimes” category. Finding that denominator can be a reasonable use of stats, IMO.
Interestingly, trying to ground psychology in neuroscience ("thoughts, attitudes, memories, language, all that stuff—is created by the nervous system") seems likely to flounder, as neuroscience itself is floundering on the hard problem of consciousness (check out The World Behind the World, written by neuroscientist Erik Hoel). It might be more productive to engage with the stuff of psychology as some kind of fundamental of reality that has an uncertain relationship with matter.
This reminds me of this paper which contrasts qualitative and quantitative research, may be of interest.
https://public.wsu.edu/~tnridout/mahoney_goertz20061.pdf
1-"But the social sciences aren't like that."
They're not science. Sorry - not sorry. Science has the hallmarks of reproducibility and predictability.
2-"Well, here’s one: everything we consider human psychology—thoughts, attitudes, memories, language, all that stuff—is created by the nervous system."
Not so fast. There is disagreement whether all that stuff is created by the nervous system or just mediated by the nervous system (with creation elsewhere).
3-What you wished the Den of Hate guy had told you? Excellent! Good luck applying that to the social "sciences"!
:-)