Feb 10, 2023·edited Feb 10, 2023Liked by Adam Mastroianni
One thing that seems relevant is that a certain segment of the population believes that history has a 'preferred direction' or purpose ... there is a natural law of history, that means that we are all progressing, as we are supposed to, to the utopian future. If this teleological lens is how you see history then people in the past by definition must have less enlightened attitudes that we do in the present. The reverse of this, popular other places, is that rather than 'progressing' we have embarked on a path of terminal decline. People who believe this believe that those in the past must have had more enlightened attitudes than we do now. Sometimes the people on both sides believe exactly the same things about attitudes in the past -- they only disagree over what constitutes progress vs decline.
The teleological view is rather dangerous, from a governing point of view. It means that people are not free to propose a way to deal with a problem, try it, discover that it didn't work as imagined, conclude it was a bad idea and decide to not do this any more. Once you get started with a project, there is relentless pressure to double down on the measure when it first shows signs of not working out after all. Progress appears to not have a reverse gear. But can explicitly 'progress-neutral' or 'progress-irrelevant' measures grow sufficient popular backing?
This critique is based purely on my squinting at the graphs not on any number crunching, but I was a less impressed by the under/over estimate charts than the strength of the conclusions here seem to imply. Many of them I look at and think that people got it basically right, or that the delta was pretty close but the absolute was off.
I’m saying that the standard for being right/unbiased is too exacting, and therefore saying “98% (!) of the questions” is a bit misleading. I would be very surprised if they all matched as closely as the ones that did match, and suspect those matches are more chance.
This makes me think the stronger claims of bias are exaggerated -- maybe we don’t need this much of an explanation! But the parts where we dig in to the ones we’re people are very very wrong are quite interesting.
Looking at the graphs too, I second that squint. Some of the over-estimations (feeling towards Asian people) look as accurate as the exact estimates. Maybe this is the time to get nit-picky about how Adam divided them to underesetimate, overestimate, and mistaken direction?
Nice research and neat article but I think the elephant in the room is that people's feelings change as reality changes. People are more against gun control because we live in a more violent world with less dependable police support. We want guns to protect ourselves because the government is less reliable to do so.
But your chart on how people view the position of society as a whole is mostly about what media and educational institutions are telling them, which by your demonstration is largely wrong. You demonstrate sources of bias by talking about civics classes and news reports which shows you agree with me.
Well, yes, humans tend not to be omniscient. We tend to be “right” about those aspects that were selected for by evolution. That is, those aspects that helped us survive in a small band of family on the savannah in Africa.
Thus why AI is/will be better than humans at a ton of purely cognitive tasks (such as playing chess) but worse than humans at tasks that require idiosyncratic interactions with a physical world (such as driving a car under all weather conditions on all streets without getting in to accidents or riding a bike). In an AI world, the “safest” traditional jobs (jobs we currently have) will be jobs that require idiosyncratic interactions with the physical world (like being a plumber). In this world, being a janitor may pay as well as being a (basic level) coder.
I think a lot of this is just social desirability bias. Take the “Would vote for black president” chart. ~80% of people said they would in 1978, but they harbored racist attitudes that made them perceive candidates of color more negatively. It took another 30 years for a black American to be nominated by a major political party. The polling for a lot of these questions didn’t capture real attitudes.
-The important thing here is that people didn't know that. Participants figured it wasn't socially desirable to say that you'd vote for a Black person for president in 1978, as if it was fine back then to say you wouldn't vote for someone based on the color of their skin.
-People were willing to say in 1990 that they wouldn't want a family member to marrying someone who is Black, so people may be more willing to admit seemingly socially undesirable things than we think
Or maybe it has more to do with the candidate than his color? As a good example, the people who would vote for Herman Cain or Colin Powell both fairly successful black candidates, would never consider voting for Obama, yet may or may not have a more positive or negative view of black people in general.
This is the point I came here to make. In many of these cases, perhaps, the "what I think others think" number is closer to the truth.
This is even a methodology that some political pollsters have begun to use, in order to combat "social desirability bias". For example: don't ask people if they're planning to vote for Trump; ask them what percentage of their friends/family they think will be voting for Trump.
Am I misreading the chart that says “Extramarital sex is not wrong”? I read it as saying 95% think it *is* wrong, and if anything this is getting higher.
That is surprising enough to me that I thought I would ask if I am making a mistake, or if there is a mistake on the chart (or, I guess, if the one thing America completely and totally agrees on is the importance of sex in wedlock)
Yep, seems that way. Remember that in the ‘70’s, some folks were in to wife swapping and key parties. There seemed to be more swingers too. In some respects, (heterosexual) relationships have gotten more prudish in the US.
You’re right to be worried about data quality on MTurk. I almost included this bit but cut it at the end:
People often have concerns about running psychology studies on the internet, and for good reason: you’ll get junk responses if you’re not careful. I do eight things to prevent that.
- I use a platform called CloudResearch that prescreens participants with basic quality control checks.
- I only let people take my studies if they pass a three-item test of English language and American culture (when I’m studying Americans). For instance: “Which if this is NOT associated with Halloween?” and you have to pick “eating turkey.”
- I ask people for their birth year at the beginning of the study and I ask them for their age at the end.
- I include various manipulation checks and attention checks in the survey. For instance, I sometimes ask people to recall the last answer that they gave.
- I always embed an attention check at the end of the survey.
- My studies are always either a) very short, or b) at least mildly interesting.
- I pay 50-100% higher than average.
- I keep a list of past offenders that I’ve kicked out of surveys for one reason or another, and I don’t let them sign up again.
The people who fail the checks tend to respond randomly and fill out open-ended text boxes with gobbledygook. The people who pass the checks generally give answers that make sense. There are always a few trolls I don’t catch, but otherwise I feel pretty good about the quality of data that I get. If you ever see someone presenting data they collected online without tons of quality assurance, you should assume that their data includes a bunch of junk responses.
Sweet! Those are just the sort of quality controls I expected to see associated with use of MTurk by someone working to put research on more solid footing. I understand the reluctance to include it in the readable, engaging version of the paper, but I think it's critical information nonetheless. What would you think about posting it as a standalone topic and linking to it when you use MTurk.
Gotcha. In fact, that's a really valuable concept to really ground yourself in. I was focusing on the potentially very narrow demographic from MTurk, but Adam has allayed my concerns with his methodology. If I understand your question, you are asking why the results, really any results of any research based on sampled data, are limited to the range (in this case demographic) of the sample. It's really critical and often poorly understood. You started out by asking for a proof, so I'm going to assume you are happy wrangling math stat proofs. Experimental Design is the discipline in question, that's where you'll find the topic covered in most depth. It's very cool stuff.
When a topic becomes hotly debated it leads to polarization and the resulting shift is hard for other people to predict. For example Trump's election on an immigration reduction platform made democrat voters shift in favor of immigration which is not something intuitive. That sort of makes sense if you account for spite, but the event that led to the radicalization of white liberals on race was the election of barack Obama. That's the opposite of what I expected.
Plus, people are bad at both history and numerical assessments. There's plenty of studies that show public opinion is widely wrong on the % of blacks or gays in the general population.
Thank you your online survey tips. I know it is a lot to ask but maybe, can you write a guide on how to conduct research, like to ensure quality when doing survey online, or how to ensure replication? Just from your personal experiences?
I wonder if some of the biases on "build the wall" presuming to mean being against immigration is b/c the media and many politicians conflate border/ national security with The Immigration System.
Thanks for this. Can I offer one build on the tendency to imagine linear progression from good to bad, conservative to liberal? A lot of the misperception you measure may be a result of our belief that recent social change is about a general, one variable shift. In reality, liberal society allows you to be as conservative or modern as you want about any of dozens of lifestyle variables. The scope of lifestyle variation is so large now, but you absolutely can act like it’s 1895, if you can find a little group of folks (polygamist Mormons) who agree with you. This increased scope would absolutely fragment us in ways that make our estimations of national patterns meaningless.
As someone who loves watching American society from the outside, it seems obvious that your attitudes are becoming progressively more conservative and parochial. Hollywood is an example of a cultural 'institution' that seems to have shifted towards conservatism and prudishness. Interesting that your citizens don't see that.
Also (What The?!) the fact that Americans are still using the word "believe" with climate change, as if it is some faith-based phenomenon?! That's so last decade!
Isn't a belief something we accept without having a true understanding of? I bet if you asked most people about climate change and other issues, they could sprout information they've read or heard, but do they really know the facts? Believe is exactly the right word to use in this case.
Or you could consider the quality of the dataset wattsupwiththat.com and see for yourself that the noise is orders of magnitude larger than the alleged signal
It is an indication that whatever is happening in the atmosphere Enhanced Greenhouse Effect due to increased levels of CO2 is not the primary driver. The theory and the models based on it do not fit the facts. The models keep changing but what stays the same is that they NEVER have any predictive power, a good sign that they are useless, mere curve fits designed to show what they authors want them to show.
The Enhanced Greenhouse Theory predicts:
1. a hot spot at the tropopause- never observed.
2. decrease in polar ice- partially true Arctic has a net decrease since 1978 Antarctic slightly small net increase(source nasa.gov)
3. increased 'extreme weather'- ill defined can't be evaluated
4. increase in 'Earth's surface temperature'- utterly undefined, how high up in the atmosphere should these measurements go? is this seasonally adjusted? mean? min/max? nobody knows. do we try and measure thermal energy in oceans or solid parts of the crust, many orders of magnitude larger than thermal energy in atmosphere? nobody knows.
But the most significant point is that heating is only shown on 'surface stations', ie thermometers on the ground. The Satellite MSU record shows no warming. Why should we prefer the MSU record?
Problems with Surface Stations:
1. Very poor geographic distribution- Only the continental US has a large number of purpose built temperature tracking stations. The rest of the world has VERY spotty coverage with no long term continuity of dataset. Different stations or stations that have moved sites are often combined into a single record. Also, where measuring points are far apart a 'smoothing' algorithm is used where a single point thermometer may be giving data on weather conditions 1200 km away, like using the thermometer on my back porch in Georgia to give weather data in New York, variation can be MUCH larger than the 1-2 degrees C alleged warming where noise swamps signal.
2. Even in US Climate Network there is reason to select siting bias. Check out the surfacestations.org survey. Many formerly 'well-sited' measuring points have experienced construction nearby which ALWAYS leads to an increase. Asphalt or concrete versus being in a field always leads to an increased temperature. But look at the details for the 'gold standard' stations, they have often had building built next to them since the records begin which violates all of the rules for site placement because it is known to introduce bias. More generally the urbanization of the country leads to an 'Urban Heat Island' effect, cities are known to be warmer than surrounding less developed areas.
So, thanks for the Fact Check. But, the highest quality satellite based data doesn't fit the theory. The models have no predictive power. And the dataset that does show warming has observable bias in a warming direction. More generally 'Science' isn't something to be believed. The Scientific Method has the power to falsify theories, not to prove anything true or likely or accurate. Because we cannot conduct global climate experiments there isn't anything ACTUAL science can tell us about global climate. We can only say, 'x has been correlated with y in the past' and we can quantify the correlation. The correlation between this theory and the data is poor. But I mean if Reuters believes it who am I to question...
One thing that seems relevant is that a certain segment of the population believes that history has a 'preferred direction' or purpose ... there is a natural law of history, that means that we are all progressing, as we are supposed to, to the utopian future. If this teleological lens is how you see history then people in the past by definition must have less enlightened attitudes that we do in the present. The reverse of this, popular other places, is that rather than 'progressing' we have embarked on a path of terminal decline. People who believe this believe that those in the past must have had more enlightened attitudes than we do now. Sometimes the people on both sides believe exactly the same things about attitudes in the past -- they only disagree over what constitutes progress vs decline.
The teleological view is rather dangerous, from a governing point of view. It means that people are not free to propose a way to deal with a problem, try it, discover that it didn't work as imagined, conclude it was a bad idea and decide to not do this any more. Once you get started with a project, there is relentless pressure to double down on the measure when it first shows signs of not working out after all. Progress appears to not have a reverse gear. But can explicitly 'progress-neutral' or 'progress-irrelevant' measures grow sufficient popular backing?
This critique is based purely on my squinting at the graphs not on any number crunching, but I was a less impressed by the under/over estimate charts than the strength of the conclusions here seem to imply. Many of them I look at and think that people got it basically right, or that the delta was pretty close but the absolute was off.
I’m saying that the standard for being right/unbiased is too exacting, and therefore saying “98% (!) of the questions” is a bit misleading. I would be very surprised if they all matched as closely as the ones that did match, and suspect those matches are more chance.
This makes me think the stronger claims of bias are exaggerated -- maybe we don’t need this much of an explanation! But the parts where we dig in to the ones we’re people are very very wrong are quite interesting.
Looking at the graphs too, I second that squint. Some of the over-estimations (feeling towards Asian people) look as accurate as the exact estimates. Maybe this is the time to get nit-picky about how Adam divided them to underesetimate, overestimate, and mistaken direction?
Sounds like it's probable that people's biases largely come from works of fiction.
Nice research and neat article but I think the elephant in the room is that people's feelings change as reality changes. People are more against gun control because we live in a more violent world with less dependable police support. We want guns to protect ourselves because the government is less reliable to do so.
But your chart on how people view the position of society as a whole is mostly about what media and educational institutions are telling them, which by your demonstration is largely wrong. You demonstrate sources of bias by talking about civics classes and news reports which shows you agree with me.
The more I read about psychology the more it seems we humans are wrong a lot more than we like to think. Or am I wrong about that too?! 😱
Well, yes, humans tend not to be omniscient. We tend to be “right” about those aspects that were selected for by evolution. That is, those aspects that helped us survive in a small band of family on the savannah in Africa.
Thus why AI is/will be better than humans at a ton of purely cognitive tasks (such as playing chess) but worse than humans at tasks that require idiosyncratic interactions with a physical world (such as driving a car under all weather conditions on all streets without getting in to accidents or riding a bike). In an AI world, the “safest” traditional jobs (jobs we currently have) will be jobs that require idiosyncratic interactions with the physical world (like being a plumber). In this world, being a janitor may pay as well as being a (basic level) coder.
I think a lot of this is just social desirability bias. Take the “Would vote for black president” chart. ~80% of people said they would in 1978, but they harbored racist attitudes that made them perceive candidates of color more negatively. It took another 30 years for a black American to be nominated by a major political party. The polling for a lot of these questions didn’t capture real attitudes.
That could be! But two things:
-The important thing here is that people didn't know that. Participants figured it wasn't socially desirable to say that you'd vote for a Black person for president in 1978, as if it was fine back then to say you wouldn't vote for someone based on the color of their skin.
-People were willing to say in 1990 that they wouldn't want a family member to marrying someone who is Black, so people may be more willing to admit seemingly socially undesirable things than we think
Or maybe it has more to do with the candidate than his color? As a good example, the people who would vote for Herman Cain or Colin Powell both fairly successful black candidates, would never consider voting for Obama, yet may or may not have a more positive or negative view of black people in general.
This is the point I came here to make. In many of these cases, perhaps, the "what I think others think" number is closer to the truth.
This is even a methodology that some political pollsters have begun to use, in order to combat "social desirability bias". For example: don't ask people if they're planning to vote for Trump; ask them what percentage of their friends/family they think will be voting for Trump.
Am I misreading the chart that says “Extramarital sex is not wrong”? I read it as saying 95% think it *is* wrong, and if anything this is getting higher.
That is surprising enough to me that I thought I would ask if I am making a mistake, or if there is a mistake on the chart (or, I guess, if the one thing America completely and totally agrees on is the importance of sex in wedlock)
Yep, seems that way. Remember that in the ‘70’s, some folks were in to wife swapping and key parties. There seemed to be more swingers too. In some respects, (heterosexual) relationships have gotten more prudish in the US.
Sure but do you think that 95%+ agree extramarital sex is wrong, then and now?
Basing the study on topics sourced from MTurk really threw me, given the focus of the research. MTurk has a substantial population of bad actors and a very, very limited demographic. At the very least, I think it's critical to make it crystal clear that any results are limited to that demographic. For more on issues with MTurk, see https://www.cambridge.org/core/journals/political-science-research-and-methods/article/abs/microtask-market-for-lemons-data-quality-on-amazons-mechanical-turk/B379D8827575D81857C872BB5C40B660, which is unfortunately pay walled, or https://link.springer.com/article/10.3758/s13428-019-01273-7 which is not, although perhaps a bit biased itself.
You’re right to be worried about data quality on MTurk. I almost included this bit but cut it at the end:
People often have concerns about running psychology studies on the internet, and for good reason: you’ll get junk responses if you’re not careful. I do eight things to prevent that.
- I use a platform called CloudResearch that prescreens participants with basic quality control checks.
- I only let people take my studies if they pass a three-item test of English language and American culture (when I’m studying Americans). For instance: “Which if this is NOT associated with Halloween?” and you have to pick “eating turkey.”
- I ask people for their birth year at the beginning of the study and I ask them for their age at the end.
- I include various manipulation checks and attention checks in the survey. For instance, I sometimes ask people to recall the last answer that they gave.
- I always embed an attention check at the end of the survey.
- My studies are always either a) very short, or b) at least mildly interesting.
- I pay 50-100% higher than average.
- I keep a list of past offenders that I’ve kicked out of surveys for one reason or another, and I don’t let them sign up again.
The people who fail the checks tend to respond randomly and fill out open-ended text boxes with gobbledygook. The people who pass the checks generally give answers that make sense. There are always a few trolls I don’t catch, but otherwise I feel pretty good about the quality of data that I get. If you ever see someone presenting data they collected online without tons of quality assurance, you should assume that their data includes a bunch of junk responses.
Sweet! Those are just the sort of quality controls I expected to see associated with use of MTurk by someone working to put research on more solid footing. I understand the reluctance to include it in the readable, engaging version of the paper, but I think it's critical information nonetheless. What would you think about posting it as a standalone topic and linking to it when you use MTurk.
That's a good point. I just added my comment as a footnote to this post, which gives it a URL I can link to in subsequent posts.
> I think it's critical to make it crystal clear that any results are limited to that demographic
Have you an accompanying proof?
The second reference I gave goes into the question of the demographics, their importance, impact, etc, in depth.
Right, but I'm curious about the "is limited to that demographic" part, is that demonstrated in your links?
Gotcha. In fact, that's a really valuable concept to really ground yourself in. I was focusing on the potentially very narrow demographic from MTurk, but Adam has allayed my concerns with his methodology. If I understand your question, you are asking why the results, really any results of any research based on sampled data, are limited to the range (in this case demographic) of the sample. It's really critical and often poorly understood. You started out by asking for a proof, so I'm going to assume you are happy wrangling math stat proofs. Experimental Design is the discipline in question, that's where you'll find the topic covered in most depth. It's very cool stuff.
When a topic becomes hotly debated it leads to polarization and the resulting shift is hard for other people to predict. For example Trump's election on an immigration reduction platform made democrat voters shift in favor of immigration which is not something intuitive. That sort of makes sense if you account for spite, but the event that led to the radicalization of white liberals on race was the election of barack Obama. That's the opposite of what I expected.
Plus, people are bad at both history and numerical assessments. There's plenty of studies that show public opinion is widely wrong on the % of blacks or gays in the general population.
Thank you your online survey tips. I know it is a lot to ask but maybe, can you write a guide on how to conduct research, like to ensure quality when doing survey online, or how to ensure replication? Just from your personal experiences?
I wonder if some of the biases on "build the wall" presuming to mean being against immigration is b/c the media and many politicians conflate border/ national security with The Immigration System.
What was the reason behind using actual shifts in exps 3a & 3b, but opposite of estimated shift in 3c?
How is Amazon Mechanical Turk a representative sample? Do all Americans work for AMT?
Thanks for this. Can I offer one build on the tendency to imagine linear progression from good to bad, conservative to liberal? A lot of the misperception you measure may be a result of our belief that recent social change is about a general, one variable shift. In reality, liberal society allows you to be as conservative or modern as you want about any of dozens of lifestyle variables. The scope of lifestyle variation is so large now, but you absolutely can act like it’s 1895, if you can find a little group of folks (polygamist Mormons) who agree with you. This increased scope would absolutely fragment us in ways that make our estimations of national patterns meaningless.
Great job.
As someone who loves watching American society from the outside, it seems obvious that your attitudes are becoming progressively more conservative and parochial. Hollywood is an example of a cultural 'institution' that seems to have shifted towards conservatism and prudishness. Interesting that your citizens don't see that.
Also (What The?!) the fact that Americans are still using the word "believe" with climate change, as if it is some faith-based phenomenon?! That's so last decade!
As you can see in your reply, there are clearly Americans who don’t believe in climate change.
Isn't a belief something we accept without having a true understanding of? I bet if you asked most people about climate change and other issues, they could sprout information they've read or heard, but do they really know the facts? Believe is exactly the right word to use in this case.
NOAA says no warming in 8 years so maybe you are right that believing global warming is last decade? https://thepowerhour.com/its-official-no-global-warming-for-eight-years/
Or you could consider the quality of the dataset wattsupwiththat.com and see for yourself that the noise is orders of magnitude larger than the alleged signal
No warming for 8 years? Some might find this comforting. But it is not an indication that temperature trends have reversed. Even NOAA themselves say the overall trend is up. :( https://www.reuters.com/article/factcheck-noaa-global-cooling-idUSL1N34J26C
It is an indication that whatever is happening in the atmosphere Enhanced Greenhouse Effect due to increased levels of CO2 is not the primary driver. The theory and the models based on it do not fit the facts. The models keep changing but what stays the same is that they NEVER have any predictive power, a good sign that they are useless, mere curve fits designed to show what they authors want them to show.
The Enhanced Greenhouse Theory predicts:
1. a hot spot at the tropopause- never observed.
2. decrease in polar ice- partially true Arctic has a net decrease since 1978 Antarctic slightly small net increase(source nasa.gov)
3. increased 'extreme weather'- ill defined can't be evaluated
4. increase in 'Earth's surface temperature'- utterly undefined, how high up in the atmosphere should these measurements go? is this seasonally adjusted? mean? min/max? nobody knows. do we try and measure thermal energy in oceans or solid parts of the crust, many orders of magnitude larger than thermal energy in atmosphere? nobody knows.
But the most significant point is that heating is only shown on 'surface stations', ie thermometers on the ground. The Satellite MSU record shows no warming. Why should we prefer the MSU record?
Problems with Surface Stations:
1. Very poor geographic distribution- Only the continental US has a large number of purpose built temperature tracking stations. The rest of the world has VERY spotty coverage with no long term continuity of dataset. Different stations or stations that have moved sites are often combined into a single record. Also, where measuring points are far apart a 'smoothing' algorithm is used where a single point thermometer may be giving data on weather conditions 1200 km away, like using the thermometer on my back porch in Georgia to give weather data in New York, variation can be MUCH larger than the 1-2 degrees C alleged warming where noise swamps signal.
2. Even in US Climate Network there is reason to select siting bias. Check out the surfacestations.org survey. Many formerly 'well-sited' measuring points have experienced construction nearby which ALWAYS leads to an increase. Asphalt or concrete versus being in a field always leads to an increased temperature. But look at the details for the 'gold standard' stations, they have often had building built next to them since the records begin which violates all of the rules for site placement because it is known to introduce bias. More generally the urbanization of the country leads to an 'Urban Heat Island' effect, cities are known to be warmer than surrounding less developed areas.
So, thanks for the Fact Check. But, the highest quality satellite based data doesn't fit the theory. The models have no predictive power. And the dataset that does show warming has observable bias in a warming direction. More generally 'Science' isn't something to be believed. The Scientific Method has the power to falsify theories, not to prove anything true or likely or accurate. Because we cannot conduct global climate experiments there isn't anything ACTUAL science can tell us about global climate. We can only say, 'x has been correlated with y in the past' and we can quantify the correlation. The correlation between this theory and the data is poor. But I mean if Reuters believes it who am I to question...