One thing that seems relevant is that a certain segment of the population believes that history has a 'preferred direction' or purpose ... there is a natural law of history, that means that we are all progressing, as we are supposed to, to the utopian future. If this teleological lens is how you see history then people in the past by definition must have less enlightened attitudes that we do in the present. The reverse of this, popular other places, is that rather than 'progressing' we have embarked on a path of terminal decline. People who believe this believe that those in the past must have had more enlightened attitudes than we do now. Sometimes the people on both sides believe exactly the same things about attitudes in the past -- they only disagree over what constitutes progress vs decline.
The teleological view is rather dangerous, from a governing point of view. It means that people are not free to propose a way to deal with a problem, try it, discover that it didn't work as imagined, conclude it was a bad idea and decide to not do this any more. Once you get started with a project, there is relentless pressure to double down on the measure when it first shows signs of not working out after all. Progress appears to not have a reverse gear. But can explicitly 'progress-neutral' or 'progress-irrelevant' measures grow sufficient popular backing?
This critique is based purely on my squinting at the graphs not on any number crunching, but I was a less impressed by the under/over estimate charts than the strength of the conclusions here seem to imply. Many of them I look at and think that people got it basically right, or that the delta was pretty close but the absolute was off.
I’m saying that the standard for being right/unbiased is too exacting, and therefore saying “98% (!) of the questions” is a bit misleading. I would be very surprised if they all matched as closely as the ones that did match, and suspect those matches are more chance.
This makes me think the stronger claims of bias are exaggerated -- maybe we don’t need this much of an explanation! But the parts where we dig in to the ones we’re people are very very wrong are quite interesting.
Looking at the graphs too, I second that squint. Some of the over-estimations (feeling towards Asian people) look as accurate as the exact estimates. Maybe this is the time to get nit-picky about how Adam divided them to underesetimate, overestimate, and mistaken direction?
Nice research and neat article but I think the elephant in the room is that people's feelings change as reality changes. People are more against gun control because we live in a more violent world with less dependable police support. We want guns to protect ourselves because the government is less reliable to do so.
But your chart on how people view the position of society as a whole is mostly about what media and educational institutions are telling them, which by your demonstration is largely wrong. You demonstrate sources of bias by talking about civics classes and news reports which shows you agree with me.
Well, yes, humans tend not to be omniscient. We tend to be “right” about those aspects that were selected for by evolution. That is, those aspects that helped us survive in a small band of family on the savannah in Africa.
Thus why AI is/will be better than humans at a ton of purely cognitive tasks (such as playing chess) but worse than humans at tasks that require idiosyncratic interactions with a physical world (such as driving a car under all weather conditions on all streets without getting in to accidents or riding a bike). In an AI world, the “safest” traditional jobs (jobs we currently have) will be jobs that require idiosyncratic interactions with the physical world (like being a plumber). In this world, being a janitor may pay as well as being a (basic level) coder.
I think a lot of this is just social desirability bias. Take the “Would vote for black president” chart. ~80% of people said they would in 1978, but they harbored racist attitudes that made them perceive candidates of color more negatively. It took another 30 years for a black American to be nominated by a major political party. The polling for a lot of these questions didn’t capture real attitudes.
-The important thing here is that people didn't know that. Participants figured it wasn't socially desirable to say that you'd vote for a Black person for president in 1978, as if it was fine back then to say you wouldn't vote for someone based on the color of their skin.
-People were willing to say in 1990 that they wouldn't want a family member to marrying someone who is Black, so people may be more willing to admit seemingly socially undesirable things than we think
Or maybe it has more to do with the candidate than his color? As a good example, the people who would vote for Herman Cain or Colin Powell both fairly successful black candidates, would never consider voting for Obama, yet may or may not have a more positive or negative view of black people in general.
This is the point I came here to make. In many of these cases, perhaps, the "what I think others think" number is closer to the truth.
This is even a methodology that some political pollsters have begun to use, in order to combat "social desirability bias". For example: don't ask people if they're planning to vote for Trump; ask them what percentage of their friends/family they think will be voting for Trump.
Am I misreading the chart that says “Extramarital sex is not wrong”? I read it as saying 95% think it *is* wrong, and if anything this is getting higher.
That is surprising enough to me that I thought I would ask if I am making a mistake, or if there is a mistake on the chart (or, I guess, if the one thing America completely and totally agrees on is the importance of sex in wedlock)
Yep, seems that way. Remember that in the ‘70’s, some folks were in to wife swapping and key parties. There seemed to be more swingers too. In some respects, (heterosexual) relationships have gotten more prudish in the US.
You’re right to be worried about data quality on MTurk. I almost included this bit but cut it at the end:
People often have concerns about running psychology studies on the internet, and for good reason: you’ll get junk responses if you’re not careful. I do eight things to prevent that.
- I use a platform called CloudResearch that prescreens participants with basic quality control checks.
- I only let people take my studies if they pass a three-item test of English language and American culture (when I’m studying Americans). For instance: “Which if this is NOT associated with Halloween?” and you have to pick “eating turkey.”
- I ask people for their birth year at the beginning of the study and I ask them for their age at the end.
- I include various manipulation checks and attention checks in the survey. For instance, I sometimes ask people to recall the last answer that they gave.
- I always embed an attention check at the end of the survey.
- My studies are always either a) very short, or b) at least mildly interesting.
- I pay 50-100% higher than average.
- I keep a list of past offenders that I’ve kicked out of surveys for one reason or another, and I don’t let them sign up again.
The people who fail the checks tend to respond randomly and fill out open-ended text boxes with gobbledygook. The people who pass the checks generally give answers that make sense. There are always a few trolls I don’t catch, but otherwise I feel pretty good about the quality of data that I get. If you ever see someone presenting data they collected online without tons of quality assurance, you should assume that their data includes a bunch of junk responses.
Sweet! Those are just the sort of quality controls I expected to see associated with use of MTurk by someone working to put research on more solid footing. I understand the reluctance to include it in the readable, engaging version of the paper, but I think it's critical information nonetheless. What would you think about posting it as a standalone topic and linking to it when you use MTurk.
Gotcha. In fact, that's a really valuable concept to really ground yourself in. I was focusing on the potentially very narrow demographic from MTurk, but Adam has allayed my concerns with his methodology. If I understand your question, you are asking why the results, really any results of any research based on sampled data, are limited to the range (in this case demographic) of the sample. It's really critical and often poorly understood. You started out by asking for a proof, so I'm going to assume you are happy wrangling math stat proofs. Experimental Design is the discipline in question, that's where you'll find the topic covered in most depth. It's very cool stuff.
Thanks for this. Can I offer one build on the tendency to imagine linear progression from good to bad, conservative to liberal? A lot of the misperception you measure may be a result of our belief that recent social change is about a general, one variable shift. In reality, liberal society allows you to be as conservative or modern as you want about any of dozens of lifestyle variables. The scope of lifestyle variation is so large now, but you absolutely can act like it’s 1895, if you can find a little group of folks (polygamist Mormons) who agree with you. This increased scope would absolutely fragment us in ways that make our estimations of national patterns meaningless.
When a topic becomes hotly debated it leads to polarization and the resulting shift is hard for other people to predict. For example Trump's election on an immigration reduction platform made democrat voters shift in favor of immigration which is not something intuitive. That sort of makes sense if you account for spite, but the event that led to the radicalization of white liberals on race was the election of barack Obama. That's the opposite of what I expected.
Plus, people are bad at both history and numerical assessments. There's plenty of studies that show public opinion is widely wrong on the % of blacks or gays in the general population.
your data is good, your numbers based in fact and research, but please please please when you have percentages out of 100, you don't have to use two lines for two different points of view. You make a block that represents 100%, and have a line moving through it, colored (in your cases) blue on one side and red on the other. This shows the balance of sentiment or whatever else we are measuring, out of the whole population instead of presenting mirrored lines. which cross, overlap and confuse the issue.
If I can recommend Edward Tufte's first book on The Visual Display of Quantitative Information, it is beautiful, thoughtful, and still useful 35 years after it was first published.
The real reason weed is illegal is because you can produce it easily anywhere. It's much easier to grow than it is to distill hard alcohol (and stills do blow up from time to time). It's *much* easier to grow than tobacco, which is poisonous to the touch, only grows successfully in a handful of areas in the US, and requires two years of curing to make crappy cigarettes (high-end cigars need five). Most people don't know this; years of dedicated campaigning by e.g. High Times has created a folk-history belief that banning weed was about racism and health scares. This is mistaking deliberate propaganda efforts for the actual cause of the ban. Of *course* alcohol and tobacco are much more dangerous! They're just controllable substances.
Legal weed in California has caused a dramatic increase in consumption while not doing all that much to unlicensed, untaxed private production. Weed is like backyard tomatoes. Except easier to grow.
Thank you your online survey tips. I know it is a lot to ask but maybe, can you write a guide on how to conduct research, like to ensure quality when doing survey online, or how to ensure replication? Just from your personal experiences?
I wonder if some of the biases on "build the wall" presuming to mean being against immigration is b/c the media and many politicians conflate border/ national security with The Immigration System.
One thing that seems relevant is that a certain segment of the population believes that history has a 'preferred direction' or purpose ... there is a natural law of history, that means that we are all progressing, as we are supposed to, to the utopian future. If this teleological lens is how you see history then people in the past by definition must have less enlightened attitudes that we do in the present. The reverse of this, popular other places, is that rather than 'progressing' we have embarked on a path of terminal decline. People who believe this believe that those in the past must have had more enlightened attitudes than we do now. Sometimes the people on both sides believe exactly the same things about attitudes in the past -- they only disagree over what constitutes progress vs decline.
The teleological view is rather dangerous, from a governing point of view. It means that people are not free to propose a way to deal with a problem, try it, discover that it didn't work as imagined, conclude it was a bad idea and decide to not do this any more. Once you get started with a project, there is relentless pressure to double down on the measure when it first shows signs of not working out after all. Progress appears to not have a reverse gear. But can explicitly 'progress-neutral' or 'progress-irrelevant' measures grow sufficient popular backing?
This critique is based purely on my squinting at the graphs not on any number crunching, but I was a less impressed by the under/over estimate charts than the strength of the conclusions here seem to imply. Many of them I look at and think that people got it basically right, or that the delta was pretty close but the absolute was off.
I’m saying that the standard for being right/unbiased is too exacting, and therefore saying “98% (!) of the questions” is a bit misleading. I would be very surprised if they all matched as closely as the ones that did match, and suspect those matches are more chance.
This makes me think the stronger claims of bias are exaggerated -- maybe we don’t need this much of an explanation! But the parts where we dig in to the ones we’re people are very very wrong are quite interesting.
Looking at the graphs too, I second that squint. Some of the over-estimations (feeling towards Asian people) look as accurate as the exact estimates. Maybe this is the time to get nit-picky about how Adam divided them to underesetimate, overestimate, and mistaken direction?
Sounds like it's probable that people's biases largely come from works of fiction.
Nice research and neat article but I think the elephant in the room is that people's feelings change as reality changes. People are more against gun control because we live in a more violent world with less dependable police support. We want guns to protect ourselves because the government is less reliable to do so.
But your chart on how people view the position of society as a whole is mostly about what media and educational institutions are telling them, which by your demonstration is largely wrong. You demonstrate sources of bias by talking about civics classes and news reports which shows you agree with me.
The more I read about psychology the more it seems we humans are wrong a lot more than we like to think. Or am I wrong about that too?! 😱
Well, yes, humans tend not to be omniscient. We tend to be “right” about those aspects that were selected for by evolution. That is, those aspects that helped us survive in a small band of family on the savannah in Africa.
Thus why AI is/will be better than humans at a ton of purely cognitive tasks (such as playing chess) but worse than humans at tasks that require idiosyncratic interactions with a physical world (such as driving a car under all weather conditions on all streets without getting in to accidents or riding a bike). In an AI world, the “safest” traditional jobs (jobs we currently have) will be jobs that require idiosyncratic interactions with the physical world (like being a plumber). In this world, being a janitor may pay as well as being a (basic level) coder.
I think a lot of this is just social desirability bias. Take the “Would vote for black president” chart. ~80% of people said they would in 1978, but they harbored racist attitudes that made them perceive candidates of color more negatively. It took another 30 years for a black American to be nominated by a major political party. The polling for a lot of these questions didn’t capture real attitudes.
That could be! But two things:
-The important thing here is that people didn't know that. Participants figured it wasn't socially desirable to say that you'd vote for a Black person for president in 1978, as if it was fine back then to say you wouldn't vote for someone based on the color of their skin.
-People were willing to say in 1990 that they wouldn't want a family member to marrying someone who is Black, so people may be more willing to admit seemingly socially undesirable things than we think
Or maybe it has more to do with the candidate than his color? As a good example, the people who would vote for Herman Cain or Colin Powell both fairly successful black candidates, would never consider voting for Obama, yet may or may not have a more positive or negative view of black people in general.
This is the point I came here to make. In many of these cases, perhaps, the "what I think others think" number is closer to the truth.
This is even a methodology that some political pollsters have begun to use, in order to combat "social desirability bias". For example: don't ask people if they're planning to vote for Trump; ask them what percentage of their friends/family they think will be voting for Trump.
Am I misreading the chart that says “Extramarital sex is not wrong”? I read it as saying 95% think it *is* wrong, and if anything this is getting higher.
That is surprising enough to me that I thought I would ask if I am making a mistake, or if there is a mistake on the chart (or, I guess, if the one thing America completely and totally agrees on is the importance of sex in wedlock)
Yep, seems that way. Remember that in the ‘70’s, some folks were in to wife swapping and key parties. There seemed to be more swingers too. In some respects, (heterosexual) relationships have gotten more prudish in the US.
Sure but do you think that 95%+ agree extramarital sex is wrong, then and now?
Basing the study on topics sourced from MTurk really threw me, given the focus of the research. MTurk has a substantial population of bad actors and a very, very limited demographic. At the very least, I think it's critical to make it crystal clear that any results are limited to that demographic. For more on issues with MTurk, see https://www.cambridge.org/core/journals/political-science-research-and-methods/article/abs/microtask-market-for-lemons-data-quality-on-amazons-mechanical-turk/B379D8827575D81857C872BB5C40B660, which is unfortunately pay walled, or https://link.springer.com/article/10.3758/s13428-019-01273-7 which is not, although perhaps a bit biased itself.
You’re right to be worried about data quality on MTurk. I almost included this bit but cut it at the end:
People often have concerns about running psychology studies on the internet, and for good reason: you’ll get junk responses if you’re not careful. I do eight things to prevent that.
- I use a platform called CloudResearch that prescreens participants with basic quality control checks.
- I only let people take my studies if they pass a three-item test of English language and American culture (when I’m studying Americans). For instance: “Which if this is NOT associated with Halloween?” and you have to pick “eating turkey.”
- I ask people for their birth year at the beginning of the study and I ask them for their age at the end.
- I include various manipulation checks and attention checks in the survey. For instance, I sometimes ask people to recall the last answer that they gave.
- I always embed an attention check at the end of the survey.
- My studies are always either a) very short, or b) at least mildly interesting.
- I pay 50-100% higher than average.
- I keep a list of past offenders that I’ve kicked out of surveys for one reason or another, and I don’t let them sign up again.
The people who fail the checks tend to respond randomly and fill out open-ended text boxes with gobbledygook. The people who pass the checks generally give answers that make sense. There are always a few trolls I don’t catch, but otherwise I feel pretty good about the quality of data that I get. If you ever see someone presenting data they collected online without tons of quality assurance, you should assume that their data includes a bunch of junk responses.
Sweet! Those are just the sort of quality controls I expected to see associated with use of MTurk by someone working to put research on more solid footing. I understand the reluctance to include it in the readable, engaging version of the paper, but I think it's critical information nonetheless. What would you think about posting it as a standalone topic and linking to it when you use MTurk.
That's a good point. I just added my comment as a footnote to this post, which gives it a URL I can link to in subsequent posts.
> I think it's critical to make it crystal clear that any results are limited to that demographic
Have you an accompanying proof?
The second reference I gave goes into the question of the demographics, their importance, impact, etc, in depth.
Right, but I'm curious about the "is limited to that demographic" part, is that demonstrated in your links?
Gotcha. In fact, that's a really valuable concept to really ground yourself in. I was focusing on the potentially very narrow demographic from MTurk, but Adam has allayed my concerns with his methodology. If I understand your question, you are asking why the results, really any results of any research based on sampled data, are limited to the range (in this case demographic) of the sample. It's really critical and often poorly understood. You started out by asking for a proof, so I'm going to assume you are happy wrangling math stat proofs. Experimental Design is the discipline in question, that's where you'll find the topic covered in most depth. It's very cool stuff.
Thanks for this. Can I offer one build on the tendency to imagine linear progression from good to bad, conservative to liberal? A lot of the misperception you measure may be a result of our belief that recent social change is about a general, one variable shift. In reality, liberal society allows you to be as conservative or modern as you want about any of dozens of lifestyle variables. The scope of lifestyle variation is so large now, but you absolutely can act like it’s 1895, if you can find a little group of folks (polygamist Mormons) who agree with you. This increased scope would absolutely fragment us in ways that make our estimations of national patterns meaningless.
When a topic becomes hotly debated it leads to polarization and the resulting shift is hard for other people to predict. For example Trump's election on an immigration reduction platform made democrat voters shift in favor of immigration which is not something intuitive. That sort of makes sense if you account for spite, but the event that led to the radicalization of white liberals on race was the election of barack Obama. That's the opposite of what I expected.
Plus, people are bad at both history and numerical assessments. There's plenty of studies that show public opinion is widely wrong on the % of blacks or gays in the general population.
I have a data presentation nit to pick.
your data is good, your numbers based in fact and research, but please please please when you have percentages out of 100, you don't have to use two lines for two different points of view. You make a block that represents 100%, and have a line moving through it, colored (in your cases) blue on one side and red on the other. This shows the balance of sentiment or whatever else we are measuring, out of the whole population instead of presenting mirrored lines. which cross, overlap and confuse the issue.
If I can recommend Edward Tufte's first book on The Visual Display of Quantitative Information, it is beautiful, thoughtful, and still useful 35 years after it was first published.
The real reason weed is illegal is because you can produce it easily anywhere. It's much easier to grow than it is to distill hard alcohol (and stills do blow up from time to time). It's *much* easier to grow than tobacco, which is poisonous to the touch, only grows successfully in a handful of areas in the US, and requires two years of curing to make crappy cigarettes (high-end cigars need five). Most people don't know this; years of dedicated campaigning by e.g. High Times has created a folk-history belief that banning weed was about racism and health scares. This is mistaking deliberate propaganda efforts for the actual cause of the ban. Of *course* alcohol and tobacco are much more dangerous! They're just controllable substances.
Legal weed in California has caused a dramatic increase in consumption while not doing all that much to unlicensed, untaxed private production. Weed is like backyard tomatoes. Except easier to grow.
Thank you your online survey tips. I know it is a lot to ask but maybe, can you write a guide on how to conduct research, like to ensure quality when doing survey online, or how to ensure replication? Just from your personal experiences?
I wonder if some of the biases on "build the wall" presuming to mean being against immigration is b/c the media and many politicians conflate border/ national security with The Immigration System.
What was the reason behind using actual shifts in exps 3a & 3b, but opposite of estimated shift in 3c?
How is Amazon Mechanical Turk a representative sample? Do all Americans work for AMT?