In terms of control, I'm thinking of Wiseman's studies on luck. (Not a scientist, have no idea about the math on the original papers, but it sounds compelling enough.) He found that some people are more lucky than other people, and nearly all lucky people consider themselves to be lucky. His thesis, based on a couple ingenious experiments, is that lucky people, for whatever reason, *notice things in their environment more readily*, most usually opportunities. Visualizing does seem to work, at least not to hurt, some people; perhaps it's a way of trying to train yourself to notice things differently. It certainly crops up with surprising regularity in the careers of authors you've heard of.
Interestingly, *actual* medieval magic practitioners believed this is how magic really worked. It wasn't some kind of empirical or physics-based process as so many science folks assume.
The one common characteristics among successful entrepreneurs is an Internal Locus of Control (the belief that success is up to them and not from luck). That is consistent with what Wiseman found. My economics dissertation in 1981 paired this with the theoretical framework of Israel Kirzner defining entrepreneurs as alert to disequilibrium opportunities other are not.
Do they have an Internal Locus of Control because they succeeded though? Or is it an inherent characteristic they had before they went into entrepreneurship that helped lead to their success (correlation vs. causation)
Good question. I researched the evidence (perceptual typically) of heightened alertness in people with internal LOC. It makes sense as if you have an external LOC you’d be less inclined to look for opportunities. I am sure there is a strong feedback loop as well, but the start, before the success, requires logically a belief you can succeed rather than waiting for luck.
Regarding "just giving people money," wasn't it demonstrated that pandemic-era no-strings-attached cash distributions of about $2k per family literally lowered the national poverty rate? Why are we talking about whether the people were more or less depressed?
I'd rather not be starving. And actually I misspoke, I think it was specifically a reduction of *child poverty.* I also take the claims of increased depression with a grain of salt. The whole point of this post is social science research isn't very trustworthy, and measuring sadness strikes me as one of those things that's fuzzier than not. I also think there needs to be some theoretical connection between being less poor and more sad. What would that actually be?
Didn't those distributions also contribute to inflation? I'm reminded of wheelbarrows of money in Weimar Germany, or wheelbarrows of leaves when the B-ship landed on a new planet in a Douglas Adams book.
Sure, one of the best trade-offs there is. But we shouldn't be trading off, and the facts of it still don't change. And I got $1800 from that covid program. Yes, I took it, but I'm not a kid, and I have none.
What would have been best (few if any trade-offs) is if the statists didn't forcibly shut everything down and try to control us (they're not likely to get away with that again, not over a pandemic, at least). The payouts were a government program to make up for the previous government program. Like most government programs.
Some interventions do work. A beautiful natural experiment showed that an investment in human capital can have positive effects still measurable after 250 years. The intervention was education offered to Amazonian tribal members by Jesuit missionaries over the course of about 100 years. The control was the influence (but no education) of missionaries of other denominations. The missions were all abandoned due to some European upheaval. After 250 years, proximity to a former Jesuit mission vs. any other mission still has a statistically significant correlation with well-being of villages.
Thank you Matt, I will look up the study that you have shared.
At some point I would like to learn and understand the work of Helena Norberg-Hodge, Ancient Futures. I am not a contrarian, but I am wondering who can possibly prescribe improvement in similar scenarios, past and present
(1) Your intervention might not make much immediate difference. But, might have a big effect on a large number of descendants. So, if you teach a person to read, their life might not be much better, but their grandchildren's might be. Even a small difference for a large number of people is worthwhile.
(2) Make sure you are really delivering the intervention you claim. A recent NY Times article recounted how the author's daughter attended a public school where very little actual teaching occurred. So, naturally, she didn't learn much.
(3) Your positive results could be swamped by much greater negative influences that are beyond your control.
Problem is that we’ll wind up funding not only the Vice Provost, but also an Assistant Vice Provost, plus Administrative Assistants for each, and there won’t be any money left to buy a goat.
*Doing something* is easy, and psychologically satisfying. *Confirming that what you've done is working* is usually pretty hard, and has pretty good odds of proving that what you did was useless or even actively harmful, which is psychologically quite painful.
It's not surprising at all that so many people and organizations want to do something without measuring it, it is *absolutely* surprising that we let them. From the article - when they GAO asks DHS how they're going to measure success at stopping people from producing and distributing illicit fentanyl and DHS has *no answer*, they should be marked as failing, and that should be embarrassing.
If my wife asked me to wash the dishes and I ran the tap for 30 minutes without touching anything, she wouldn't care how many splashy presentations I made about how much water I used (water usage is correlated with clean dishes, everyone knows this), she'd point at the clearly unwashed dishes and ask what the hell is wrong with me!
Any chance we could get a follow-up article exploring how a complete lack of interest in actual outcomes is allowed to flourish as much as it does?
Thought-provoking article. But the primary thought it provoked in me is that the "results" aren't checked because the actual intended result has already been achieved: successful transfer of money, power, or prestige.
It seems to me that all of the failures you mention have that one successful result in common.
And once that one successful result is achieved, there is little incentive to analyze the post-result trivia.
Yes, I'm saying these interventions are grifts, but you analyzed them as if they were honest.
Rossi’s zinc law seems especially important here: the programs selected for evaluation may not represent the whole field. I wonder how we could estimate the missing successes and failures without assuming that the evaluated cases are typical. Seneca’s image of leaving the inheritance enlarged makes a good argument for publishing the disappointments too.
As a policy professor, one thing I feel compelled to note is that these evaluations are all of discrete interventions rather than system changes. These are the only sorts of thing you can do an experimental evaluation of, so that's understandable, but the big forces of the social world are structural (e.g. the sprawling urban form of America is antithetical to active modes of transport) and systemic (e.g. you can give a kid a library card but if all his friends think reading is lame it won't do much). Most of the biggest effects in policy are the result of system changes e.g. the advent of the UK welfare state after WWII.
In the more quantitative vein, you have the development of agent based models parameterised using results from randomised-control trials of single interventions. These then allow you to simulate the impacts of a system level intervention. They've been quite successfully used in public health, notably childhood obesity (e.g. the Romp and Chomp program), where you need several interventions simultaneously (canteen menu, bike to school, parental cooking classes) and at multiple scales of policy (school curriculum, local bike infrastructure, food system, etc). Ross Hammond does some of the best work on this stuff, e.g.
Part of our problem is that the mania for *certainty* among academics, especially over in micro-econometrics (my PhD), means that they poo these sorts of approaches and insist on things amenable to experimental designs. The first signs of a sea change in this regard are visible, but it's going to take a very long time to change course:
Incidentally, the field of "Evaluation", which emerged out of policy studies, always had a more measured perspective on randomised-control trials, viewing them as one very important plank of a wider evidence and learning enterprise, e.g.
"giving away dunderheaded donations is better than a lot of things that billionaires could be doing with their money, like buying a second superyacht or digging an additional doomsday bunker"
I beg to differ. Rich people spending their wealth leads to increased employment, enhanced skills, vibrant industries and people being able buy homes and put food on the table. A far better outcome than charity and sociology interventions.
Good point as it relates to the article. Buying a $100M yacht certainly does have some economic benefit for the industries of building and maintaining these ships. I would have thought using that $100M to specifically do good would have a larger benefit, but alas, what this article points out is that may not be the case.
A workplace wellness programme can appear successful before it has changed anything.
In the Illinois Workplace Wellness Study, Jones, Molitor and Reif randomised 4,834 university employees and found no statistically significant effects on 40 of the 42 outcomes measured in the first year. Those who chose to participate had already been healthier.
That distinction raises a larger question. How often do organisations measure employees' choices while leaving working hours, workload and physical environments unexamined?
The study does not answer that question. It does make the question considerably harder to ignore.
This brings to mind another generalization: effectiveness of interventions works on an individual face to face level and is dependent on each unique context, rather than on any general principle. Let's call it "the non-generalization generalization." I'm also thinking of the hypnotherapist Milton Erickson, "Uncommon Therapy", He appeared to be effective, by using hypnotic tricks to effect positive change in his patients. But there were no common principles to draw from his technique, which focused entirely on the uniqueness of each individual patient. It's been about thirty years since I read about him. I guess I could say that if there was anything close to a common principle it was the importance of storytelling, which appears also to be fundamental to this blog.
This is a great comment. Like how if you wanted to maximally improve the education of as many kids as possible, then your best bet is almost certainly to just become a good teacher and teach them yourself, rather than be some curriculum-designing wonk or even donate tons of money to schools
I wouldn't want to say that hypnosis is necessarily the reason Erickson was successful, since I believe all the things he did could be done without using hypnosis, as Jay Haley was able to demonstrate. But it does seem to require a real relationship between therapist and client.
We are the Solutionists! I liked the provocation about the goat sacrifice, and it appears to me (my gut feeling) that we should not make fun of Greeks and Romans who visited Oracle of Delphi with gifts to impress, for 800+ years…and to make sure the divine providence remains on their side.
You don't fail to surprise me. I'm always in awe of what you write. Thankyou soo much :) Also, this isn't fair that won't get the surprise if I purchase your book just because I from India! Not fair Adam :/
Lovee what you write. This article will change my perspective about a lot of things. I needed this especially when I wish to pursue Economics and Policy.
In terms of control, I'm thinking of Wiseman's studies on luck. (Not a scientist, have no idea about the math on the original papers, but it sounds compelling enough.) He found that some people are more lucky than other people, and nearly all lucky people consider themselves to be lucky. His thesis, based on a couple ingenious experiments, is that lucky people, for whatever reason, *notice things in their environment more readily*, most usually opportunities. Visualizing does seem to work, at least not to hurt, some people; perhaps it's a way of trying to train yourself to notice things differently. It certainly crops up with surprising regularity in the careers of authors you've heard of.
Interestingly, *actual* medieval magic practitioners believed this is how magic really worked. It wasn't some kind of empirical or physics-based process as so many science folks assume.
The one common characteristics among successful entrepreneurs is an Internal Locus of Control (the belief that success is up to them and not from luck). That is consistent with what Wiseman found. My economics dissertation in 1981 paired this with the theoretical framework of Israel Kirzner defining entrepreneurs as alert to disequilibrium opportunities other are not.
Do they have an Internal Locus of Control because they succeeded though? Or is it an inherent characteristic they had before they went into entrepreneurship that helped lead to their success (correlation vs. causation)
Good question. I researched the evidence (perceptual typically) of heightened alertness in people with internal LOC. It makes sense as if you have an external LOC you’d be less inclined to look for opportunities. I am sure there is a strong feedback loop as well, but the start, before the success, requires logically a belief you can succeed rather than waiting for luck.
Regarding "just giving people money," wasn't it demonstrated that pandemic-era no-strings-attached cash distributions of about $2k per family literally lowered the national poverty rate? Why are we talking about whether the people were more or less depressed?
Would you rather be depressed or poor? Serious question. I've been both, and I would choose poor
I'd rather not be starving. And actually I misspoke, I think it was specifically a reduction of *child poverty.* I also take the claims of increased depression with a grain of salt. The whole point of this post is social science research isn't very trustworthy, and measuring sadness strikes me as one of those things that's fuzzier than not. I also think there needs to be some theoretical connection between being less poor and more sad. What would that actually be?
Ah. I'm with you 💯 re: child poverty
Didn't those distributions also contribute to inflation? I'm reminded of wheelbarrows of money in Weimar Germany, or wheelbarrows of leaves when the B-ship landed on a new planet in a Douglas Adams book.
Lots of things contribute to inflation. Ensuring children are fed seems like a decent tradeoff.
Sure, one of the best trade-offs there is. But we shouldn't be trading off, and the facts of it still don't change. And I got $1800 from that covid program. Yes, I took it, but I'm not a kid, and I have none.
What would have been best (few if any trade-offs) is if the statists didn't forcibly shut everything down and try to control us (they're not likely to get away with that again, not over a pandemic, at least). The payouts were a government program to make up for the previous government program. Like most government programs.
Some interventions do work. A beautiful natural experiment showed that an investment in human capital can have positive effects still measurable after 250 years. The intervention was education offered to Amazonian tribal members by Jesuit missionaries over the course of about 100 years. The control was the influence (but no education) of missionaries of other denominations. The missions were all abandoned due to some European upheaval. After 250 years, proximity to a former Jesuit mission vs. any other mission still has a statistically significant correlation with well-being of villages.
So, my suggestion to you: Don't be such a cynic!
Where can I look this up please? How the well-being is understood and being contrasted?
https://cepr.org/voxeu/columns/missionaries-human-capital-transmission-and-economic-persistence-south-america
https://academic.oup.com/qje/article/134/1/507/5123742
https://www.washingtonpost.com/business/2018/11/09/years-after-jesuits-were-expelled-towns-near-their-missions-still-have-higher-education-incomes/?utm_term=.ba46c643a28f
Thank you Matt, I will look up the study that you have shared.
At some point I would like to learn and understand the work of Helena Norberg-Hodge, Ancient Futures. I am not a contrarian, but I am wondering who can possibly prescribe improvement in similar scenarios, past and present
Dina, the lessons I would take away are these:
(1) Your intervention might not make much immediate difference. But, might have a big effect on a large number of descendants. So, if you teach a person to read, their life might not be much better, but their grandchildren's might be. Even a small difference for a large number of people is worthwhile.
(2) Make sure you are really delivering the intervention you claim. A recent NY Times article recounted how the author's daughter attended a public school where very little actual teaching occurred. So, naturally, she didn't learn much.
(3) Your positive results could be swamped by much greater negative influences that are beyond your control.
“How can you say we know nothing when we have so many TED talks?”
OMFG.
This deserves a Nobel prize. Just for this line.
This will require a Dean, if not a Vice Provost, for Goat Sacrifices. What a great article! Thanks for your work and insight.
Problem is that we’ll wind up funding not only the Vice Provost, but also an Assistant Vice Provost, plus Administrative Assistants for each, and there won’t be any money left to buy a goat.
Definitely a wise approach.
You may want to follow the Taoists and replace the blood, wine, and meat offerings with water, fruit, and veggies, though.
Not only is the mess easier to clean up, but policy officials and academics can think more clearly when not drunk or choking on burnt-fat smoke.
Plus (so I'm told), the vegan deities such rituals attract are all-around more pleasant than the "now we want first-borns" variety.
You get the humiliation ritual benefits without the slippery-slope descent into a scapegoat industrial complex.
*Doing something* is easy, and psychologically satisfying. *Confirming that what you've done is working* is usually pretty hard, and has pretty good odds of proving that what you did was useless or even actively harmful, which is psychologically quite painful.
It's not surprising at all that so many people and organizations want to do something without measuring it, it is *absolutely* surprising that we let them. From the article - when they GAO asks DHS how they're going to measure success at stopping people from producing and distributing illicit fentanyl and DHS has *no answer*, they should be marked as failing, and that should be embarrassing.
If my wife asked me to wash the dishes and I ran the tap for 30 minutes without touching anything, she wouldn't care how many splashy presentations I made about how much water I used (water usage is correlated with clean dishes, everyone knows this), she'd point at the clearly unwashed dishes and ask what the hell is wrong with me!
Any chance we could get a follow-up article exploring how a complete lack of interest in actual outcomes is allowed to flourish as much as it does?
Thought-provoking article. But the primary thought it provoked in me is that the "results" aren't checked because the actual intended result has already been achieved: successful transfer of money, power, or prestige.
It seems to me that all of the failures you mention have that one successful result in common.
And once that one successful result is achieved, there is little incentive to analyze the post-result trivia.
Yes, I'm saying these interventions are grifts, but you analyzed them as if they were honest.
Rossi’s zinc law seems especially important here: the programs selected for evaluation may not represent the whole field. I wonder how we could estimate the missing successes and failures without assuming that the evaluated cases are typical. Seneca’s image of leaving the inheritance enlarged makes a good argument for publishing the disappointments too.
Great post as always.
As a policy professor, one thing I feel compelled to note is that these evaluations are all of discrete interventions rather than system changes. These are the only sorts of thing you can do an experimental evaluation of, so that's understandable, but the big forces of the social world are structural (e.g. the sprawling urban form of America is antithetical to active modes of transport) and systemic (e.g. you can give a kid a library card but if all his friends think reading is lame it won't do much). Most of the biggest effects in policy are the result of system changes e.g. the advent of the UK welfare state after WWII.
System change evaluation is a nightmare, but there are some advances being made, e.g. https://www.socialfinance.org.uk/evidence/navigating-system-change-evaluation.
In the more quantitative vein, you have the development of agent based models parameterised using results from randomised-control trials of single interventions. These then allow you to simulate the impacts of a system level intervention. They've been quite successfully used in public health, notably childhood obesity (e.g. the Romp and Chomp program), where you need several interventions simultaneously (canteen menu, bike to school, parental cooking classes) and at multiple scales of policy (school curriculum, local bike infrastructure, food system, etc). Ross Hammond does some of the best work on this stuff, e.g.
https://www.pnas.org/doi/10.1073/pnas.2314993121
Part of our problem is that the mania for *certainty* among academics, especially over in micro-econometrics (my PhD), means that they poo these sorts of approaches and insist on things amenable to experimental designs. The first signs of a sea change in this regard are visible, but it's going to take a very long time to change course:
https://www.nber.org/system/files/working_papers/w34381/w34381.pdf
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4445710
Incidentally, the field of "Evaluation", which emerged out of policy studies, always had a more measured perspective on randomised-control trials, viewing them as one very important plank of a wider evidence and learning enterprise, e.g.
https://www.amazon.co.uk/Evidence-Based-Policy-Perspective-Ray-Pawson/dp/1412910609
"giving away dunderheaded donations is better than a lot of things that billionaires could be doing with their money, like buying a second superyacht or digging an additional doomsday bunker"
I beg to differ. Rich people spending their wealth leads to increased employment, enhanced skills, vibrant industries and people being able buy homes and put food on the table. A far better outcome than charity and sociology interventions.
Good point as it relates to the article. Buying a $100M yacht certainly does have some economic benefit for the industries of building and maintaining these ships. I would have thought using that $100M to specifically do good would have a larger benefit, but alas, what this article points out is that may not be the case.
Good point.
A workplace wellness programme can appear successful before it has changed anything.
In the Illinois Workplace Wellness Study, Jones, Molitor and Reif randomised 4,834 university employees and found no statistically significant effects on 40 of the 42 outcomes measured in the first year. Those who chose to participate had already been healthier.
That distinction raises a larger question. How often do organisations measure employees' choices while leaving working hours, workload and physical environments unexamined?
The study does not answer that question. It does make the question considerably harder to ignore.
This brings to mind another generalization: effectiveness of interventions works on an individual face to face level and is dependent on each unique context, rather than on any general principle. Let's call it "the non-generalization generalization." I'm also thinking of the hypnotherapist Milton Erickson, "Uncommon Therapy", He appeared to be effective, by using hypnotic tricks to effect positive change in his patients. But there were no common principles to draw from his technique, which focused entirely on the uniqueness of each individual patient. It's been about thirty years since I read about him. I guess I could say that if there was anything close to a common principle it was the importance of storytelling, which appears also to be fundamental to this blog.
This is a great comment. Like how if you wanted to maximally improve the education of as many kids as possible, then your best bet is almost certainly to just become a good teacher and teach them yourself, rather than be some curriculum-designing wonk or even donate tons of money to schools
I wouldn't want to say that hypnosis is necessarily the reason Erickson was successful, since I believe all the things he did could be done without using hypnosis, as Jay Haley was able to demonstrate. But it does seem to require a real relationship between therapist and client.
We are the Solutionists! I liked the provocation about the goat sacrifice, and it appears to me (my gut feeling) that we should not make fun of Greeks and Romans who visited Oracle of Delphi with gifts to impress, for 800+ years…and to make sure the divine providence remains on their side.
I'm not religious or anything but maybe we'll one day discover that “the thing that works” was written in the Ten Commandments.
You don't fail to surprise me. I'm always in awe of what you write. Thankyou soo much :) Also, this isn't fair that won't get the surprise if I purchase your book just because I from India! Not fair Adam :/
Lovee what you write. This article will change my perspective about a lot of things. I needed this especially when I wish to pursue Economics and Policy.
Thankyou 🌼