You said such a variety of treasures in here, but I'm just stuck on the visual of a dispirited 18 year old sitting in their shitty car with a rejection letter that they paid to get. Driving home the point with a bulldozer, check!
Some good points and concepts. I must say, though, you fall into a trap (another one!) that I see a lot of professors fall into: inaccurately believing that grades /do/ not materially impact students' lives just because they /should/ not.
You think of it as Monopoly money--for the students, these arbitrary nonsense numbers literally determine their entire livelihoods. Can they keep their financial aid? Can they graduate? Can they get a good job after school?
Telling them the grades don't matter and are meaningless to /you/ doesn't help /them/ unless the next part of that speech is, "so you'll all get As."
Context from the original post: Columbia Business School suppresses grades, meaning nobody but the student and the professor are supposed to see them. And I taught MBA students, so their grades were as close to Monopoly money as it gets. https://www.experimental-history.com/p/i-wanted-to-be-a-teacher-but-they
Grades do matter to professors but in a different way. I had a very hard-working class one semester, the energy was high, and it was my best teaching. Everyone got an A. What was the conclusion? I was deemed to be "easy." The students were viewed as slackers looking for an easy A. I was viewed as a pushover, rather than an exceptionally good teacher, and my reputation within the department suffered. There is a politics to grading and the interests of students and professors are in conflict.
Oh, this was very interesting. I live in India, and the government has been hyperfocused on improving our Ease of Doing Business ranking. I also own a business, and doing it has not become significantly easier. Now I know what this is called. :) Also, "irreducible woo" is a fantastic phrase, that I intend to use regularly, with credit to you.
I was teaching at Clemson when the institutional goal was to get into the top 25. Actually, it was pretty fun. The admin's attitude was "we're not sure how to get there, so try anything," as they handed out money for anything we proposed (nothing was too crazy to be considered). When the goal was achieved, the freedom to experiment ended: the admin said just keep doing the same thing (nothing new) so that we don't drop in the rankings.
Oct 31, 2023·edited Oct 31, 2023Liked by Adam Mastroianni
This was great to listen to. Observations:
1. Adam sounds like a young Jeff Goldblum circa Tenspeed and Brown Shoe (1980 TV series if you don't care to look it up);
2. U.S. News has dispensed with the "news" and "reporting" pretexts and eliminated the magazine altogether (Goodhart's law probably plays into that somehow).
3. Cousins of the U.S. News university rankings are the "vanity" reviews for lawyers: Best Lawyers (somehow associated with U.S. News) and Superlawyers. Everyone knows they are vanity marketing lists, but most lawyers--good lawyers mind you--dutifully list them on their web page bios.
I admit to being an Optimize Guy. However, I'm a multi-objective, Pareto Optimize Guy. So I seek to be on the efficient frontier (first Pareto front) with respect to interesting work, wealth, stress, personal relationships, exercise, health, leisure time, tasty food, LEGO builds, and naps.
The cute thing about Pareto optimization is that once the number of dimensions (objectives) is large enough, pretty much everything looks optimal, or at least is not provably sub-optimal. So I don't think I have Goodharted myself the way all those single-objective Optimize Guys have.
Luckily thousands of cohorts of your ancestors have already created a perfect objective that optimizes across multiple frontiers and various life scenarios. It's called religion.
wow, way to optimize the optimize game! ... Seriously though, I think you have a point which Adam hasn't mentioned: some people derive satisfaction from the optimizing itself. It's fun to figure out how to target all those areas you mentioned in some "optimal" way. I think the key to make that endeavor successful is to not overly quantify things that are inherently qualitative. Maybe 80/20ing your (DHs) life might be the right amount of quantification without Goodharting yourself? Probably differs from person to person. (A psychologist in me wonders if it'd be a fun endeavor to come up with a "Goodhart scale", measure people's tendency to Goodhart themselves (probably a combination of conscientiousness and social desirability?), and come up with a bunch of outcome variables associated with this "personality" trait. Bam, a Nature-worthy article that's probably unreplicable but, oh boy, it gives me the "kinda-out-there-yet-still-possible" vibes)
On a related note, what I also think the piece missed is this: if Goodharting means setting up a metric and then struggling to achieve it, then this is a great way to jumpstart some meaningful changes in your life. For instance, you can set up a habit tracker for your meditation practice, sticking to it for a month or so (or however long it takes you to build a habit) and ditch it as soon as you notice that you'd meditate anyway, even without the tracker and the reminder. In theory, as long as you use the Goodhart logic only in this beginning stage, you get the benefits without the downsides.
Huge thank you for this one. Must be the Taoist in me, but I've got to the point where seeing the word "optimize" outside of the context of machine processes annoys me. I just don't gel with it as a life approach.
Can you Goodhart improv comedy? I know someone who acted with Quipfire and later the Imps who made me laugh until I couldn't breathe. His was a life worth living, even if he ends up busking for quarters in the subway.
Jerry Seinfeld is the quintessential "optimize guy" of comedy. He literally tested every line in his jokes 5 times (at a low-ranking/unimportant comedy club) before adding them to his professional set. He also, quite literally, memorizes how much time to pause before transitioning to the next line/joke.
I would so love to read your take on assessment in higher education, which is what I immediately thought of when I read, "Any measure that becomes a target ceases to be a good measure." I am fairly convinced that the creeping tyranny of assessment is slowly ruining higher education in the U.S., which wasn't in such good shape to begin with. For many reasons but chief among them, learning is something that's really hard to measure. My favorite quote from Wendell Berry on this subject: “There is, thank God, no teacher-meter, and there is never going to be one. A teacher’s major contribution may pop out anonymously in the life of some ex-student’s grandchild. A teacher, finally, has nothing to go on but faith, a student nothing to offer in return but testimony.” Assessment is the full-hardy project of trying to build that teacher-meter.
Photo was taken at a county fair for the local paper. Group was demonstrating Hollywood stunt men activities. I can assure you I was not harmed during the exhibition, I can’t vouch for the guy on fire. It was only in Black and White back in the 1980s, not that intense modern colorful flame of this century:-)
Yah the fancy business schools also seem to breed super gamer guys. Met the target for that year but the wheels fall off the following year. I worked for a chemical company as an engineer in our technical support group. We were always asked to drive to $$ efficiency targets, which is a form of optimization. It did not take long for all the unintended consequences to actually make things worse. My target was to be moderation guy; everything in moderation. Do as much maintenance as you can afford and then stretch that a bit; prioritize safety, reliability and operability. Strangely enough, a good measure of $$ efficiency results. Check out Bruce Shnier's book: "A Hacker's Mind" This is a great look at this problem by an observer of the person gaming the system. It is based on "computer hacking", but he compares and contrasts that with hacking any kind of system (in the broadest sense of the definition of "system").
Speaking of the US News and World Report college rankings, aren't they notorious for rejiggering their weightings every year to ensure the "right" schools come out on top?
You said such a variety of treasures in here, but I'm just stuck on the visual of a dispirited 18 year old sitting in their shitty car with a rejection letter that they paid to get. Driving home the point with a bulldozer, check!
Some good points and concepts. I must say, though, you fall into a trap (another one!) that I see a lot of professors fall into: inaccurately believing that grades /do/ not materially impact students' lives just because they /should/ not.
You think of it as Monopoly money--for the students, these arbitrary nonsense numbers literally determine their entire livelihoods. Can they keep their financial aid? Can they graduate? Can they get a good job after school?
Telling them the grades don't matter and are meaningless to /you/ doesn't help /them/ unless the next part of that speech is, "so you'll all get As."
Context from the original post: Columbia Business School suppresses grades, meaning nobody but the student and the professor are supposed to see them. And I taught MBA students, so their grades were as close to Monopoly money as it gets. https://www.experimental-history.com/p/i-wanted-to-be-a-teacher-but-they
Grades do matter to professors but in a different way. I had a very hard-working class one semester, the energy was high, and it was my best teaching. Everyone got an A. What was the conclusion? I was deemed to be "easy." The students were viewed as slackers looking for an easy A. I was viewed as a pushover, rather than an exceptionally good teacher, and my reputation within the department suffered. There is a politics to grading and the interests of students and professors are in conflict.
That's so frustrating!
The GPA to keep fin aid is what? 2.0? And really only your first job out of college cares about your grades.
I think this is a key point. While I agree that Goodhart games are not an end in themselves, they are often a means to an end
Oh, this was very interesting. I live in India, and the government has been hyperfocused on improving our Ease of Doing Business ranking. I also own a business, and doing it has not become significantly easier. Now I know what this is called. :) Also, "irreducible woo" is a fantastic phrase, that I intend to use regularly, with credit to you.
I was teaching at Clemson when the institutional goal was to get into the top 25. Actually, it was pretty fun. The admin's attitude was "we're not sure how to get there, so try anything," as they handed out money for anything we proposed (nothing was too crazy to be considered). When the goal was achieved, the freedom to experiment ended: the admin said just keep doing the same thing (nothing new) so that we don't drop in the rankings.
This was great to listen to. Observations:
1. Adam sounds like a young Jeff Goldblum circa Tenspeed and Brown Shoe (1980 TV series if you don't care to look it up);
2. U.S. News has dispensed with the "news" and "reporting" pretexts and eliminated the magazine altogether (Goodhart's law probably plays into that somehow).
3. Cousins of the U.S. News university rankings are the "vanity" reviews for lawyers: Best Lawyers (somehow associated with U.S. News) and Superlawyers. Everyone knows they are vanity marketing lists, but most lawyers--good lawyers mind you--dutifully list them on their web page bios.
*edit* to correct misspelling of Goodhart!
I admit to being an Optimize Guy. However, I'm a multi-objective, Pareto Optimize Guy. So I seek to be on the efficient frontier (first Pareto front) with respect to interesting work, wealth, stress, personal relationships, exercise, health, leisure time, tasty food, LEGO builds, and naps.
The cute thing about Pareto optimization is that once the number of dimensions (objectives) is large enough, pretty much everything looks optimal, or at least is not provably sub-optimal. So I don't think I have Goodharted myself the way all those single-objective Optimize Guys have.
DH, you're an optimized Optimize Guy. You are meta optimized!
Luckily thousands of cohorts of your ancestors have already created a perfect objective that optimizes across multiple frontiers and various life scenarios. It's called religion.
wow, way to optimize the optimize game! ... Seriously though, I think you have a point which Adam hasn't mentioned: some people derive satisfaction from the optimizing itself. It's fun to figure out how to target all those areas you mentioned in some "optimal" way. I think the key to make that endeavor successful is to not overly quantify things that are inherently qualitative. Maybe 80/20ing your (DHs) life might be the right amount of quantification without Goodharting yourself? Probably differs from person to person. (A psychologist in me wonders if it'd be a fun endeavor to come up with a "Goodhart scale", measure people's tendency to Goodhart themselves (probably a combination of conscientiousness and social desirability?), and come up with a bunch of outcome variables associated with this "personality" trait. Bam, a Nature-worthy article that's probably unreplicable but, oh boy, it gives me the "kinda-out-there-yet-still-possible" vibes)
On a related note, what I also think the piece missed is this: if Goodharting means setting up a metric and then struggling to achieve it, then this is a great way to jumpstart some meaningful changes in your life. For instance, you can set up a habit tracker for your meditation practice, sticking to it for a month or so (or however long it takes you to build a habit) and ditch it as soon as you notice that you'd meditate anyway, even without the tracker and the reminder. In theory, as long as you use the Goodhart logic only in this beginning stage, you get the benefits without the downsides.
If you come up with that Goodhart Scale, I would like to see it.
dear adam,
as a former (?) or at least ASPIRING former Optimize Guy, this resonates hard.
thank you for sharing about Goodhart's Law. it's nice to have a Name For A Thing!
beautiful meaningful stuff, super appreciated!
love,
myq
I think you just shot to the top of the league tables for “most uses of the name Goodhart in one newsletter” :)
Huge thank you for this one. Must be the Taoist in me, but I've got to the point where seeing the word "optimize" outside of the context of machine processes annoys me. I just don't gel with it as a life approach.
Can you Goodhart improv comedy? I know someone who acted with Quipfire and later the Imps who made me laugh until I couldn't breathe. His was a life worth living, even if he ends up busking for quarters in the subway.
Jerry Seinfeld is the quintessential "optimize guy" of comedy. He literally tested every line in his jokes 5 times (at a low-ranking/unimportant comedy club) before adding them to his professional set. He also, quite literally, memorizes how much time to pause before transitioning to the next line/joke.
And I never found Seinfeld funny.
I would so love to read your take on assessment in higher education, which is what I immediately thought of when I read, "Any measure that becomes a target ceases to be a good measure." I am fairly convinced that the creeping tyranny of assessment is slowly ruining higher education in the U.S., which wasn't in such good shape to begin with. For many reasons but chief among them, learning is something that's really hard to measure. My favorite quote from Wendell Berry on this subject: “There is, thank God, no teacher-meter, and there is never going to be one. A teacher’s major contribution may pop out anonymously in the life of some ex-student’s grandchild. A teacher, finally, has nothing to go on but faith, a student nothing to offer in return but testimony.” Assessment is the full-hardy project of trying to build that teacher-meter.
The Berry quote is gold.
Definitely. A quote that's stuck with me through the years.
What exactly was your dad doing when he took this picture?
Photo was taken at a county fair for the local paper. Group was demonstrating Hollywood stunt men activities. I can assure you I was not harmed during the exhibition, I can’t vouch for the guy on fire. It was only in Black and White back in the 1980s, not that intense modern colorful flame of this century:-)
Yah the fancy business schools also seem to breed super gamer guys. Met the target for that year but the wheels fall off the following year. I worked for a chemical company as an engineer in our technical support group. We were always asked to drive to $$ efficiency targets, which is a form of optimization. It did not take long for all the unintended consequences to actually make things worse. My target was to be moderation guy; everything in moderation. Do as much maintenance as you can afford and then stretch that a bit; prioritize safety, reliability and operability. Strangely enough, a good measure of $$ efficiency results. Check out Bruce Shnier's book: "A Hacker's Mind" This is a great look at this problem by an observer of the person gaming the system. It is based on "computer hacking", but he compares and contrasts that with hacking any kind of system (in the broadest sense of the definition of "system").
“I've seen people melt down over lots of dumb games: Monopoly, Scrabble, Mario Kart, mini golf, charades, bowling, trivia, and croquet.”
One of these games does not belong on this list ;-)
Fair. I understand if you only play mini golf for blood or money.
'Juking the stats'
https://www.youtube.com/watch?v=L5qD5qm8In4
Speaking of the US News and World Report college rankings, aren't they notorious for rejiggering their weightings every year to ensure the "right" schools come out on top?
Yes. It's such a flawed ranking because it's so gameable. At least Forbes isn't so gameable.