You said such a variety of treasures in here, but I'm just stuck on the visual of a dispirited 18 year old sitting in their shitty car with a rejection letter that they paid to get. Driving home the point with a bulldozer, check!
Some good points and concepts. I must say, though, you fall into a trap (another one!) that I see a lot of professors fall into: inaccurately believing that grades /do/ not materially impact students' lives just because they /should/ not.
You think of it as Monopoly money--for the students, these arbitrary nonsense numbers literally determine their entire livelihoods. Can they keep their financial aid? Can they graduate? Can they get a good job after school?
Telling them the grades don't matter and are meaningless to /you/ doesn't help /them/ unless the next part of that speech is, "so you'll all get As."
Oh, this was very interesting. I live in India, and the government has been hyperfocused on improving our Ease of Doing Business ranking. I also own a business, and doing it has not become significantly easier. Now I know what this is called. :) Also, "irreducible woo" is a fantastic phrase, that I intend to use regularly, with credit to you.
I was teaching at Clemson when the institutional goal was to get into the top 25. Actually, it was pretty fun. The admin's attitude was "we're not sure how to get there, so try anything," as they handed out money for anything we proposed (nothing was too crazy to be considered). When the goal was achieved, the freedom to experiment ended: the admin said just keep doing the same thing (nothing new) so that we don't drop in the rankings.
Oct 31, 2023·edited Oct 31, 2023Liked by Adam Mastroianni
This was great to listen to. Observations:
1. Adam sounds like a young Jeff Goldblum circa Tenspeed and Brown Shoe (1980 TV series if you don't care to look it up);
2. U.S. News has dispensed with the "news" and "reporting" pretexts and eliminated the magazine altogether (Goodhart's law probably plays into that somehow).
3. Cousins of the U.S. News university rankings are the "vanity" reviews for lawyers: Best Lawyers (somehow associated with U.S. News) and Superlawyers. Everyone knows they are vanity marketing lists, but most lawyers--good lawyers mind you--dutifully list them on their web page bios.
I admit to being an Optimize Guy. However, I'm a multi-objective, Pareto Optimize Guy. So I seek to be on the efficient frontier (first Pareto front) with respect to interesting work, wealth, stress, personal relationships, exercise, health, leisure time, tasty food, LEGO builds, and naps.
The cute thing about Pareto optimization is that once the number of dimensions (objectives) is large enough, pretty much everything looks optimal, or at least is not provably sub-optimal. So I don't think I have Goodharted myself the way all those single-objective Optimize Guys have.
Huge thank you for this one. Must be the Taoist in me, but I've got to the point where seeing the word "optimize" outside of the context of machine processes annoys me. I just don't gel with it as a life approach.
Can you Goodhart improv comedy? I know someone who acted with Quipfire and later the Imps who made me laugh until I couldn't breathe. His was a life worth living, even if he ends up busking for quarters in the subway.
I would so love to read your take on assessment in higher education, which is what I immediately thought of when I read, "Any measure that becomes a target ceases to be a good measure." I am fairly convinced that the creeping tyranny of assessment is slowly ruining higher education in the U.S., which wasn't in such good shape to begin with. For many reasons but chief among them, learning is something that's really hard to measure. My favorite quote from Wendell Berry on this subject: “There is, thank God, no teacher-meter, and there is never going to be one. A teacher’s major contribution may pop out anonymously in the life of some ex-student’s grandchild. A teacher, finally, has nothing to go on but faith, a student nothing to offer in return but testimony.” Assessment is the full-hardy project of trying to build that teacher-meter.
Yah the fancy business schools also seem to breed super gamer guys. Met the target for that year but the wheels fall off the following year. I worked for a chemical company as an engineer in our technical support group. We were always asked to drive to $$ efficiency targets, which is a form of optimization. It did not take long for all the unintended consequences to actually make things worse. My target was to be moderation guy; everything in moderation. Do as much maintenance as you can afford and then stretch that a bit; prioritize safety, reliability and operability. Strangely enough, a good measure of $$ efficiency results. Check out Bruce Shnier's book: "A Hacker's Mind" This is a great look at this problem by an observer of the person gaming the system. It is based on "computer hacking", but he compares and contrasts that with hacking any kind of system (in the broadest sense of the definition of "system").
Speaking of the US News and World Report college rankings, aren't they notorious for rejiggering their weightings every year to ensure the "right" schools come out on top?
You said such a variety of treasures in here, but I'm just stuck on the visual of a dispirited 18 year old sitting in their shitty car with a rejection letter that they paid to get. Driving home the point with a bulldozer, check!
Some good points and concepts. I must say, though, you fall into a trap (another one!) that I see a lot of professors fall into: inaccurately believing that grades /do/ not materially impact students' lives just because they /should/ not.
You think of it as Monopoly money--for the students, these arbitrary nonsense numbers literally determine their entire livelihoods. Can they keep their financial aid? Can they graduate? Can they get a good job after school?
Telling them the grades don't matter and are meaningless to /you/ doesn't help /them/ unless the next part of that speech is, "so you'll all get As."
Oh, this was very interesting. I live in India, and the government has been hyperfocused on improving our Ease of Doing Business ranking. I also own a business, and doing it has not become significantly easier. Now I know what this is called. :) Also, "irreducible woo" is a fantastic phrase, that I intend to use regularly, with credit to you.
I was teaching at Clemson when the institutional goal was to get into the top 25. Actually, it was pretty fun. The admin's attitude was "we're not sure how to get there, so try anything," as they handed out money for anything we proposed (nothing was too crazy to be considered). When the goal was achieved, the freedom to experiment ended: the admin said just keep doing the same thing (nothing new) so that we don't drop in the rankings.
This was great to listen to. Observations:
1. Adam sounds like a young Jeff Goldblum circa Tenspeed and Brown Shoe (1980 TV series if you don't care to look it up);
2. U.S. News has dispensed with the "news" and "reporting" pretexts and eliminated the magazine altogether (Goodhart's law probably plays into that somehow).
3. Cousins of the U.S. News university rankings are the "vanity" reviews for lawyers: Best Lawyers (somehow associated with U.S. News) and Superlawyers. Everyone knows they are vanity marketing lists, but most lawyers--good lawyers mind you--dutifully list them on their web page bios.
*edit* to correct misspelling of Goodhart!
I admit to being an Optimize Guy. However, I'm a multi-objective, Pareto Optimize Guy. So I seek to be on the efficient frontier (first Pareto front) with respect to interesting work, wealth, stress, personal relationships, exercise, health, leisure time, tasty food, LEGO builds, and naps.
The cute thing about Pareto optimization is that once the number of dimensions (objectives) is large enough, pretty much everything looks optimal, or at least is not provably sub-optimal. So I don't think I have Goodharted myself the way all those single-objective Optimize Guys have.
dear adam,
as a former (?) or at least ASPIRING former Optimize Guy, this resonates hard.
thank you for sharing about Goodhart's Law. it's nice to have a Name For A Thing!
beautiful meaningful stuff, super appreciated!
love,
myq
I think you just shot to the top of the league tables for “most uses of the name Goodhart in one newsletter” :)
Huge thank you for this one. Must be the Taoist in me, but I've got to the point where seeing the word "optimize" outside of the context of machine processes annoys me. I just don't gel with it as a life approach.
Can you Goodhart improv comedy? I know someone who acted with Quipfire and later the Imps who made me laugh until I couldn't breathe. His was a life worth living, even if he ends up busking for quarters in the subway.
I would so love to read your take on assessment in higher education, which is what I immediately thought of when I read, "Any measure that becomes a target ceases to be a good measure." I am fairly convinced that the creeping tyranny of assessment is slowly ruining higher education in the U.S., which wasn't in such good shape to begin with. For many reasons but chief among them, learning is something that's really hard to measure. My favorite quote from Wendell Berry on this subject: “There is, thank God, no teacher-meter, and there is never going to be one. A teacher’s major contribution may pop out anonymously in the life of some ex-student’s grandchild. A teacher, finally, has nothing to go on but faith, a student nothing to offer in return but testimony.” Assessment is the full-hardy project of trying to build that teacher-meter.
What exactly was your dad doing when he took this picture?
Yah the fancy business schools also seem to breed super gamer guys. Met the target for that year but the wheels fall off the following year. I worked for a chemical company as an engineer in our technical support group. We were always asked to drive to $$ efficiency targets, which is a form of optimization. It did not take long for all the unintended consequences to actually make things worse. My target was to be moderation guy; everything in moderation. Do as much maintenance as you can afford and then stretch that a bit; prioritize safety, reliability and operability. Strangely enough, a good measure of $$ efficiency results. Check out Bruce Shnier's book: "A Hacker's Mind" This is a great look at this problem by an observer of the person gaming the system. It is based on "computer hacking", but he compares and contrasts that with hacking any kind of system (in the broadest sense of the definition of "system").
“I've seen people melt down over lots of dumb games: Monopoly, Scrabble, Mario Kart, mini golf, charades, bowling, trivia, and croquet.”
One of these games does not belong on this list ;-)
'Juking the stats'
https://www.youtube.com/watch?v=L5qD5qm8In4
Speaking of the US News and World Report college rankings, aren't they notorious for rejiggering their weightings every year to ensure the "right" schools come out on top?