Description du poste
Scott Alexander posted a survey of utility weights for a bunch of different AI safety accomplishments, discussed more in Towards the Autograntor.
The survey asked people to assign point values to AI safety milestones. The values were based on the anchor of 1 million points = a pause with China à la AI Futures Project’s Plan A, so a milestone assigned 10,000 points is 1% as good as that. Here’s a link to our survey responses.
To be clear, we (especially Austin) didn’t spend that much time on this and have low confidence in our answers! But we think being transparent about even half-baked opinions is good.
Headline numbersAs examples, here are a few milestones where the two of us (and the autograntor reflecting Scott’s worldview) gave relatively similar values:
Milestone
Austin
Caroline
Scott
Founding of effective altruism
100,000
75,000
100,000
Small-grant ecosystem (Long-Term Future Fund, SFF and similar)
20,000
20,000
30,000
Lighthaven
10,000
5,000
9,000
OpenAI or Anthropic becomes 1 point more alignment-pilled (on a scale where 0 = NVIDIA, 100 = Anthropic)
2,000
500
3,000
Austin’s and Caroline’s average ratings for alignment research and community-building milestones were similarish to each other’s, with Caroline rating policy outcomes more highly and Austin rating social shifts more highly.
This table shows the sum of our point values within categories, for rows we both scored:
Category
Austin
Caroline
Politics (rows scored by both)
62k
96k
Alignment (rows scored by both)
68k
58k
Community building (all rows)
354k
282k
Social shifts (all rows)
80k
32k
Within community building, Caroline gave higher ratings to earlier community-building efforts and Austin gave higher ratings to later ones. To some extent this seems like a credit allocation problem—how much credit should, say, the original LessWrong get for LW 2.0? On individual milestones, some rated very differently by the two of us were:
Milestone
Austin
Caroline
Comments
Problem framing: orthogonality, instrume