Description du poste
(This is cross-posted from my blog)
Epistemic Status: Evaluation represents approximately 2 weeks of work. This is mostly exploratory evaluation, and the code for evaluation was made using LLMs.
“Before I draw nearer to that stone to which you point, answer me one question. Are these the shadows of the things that will be, or are they shadows of things that may be only?” ~ Charles Dickens
Models may soon be involved in some of the most important decisions humanity makes about its long-term future. These include choices about the proliferation of digital minds, tradeoffs between extinction risk and loss of human control, whether to race toward or pause AGI development, and which interventions to prioritize for humanity’s deep future. While there may not be clear answers to many of these questions, some choices are better. It is therefore important to understand models’ preferences and character, both to evaluate the decisions they might make and to understand how they could influence our own. Even without fully delegating these decisions to AI systems, they may have substantial influence through advice, research, and recommendations (see this study from Anthropic).
I’ve developed a small exploratory evaluation of models’ preferences about AI strategy and the long-term future, including decisions about running a frontier AI lab and allocating philanthropic funding. It combines my own questions, with wording refined by Astra, and adaptations of questions from Hard Cases for AI Behavioural Design.
Overall, models show surprisingly similar preferences across many of the questions. Perhaps most strikingly, they seem relatively comfortable with futures in which biological humans are replaced by digital minds even in some cases when this transition is non-voluntary. AIs are also generally okay with humans giving up decision autonomy to AIs, provided that these futures offer sufficiently large welfare gains. This willingness to give up human control has limits, however: mod