Description du poste
Thanks to Piotr Dworczak, Annie Liang, and Alessandro Pavan for helpful discussions.
Various Anthropic and OpenAI models provided research assistance. The text is primarily human-written with short passages written by AI and grammar and clarity checks by AI. Proofs are in a separate appendix. All errors are my responsibility.
1. Introduction
Current AI systems often think explicitly about whether they are being monitored, tested, or trained and often behave differently when they believe this to be the case (Greenblatt et al. 2024; Meinke et al. 2024). A prominent concern is that a strategically sophisticated and situationally aware AI may conceal goals during training to avoid changes it would otherwise undergo. This is a form of deceptive alignment. This note uses screening techniques from mechanism design to ask what kinds of training and deployment rules incentivise honest training behaviour and which of these best balance alignment with the benefits of deployment.
We consider a game between a designer and an AI. The AI has preferences over its deployed behaviour that are unknown to the designer. We call the AI's initial preference over deployment behaviour its type. The designer chooses and commits to a training rule and deployment rule, which together constitute a mechanism. Training shifts the AI's preferences over deployed behaviour, depending on its training behaviour. The AI anticipates these changes and dislikes departures from its original preferences. The deployment rule specifies the degree of deployment and can also depend on the AI's training behaviour. The designer likes alignment—AI preferences close to the designer's own preferences—but also bears a cost from forgoing deployment. We say that a mechanism is incentive compatible if it is optimal for the AI to act in accordance with its true type during training.
Consider a mechanism that corrects whatever misalignment the AI reveals during training and then fully deploys it. If the AI honestly re