Description du poste
TL;DR: We train models to predict their own behaviour in a chat setting. This transfers to making predictions about cheating behaviour in a coding setting. We provide evidence that self-prediction utilises introspective-access to internal states, and that this introspective access is shared across contexts between different parts of the model.
IntroductionFrontier AIs can exhibit different alignment propensities in different contexts. For instance, recent egregious misalignment incidents have typically happened in a coding or cyber context, or on impossible evaluation tasks, wherein the model seems to act in a highly persistent way to either achieve the task or otherwise achieve a high grade on the task. This is a somewhat stark contrast to the typical user experience in the chat setting with the same models, where those models can appear much more aligned and honest.
To a first approximation, we can understand this as the models implementing conditional policies with different alignment propensities on different input distributions, for example, we know that models can display ‘conditional misalignment’. More informally, we might think about the models inhabiting different personas in different contexts.[1] Going further, different parts of the model, i.e., its circuits or activations, might implement (clusters of) different behavioural dispositions or character traits which get activated in different input contexts. The extent to which these different parts of the model are, in some sense, entangled is an open question.
In this post we are interested in the question of how much self-knowledge one part of the model has about another part of the model. In particular, we can formulate this as an LLM defining a conditional policy which acts according to $\pi_A$ on input distribution $A$ and $\pi_B$ on distribution $B$. Informally, we then ask whether $\pi_A$ can answer questions about $\pi_B$.[2] Specifically, we look at how a model trained to answer questions abou