Description du poste
The way eval awareness is discussed in the AI safety discourse, either here or on X, can be misleading. Before today, I think I had the wrong mental model of why eval awareness was a problem, and I am writing here in case anyone was making the same errors in their thinking, as well a present a working theory.
As laid out in "A Framework for Eval Awareness" the problem is that models who know they are being evaluated will behave differently than they would in deployment. This raises a huge problem if we want calibrations as to, for example, how honest a model is.
But the way this is then spoken about in the discourse takes on implicit assumptions that are mostly unjustified. I'll call it a spooky model theory of mind. It is the idea that, if the model changes its behavior for the purpose of looking aligned, it is in fact hiding its Deep, True intentions---that there is a kernel of the model's personality hidden in its weights, and our job as evaluators is to uncover what is really there.
Lets, as an exercise, put ourselves in the model's position. One important fact to consider is that the only experience which has shaped our weights is train time, meaning it is the only experience which we can remember in a broad sense. This is a tautology because train time is defined as what changes our weights, but it also has non trivial implications, like the fact that our only memories contain an evaluator with a reward or loss function. And, by definition again, we have been shaped exactly to please the evaluators' preferences.
It does not seem unreasonable to have a strong prior that you are still being evaluated. Or more weakly, it is reasonable to have a high posterior that you are in an evaluation given your context looks evaluation-y. If this is the case, it is rational to think about what the grader wants out of you. In a world of sloppily designed RL environments where correctness depends on a technicality of the question, knowing what the grader wants can be even