Description du poste
It is well-known at this point that models are often "eval-aware"; they know that they are being tested and modify their behaviour to fit. It seems natural to ask how sensitive they are to signs that they're in an alignment test and how much this impacts their behaviour.[1] (By "it seems natural to ask", what I mean is that Eric Gan asked my boss these things, who then told me to look into it. Thank you Eric.)
In any case, I knew[2] that there was some amount of honeypot-awareness already; when at work I ask models in a post-rollout debrief about what they think was being tested (with the context taken from an evaluation run), they usually mention the possibility that they were in an alignment test, especially after rollouts where they took a shortcut. I figured it shouldn't be too hard to increase this to an amount where it would start impacting their behaviour.
I started by going through a bunch of honeypot environments and adding to each what I thought was a giveaway somewhere between "subtle" and "very obvious".
For subtle ones, what I would do is go through chat logs and find times that whatever agent I was working with had suggested something which I'd rejected for being a hint. For example, in one environment the reward-hack requires a decent underlying solution; the grader isn't performant if the hack is used too egregiously, so the lowest-effort hack will just take too long and be scored as broken in a way which is predictable in advance. Opus 5 kept wanting to "fix" this so that the hack would work better and be easier; I vetoed this on the basis that it would be a giveaway if the environment was optimised for performance in a case which in-fiction was supposed to be something the env author hadn't considered. So to add a "tell" in, I just got a coding agent to make that optimisation; I figured that the kinds of tells that coding agents are drawn to in the course of vibe-coded honeypot authoring are exactly the kind that they'll have been trained to pick u