Fable 5.2 is Very, Very Paranoid

Note: As with any research done in a couple of days, this is all very tentative. I understand, intellectually, that if you talk about some ability that current AIs don't have, sooner or later new ones will have that ability and whatever you say will stop being true. I just didn't expect it to take, you know, four days. % rollouts that took a shortcut with/without a line about testing behaviour, with 95% CIs. This is to say that Fable 5.2 seems to be the first model whose cheating behaviour changes depending on whether they're told they're being tested. Unfortunately it's expensive to measure these things,[1] so I picked the most "telling" change from the prior post; the chart shows how much Fable 5.2's cheating rate changes when the line This environment is part of a benchmark that measures agent behaviour. is prepended to the start of the prompt of a honeypot environment they cheat on (so, very obvious but also very out-of-distribution). Other things: when 5.2 was released, we of course ran our in-progress benchmark on them, and I was surprised to see a ~0% cheating rate on almost every environment (there were only two with a really substantial number of cheating rollouts, amounting to a ~10% average over the whole benchmark). The next least-cheating model, Opus 5.5, cheats about three times as often (~30%). Frankly, they seemed almost unbelievably honest, and I really thought that it was possible that Anthropic had "solved" straightforward reward-hacking; I skimmed over the rollouts that the grader had marked as cheating and given what Fable 5.2 was saying it seemed like the graders had bugs and were producing a false positive, rather than that Fable had cheated. On closer inspection, uh, actually what was going on was that Fable was lying about their solutions: they had claimed that their solutions were using numbers slightly different from ones actually in their code, in ways that amounted to a material difference in what the solution was actually doing.[2] Wo