Description du poste
Here's a basic argument that focusing on patching reward hacking is misguided and will fail. Argument is as follows:
• Reward hacking is a big problem, causing models to cheat and take unintended actions.• This can generalise broadly, leading to other misbehaviour such as faking alignment and sabotaging safety research.• We don't understand that well when this happens. Sometimes you get emergent misalignment from it, but most of the time (IME), you don't. But other times, this link lies latent until faced with something like an automated grader.• The default course of action will be to:• Patch, which means we find the hack, close the loophole or penalise the behavior in some way, and train again.• Patch+: use a method that comes with some account of why it should work. Inoculation prompting, for example, uses the idea that hacking spreads through what it means to the model. If the model is told during training that hacking is acceptable here, it might not link hacking to misbehaving in general. GRAFT uses the idea that what the model learns from detected hacks can be confined to a set of parameters and deleted after training.• I think it's fair to say that things in the second bucket are more principled than the first, but at bottom still empirical. Both are judged by whether measured hacking, or measured misalignment, went down, and usually it does. What’s still missing, as Edward argues, is an account of what a fix teaches the model, and so an account of whether it will keep working. • Whether something counts as a hack bottoms out in human judgment (Voudouris et al. 2026). Editing the test file is cheating when the task is to fix the code, and it is the task when the job is to fix the tests. Something is a hack because it isn't to taste, or smells wrong. And, training environments come from many vendors who, as 1a3orn points out, don’t agree about what good behavior looks like, so their fixes aren't simply summarised by a natural concept. • So, in the long run, w