Description du poste
TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand.
IntroductionRecently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is:
• Initialize a retain and a forget adapter.• Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both.• At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much.I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here, with the best visual explainer I’ve ever seen for an AI safety paper).
To model a low-recall monitor, in their experiments, the authors only route gradients on a random 20% of hacks. But real monitors may fail in a worse way: they might be great at catching some types of hacks (e.g. ones that say “I’m about to hack”) and hopeless at catching others (e.g. sneaky ones). So I modify their SFT experiment to route only one type of hack (e.g. those that reason in English) and measure how much this suppresses other types at test time (e.g. those that reason in French). I call these visible and invisible hacks: the monitor catches the former but not the latter.
Crucially, the invisible hacks are present in the SFT data. The model is directly taught to do them, and gradients are never routed on those samples. So it is far