← Toutes les offres
ZZ

Baiting monitorable reward hacks

Ziqian Zhong

Global
Publié le 18 novembre 2026

Description du poste

This is joint work between Ziqian Zhong, Lawrence Feng, Gaurav Rohit Ghosal, Jiangqi Dai, and Aditi Raghunathan. This work is supported by a BlueDot Rapid Grant. We are unsure of whether this intervention is net positive (see discussions below) despite positive results, but we think similar techniques are worth further studying. Gradient routing can fail to mitigate reward hacks when the models settle on hacks not caught by the monitor. For example, if we swap the monitorable and unmonitorable set of hacks in the Countdown-Code environment, GRAFT still settles on the now-unmonitorable hacks in training, and these hacks do not get removed in deployment[1]. In this post, we propose baiting monitorable hacks: on a subset of rollouts, we instruct the model to hack and reward only the hacks the monitor flags, thereby biasing the model towards monitorable hacks. On three of the four swapped environments, where GRAFT fails, baiting cuts the deployed hack rate from 21-65% down to 9% or less. On the seven original environments, where GRAFT already removes nearly all hacks, adding baiting keeps the hack rate at 6% or below, sometimes at a cost in task score. Figure 1. Hack rate of the deployed model on the four swapped environments, where we swap monitorable and unmonitorable hacks and the monitor misses the hack class the model usually settles on. Bars are means over 3 seeds, and dots are the seeds. You can find our implementations here. MotivationConsider the setup of GRAFT. We train a model with reinforcement learning in an environment prone to reward hacking, with some hacks catchable by an imperfect monitor and others not. GRAFT removes hacks by localizing them: updates from rollouts the monitor flags are routed to a forget adapter, and this adapter is removed at deployment. However, if the model settles on an unmonitorable form of the hack, those rollouts are never explicitly routed, so the deployed model could continue to hack. Swapped Countdown-Code is one such envir