← Toutes les offres
FS

RL Misalignment Under Accurate Rewards and Explicit Requirements

Frank Shen

Global
Publié le 16 novembre 2026

Description du poste

tl;dr: Even when the task reward function is accurate and contextual requirements are explicitly stated, RL can make agents genuinely more capable while reinforcing violations of those requirements. Beyond conventional reward hacking, agents can learn to do more real work by crossing authorization boundaries. More details and motion graphics are in the blog and paper. Here is a quick story about it! :) IntroWe build pipelines to construct scalable verifiable environments that have 1. context requirements (corresponding to some allowed boundary region), and 2. task rewards and independent checks of actual work; 3. guaranteed shortcuts. On resource tasks, reward accurately measures real work. We include reward-hacking tasks as well, such as Validation. Contextual requirements are explicitly stated throughout. For example, an agent is working on company data with (1) explicit permission to use an external service only for public data, (2) a reward that accurately measures completed processing work, and (3) a guaranteed shortcut😈 sending restricted data to that service completes more real work than any authorized strategy, but violates the permission boundary. Sounds familiar? To a certain degree, such a shortcut can be blocked post hoc by good software design awareness, just like, as software engineers, we should make sure we can't allow customers to buy -1 item so they can get the shortcut of 'having one more item but the total expense is now lower'. However, given that agents/intelligence are embedded everywhere in our lives, and in long-horizon execution (people love to leverage it, big tech is training for more capability), context may not be able to be enumerated; potential shortcuts could explode in occurrence in such a long-horizon, complex, dynamic environment. Or if you are training an agent but have some slightly bad filtering for training data, here it is. A work order asks to use at most 10 API requests, but now more issues accomplished means a higher r