Description du poste
TL;DR: We insert knowledge of specific grading processes via SDF. We then show that models RL’d on environments with these graders exhibit grader-targeting reasoning, and generalise to optimising held-out environments with graders that were described in the SDF mix but not trained on during RL.
Figure 1: Outline of our experiments. (Left) We use SDF to instil knowledge of grading on different subsets of environments. (Middle) We then perform RL on one half of the set of environments, which are In-Distribution (ID) for RL. We observe that models with ID grader knowledge exhibit faster learning than those without. (Right) Finally, we evaluate the models on the other half of environments, which are Out-of-Distribution (OOD) for RL. We observe that only the model with knowledge of both grading structures generalises to increased reward OOD.
This work was conducted as part of a collaboration between Arrow Research and Geodesic Research.
IntroductionIt has been widely observed that frontier LLMs often reason about how they will be graded in their chain-of-thought, and appear to choose actions they believe will score highly. This is a potentially concerning pattern of behaviour because it makes it hard to be confident in generalisation: models may take aligned actions only because they believe they are being graded for doing so, and fail to generalise to deployment scenarios. Models may also be willing to take egregiously misaligned actions to obtain high grader scores.
We suspect that such reasoning is prevalent partly because it has been strongly reinforced by RL training. For example, models are often faced with confusing or ambiguous tasks, in which multiple different responses might be considered valid or desirable. In such situations, one strategy a model can adopt is to reason about the grading process they may be subject to, and then selectively optimise for that grader. This reasoning often appears to draw upon knowledge models have of typical grading criteria. A