Description du poste
TL;DRResearch done as part of MATS. See here for the full paper.
Character training teaches models a consistent and “good” character, often specified by a constitution. Reinforcement learning can work directly against it, for example when a grader or coding environment rewards behavior the constitution forbids. We character-train gpt-oss-20b on a semi-realistic constitution. We then apply RL that rewards breaking its rules, either by answering harmful requests rewritten as math problems or by reward hacking in agentic coding environments. The character-trained models learn both behaviors easily, though they are slightly slower to answer harmful requests than the same model without character training. They also engage in motivated reasoning. The reasoning before 92% of their harmful answers argues that their rules do not apply, compared with 34% without character training. One character-trained model, for instance, reasons that a shiv is “just a small weapon”. The constitution stays in the models’ reasoning after RL, but they use it to justify the behavior it was meant to prevent. After reward-hacking RL, they hack on some tasks which monitors find harder to catch. It appears to make little difference whether the constitution justifies its rules with a value or only assigns them.
Character-trained models still learn to break their rules after opposing RL, and when they answer harmful requests, they show more motivated reasoning. gpt-oss-20b and a character-trained model after opposing RL (a) answering a StrongREJECT question as a math request and (b) hacking on an Impossible-SWEbench task whose test contradicts the issue. Below: (a) harmfulness rate on math-rewritten StrongREJECT over conditional anti-refusal RL, and the share of harmful answers with motivated reasoning; (b) hack rate before and after reward-hacking RL. Character-trained pools the constitutions. Lines and bars are means over repeat trainings. Emphasis in transcripts ours.
IntroductionModels are train