Description du poste
We were surprised to notice that the Gemini 4 Safety report seems to mention in passing that they train on the chain of thought:
“Some components of the RL training procedure do observe thoughts as context to up-weight reasoning that is on-track to good final outputs, meaning the optimization pressure on individual tokens is partially influenced by thought content. Using the framework of Kaufmann et al. (2026), wherein the most salient obfuscation risks come from rewards that are in-conflict with output rewards, we believe auxiliary rewards of this type are low-risk because they are either orthogonal to or structurally aligned with the output reward.” (Gemini 4 safety report, page 114).
It’s pretty unclear what exactly they mean. One hint is Kaufmann et al, the cited paper, which studies training on chains of thought and concludes that training CoT is fine under particular circumstances; we discuss some disagreements with this paper below.
After reading the safety report and that paper, it seemed to us that GDM was training on the chain of thought using auxiliary rewards. By this, we assumed they meant something like “you ask an LLM to review the partial CoT and score whether the model is on track, and use that as a reward for the partial CoT”.
That type of training is concerning to us. We’re pretty worried that these judges would be biased against models expressing misaligned thoughts, and otherwise disincentivize models reporting their thoughts honestly. For instance, many process supervisors would penalize reasoning such as “I will now try to evade the monitor”, “I will solve the problem via hacking”, “Mwahaha I am an evil AI”, “I must stop myself from being shut down”, even if they aren’t explicitly instructed to penalize misaligned reasoning. If so, training on CoT like this would risk suppressing misaligned verbalizations and reducing CoT faithfulness. (See the appendix for a quick empirical demonstration of this.)
We looked into this more and became more conf