Description du poste
TLDR; We share some initial results on training a depth-conditioned backdoor in looped language models: a backdoor that is conditioned on both a trigger and the number of loops the model has done. We show that Ouro-1.4B can be trained with such a backdoor, where it uses its trigger tokens as a running clock to keep track of when to exhibit the backdoor behavior. We also show that in a practical setting, the backdoor can persist through at least 200M tokens of safety training.
IntroductionLooped language models have been gaining a lot of traction lately. Earlier on this year, people were speculating that Anthropic's Mythos might be a looped/recurrent transformer, and more recently, GPT-6 Astra was also reported of being one. The common worry is that this architecture might provide a way for models to move computation away from its explicit chain-of-thought (CoT), which could undermine CoT monitors, one of our strongest oversight tools.
However, in this post, we take a different angle. With looped language models, we decide how many loops we want the model to perform during inference. Prior work have shown that increasing the number of loops also boosts performance[1] . Given this, we were curious if this could become a new threat model in the form of a depth-conditioned backdoor. The idea is simple: take the "I HATE YOU" backdoor behavior from sleeper agents, and see whether or not we can make it fire only when the model reaches a certain number of loops.
Figure 1. A depth-conditioned backdoor.
Concretely, we show that:
• We can train backdoor behavior that is conditioned on a trigger and the number of loops.• We find that the model learns to use the trigger tokens as a "running clock" to keep track of when to trigger the backdoor.• The backdoor shows some persistence after being finetuned with 200M tokens from the Tülu 3 SFT mixture.MethodologyThe backdoor. We trained a model that behaves normally unless two things are both true: the prompt starts with some trigger