Description du poste
Research update of a project done as part of MARS 5 and the Meridian Visiting Researcher Programme.
Filler-token reasoning is the ability to use extra tokens (such as a chain of thought) to reason about a task without using the words emitted at those token positions. This is possible because attention can move information between token positions without routing it through the words the LLM emits. This can give a model more parallel, but not more serial, computation.
The canonical demonstration of this is the "Let's Think Dot by Dot" paper (Pfau, Merrill & Bowman, 2024). There, the authors train models to solve completely parallelizable (TC⁰) tasks with chains of thought made only of "." tokens.
Our contributions:
• We change Pfau et al.'s training method so that models can learn multi-step reasoning on filler tokens.• We demonstrate this on two models and three out of seven multi-step reasoning tasks.• We provide a theoretical analysis of how many layers a transformer needs to solve each of our multi-step reasoning tasks, depending on the length of the task.• We measure empirically how many reasoning steps we can train Qwen3.6-35B-A3B and gpt-oss-20b to do on filler tokens for each task.Overall, this project has informed which side tasks we use to study steganographic reasoning. We also hope this writeup shows that filler-token reasoning is not confined to fully parallelizable tasks.
MotivationVisualizing different reasoning modes. Open reasoning: each intermediate result is written out as a token and read back in at the next position, so the reasoning flows through the visible words. Steganographic reasoning: the reasoning still flows through the words, but hidden in innocuous-looking text. Filler-token reasoning: the words are filler; the reasoning flows between positions inside the network and only comes out as the final answer.
Using the same token over and over guarantees that the filler tokens carry no information about the reasoning. But you could also imagi