Description du poste
Some researchers worry that looping (as in looped transformers) is a particularly scary way to make AI models more capable, because looped models will be harder to monitor than standard transformers of similar capability. We test this worry on NVIDIA’s Dory 4B, a publicly available looped transformer, and find evidence that it is justified.
Result 1. Looping boosts no-CoT reasoning much more than it boosts general capabilities.
Dory can run with a configurable number of loops. It was trained to use up to 4 loops, and adding loops up to 8 at inference time still improves performance. Without looping, Dory’s no-CoT capabilities (measured by NCRI, the No-CoT Reasoning Index) match what a trend fitted to open-source models predicts from how well it predicts text (measured in bits-per-byte). Its general with-CoT reasoning capability (measured by “mini-ECI”, a custom, smaller version of Epoch’s Capabilities Index) is high relative to its NCRI, so Dory sits below the NCRI-vs-mini-ECI trend, but it is similar to the best models of its size. Adding loops improves mini-ECI and bits per byte only slightly but increases NCRI substantially. In fact, if we plot NCRI against total parameters, counting looped parameters once per loop, NCRI rises faster than the trend up to 6 loops. This is concerning, since it suggests adding loops increases no-CoT capabilities much more than general capabilities, resulting in models that will be more difficult to monitor.
Result 2. Dory has near-zero CoT controllability (ability to follow instructions about what its CoT says or how it is formatted) at every loop count we ran, so we learn little about how looping affects controllability.
Remark: Looped transformers can be easily modified to become fully recurrent, as discussed here. We call Dory modified in this way recurrent-Dory, and unmodified Dory normal-Dory.
When running Dory with multiple loops (here 2), the middle layers are applied multiple times to the residual stream (left). During ordi