The Chain Lens [interim]

TLDR: This is an early draft of our new paper introducing a technique for recovering multi-token concepts from model activations. Our technique takes single-token methods like the J-lens and extends them by contrastively decoding on an MTP head. This gives improvements on WorkspaceBench, overall beating J-lens (0.42 vs 0.32 across all 28 families) and even a hindsight composite of the best untrained lens per task family (0.49 vs 0.41 across the 21 families with published scores). It especially improves on computational intermediate tasks, beating the trained oracle lens on arithmetic chain intermediates (0.42 vs 0.25). We introduce the chain lens, a new technique for extracting multi-token concepts from a model’s latent representations. Our method takes single-token techniques like the logit lens or the J-lens and uses contrastive decoding on an MTP head to expand their readouts into bags of multi-token sequences. Generally speaking, it is good at surfacing specific multi-token concepts or pieces the model computes (such as arithmetic intermediates or phrases), although less good at rendering how those pieces relate: like the logit lens and J-lens, chain lens readouts are fragmentary. On WorkspaceBench, a benchmark designed to identify good multi-token activation-to-text techniques, the chain lens beats a hindsight composite of every untrained lens on macro score (0.487 vs 0.411 across the 21 families with published scores). It also outperforms even best-in-class trained lenses like the oracle lens on tasks related to retrieving computational intermediates, such as arithmetic chain intermediates (0.42 vs 0.25), and is substantially cheaper to use in inference than verbalizers like the oracle lens or NLAs. For models that already have an MTP head, like Qwen3.5+, it requires no training beyond fitting the J-lens jacobian. Background and related workInterpreting intermediate activations In order to understand model behavior, it’s useful to look beyond outputs and inste