Description du poste
Natural language autoencoders are pretty interesting; I'm a big fan of using meta-models to interpret what's going on inside LLMs and NLAs seem like a promising route to understanding them better.
One thing I've been interested in for a while is using NLAs to interpret the differences between activations: given an activation from an earlier and later layer, can we verbalize the changes between those two layers? It seems like it would be pretty cool if we could verbalize the activation diff between each layer in a transformer, since it would let us better understand how the model does computations.
Unfortunately training an activation-diff-NLA that actually works well is pretty hard[1], so I've done something easier: create an NLA that can interpret the outputs of the attention heads of a single layer (L24) of Qwen3-8B. It, uh, kinda works? I can get explanations that are (by the way the NLA is trained) text describing the attention head outputs! It's not clear to me how useful these explanations are though, and my NLA only works well for some attention heads.
Here's a widget where you can look at some example outputs from my NLA:
This is pretty early work, and I'll probably make better versions of this kind of thing in the future! I'm not really sure how good head NLAs are because I'm not sure how to evaluate them well. Please keep in mind that I've probably made some mistakes here, and you should generally take any results here with a grain of salt.
NLA recap[You can skip this section if you're already familiar with NLAs.]
A traditional natural language autoencoder (NLA) is two models that are fine-tuned from the model we want to interpret: an activation verbalizer (AV) that takes in an activation at a specific layer as input and outputs text, and an activation reconstructor (AR) that takes that explanation as input and outputs an activation (specifically the AR is truncated at the layer the activation is from[2]).
First the AV and AR are SFTed on a bunch of Claude