Description du poste
Yesterday, Goodfire released the companion technical report to Nick Cammarata’s viral interpretability essay "Neural Networks Are Already Clean", but it seems to have gotten buried in all the recent crazy AI news.
The technical report is titled: "Neurons as an Interpretable Basis". Goodfire's Twitter announcement for it can be found here.
Like Nick Cammarata's essay, it's full of inventive, oft-interactive data displays. Don't just feed this to an AI agent and assume they can grok the post by pulling a PDF.
Quoting from the Abstract: (emphasis mine)
We study the neurons of Qwen3.6-35B-A3B using sparsification methods, attribution tools, and research agents, and find three properties that make neurons interpretable units of computation:
First, they are already functionally sparse. Ablating the 92.2% weakest MoE neuron activations for each forward pass has only modest effects on model capabilities – a 2.9% increase in language model loss.
Second, neurons have relatively few semantic groupings (modes) of computational functions, and we can identify these and describe them concisely. The majority of neurons have either one mode only or a single mode that is overwhelmingly causally significant. In cases where multiple causal modes are present, the number is small and legible to our methods.
Finally, we can use agent-driven experimentation to generate neuron descriptions in natural language, and these descriptions allow us to predict, simulate, and activate the neuron. Providing the description to a judge model allows us to (1) predict whether the neuron will fire in a given passage of text with 95% accuracy vs random text; (2) simulate its token-by-token behavior in high-activation regimes sufficiently to recover 73% of the damage (as a proportion of a ceiling of damage restorable) from the neuron’s ablation; and (3) write text that causes the neurons to fire more strongly than anything observed in a 10B-token corpus.
Overall, we argue that neurons are meaningful units