Takes on “Already Clean”

This is a post containing reflections on Goodfire's recent release, "Neural Networks Are Already Clean", which argues neuron-based interpretability may be easier than we think. The essay is a summary of a technical paper that will come out later this year. Also featuring thoughts on where this research will lead, plus speculative ideas about neuron monosemanticity (What I base views expressed in this post on: earlier this year, I worked on neuron-based interpretability for LLMs (we recently published a paper). We mostly looked at low-level properties of the neuron basis and didn’t do many case studies or scale up automated circuit interpretation. I also thought about the problem for a long time and have some hypotheses based on results we’ve seen that are likely not strictly true.) I’ll first discuss what I see as the implications of this work for LLM interpretability research and then talk about why neurons may be a useful basis: I wrote down some guesses (in a collapsible section so as to not distract from the main point) and then some directions that may be useful to study for basic science. In short, I think the agenda presented in the essay is promising and worth pursuing even outside of direct applications to language model interpretability. I have some object-level uncertainty about results for LLMs and will wait for the final paper and followup work. How useful is this?What I see as the fundamental claim here is that resolving some unforced errors in visualizing neuron activation -- • properly thresholding neurons• considering both poles of neurons separately• taking into account the fact that neurons often activate on seemingly unrelated inputs (for example, by clustering neuron activations as in the paper)-- and then constructing circuits with neurons as features leads to the best method we have for understanding language model computation at a low level. With this kind of processing, you get an “overlay” for the model, which doesn’t explain how the model