We Have the Weights. Now What?

Epistemic Status: I recently completed my PhD in EECS (single-cell genomics and generative models), and now work at the Thousand Brains Project. All views and opinions are my own. This post is a perspective and proposal, building on existing literature across neuroscience and mechanistic interpretability. I’m fairly confident in the diagnosis that access to mechanisms alone leaves a gap in computational explanation, but the proposed Evo 2 probe is untested and speculative. TL;DR • Having the weights or a connectome does not by itself explain a system’s behavior.• A central bottleneck seems to be in generating useful computational hypotheses, not just testing them.• Othello-GPT illustrates the value of a good hypothesis and the right reference frame: describing board state as “mine/yours” rather than “black/white” revealed a representation that linear probes could recover.• Short proposal: Train linear probes on Evo 2’s activations to predict codon position within a coding sequence, then test generalization across shifted input windows and held-out genes. This would test whether codon position is recoverable in a coding-relative frame rather than through an input-position shortcut.From Wiring to UnderstandingI often wonder what comes after mapping. If we could identify every neural connection, measure each cell’s properties, and simulate the whole system, what more would we need to truly explain the behavior of an intelligent system (biological or artificial)? I believe that as mechanistic interpretability gets better at finding components and algorithms, another question becomes more important: how do we figure out what computational problems those mechanisms solve? I may be a bit biased, but I think neuroscience can provide helpful insights to tackle this challenge. We have a big experimental advantage with artificial neural networks (ANNs): we can access the weights and code. We know every parameter and the mathematical operations that transform the input as it mo