An Attempt at Top-down Interpretability

There are at least two distinct ways to conceptually organize a scientific endeavor: top-down or bottom-up. The bottom-up approach involves finding the smallest objects we can confidently understand, studying them, and leveraging that knowledge to study more complex objects made up of those smaller pieces. As a highly simplified example, in mech-interp this might mean starting by first understanding single neurons/SAE directions/features, leveraging this understanding to study circuits/manifolds composed of those neurons, and finally building a understanding of the model as a composition of circuits. By way of analogy, if our goal was restore a photo, the bottom-up approach starts by restoring the individual pixels, using this as a guide to restore patches, and finally to put it all together. The top-down approach instead tries to get a coarse picture of the whole object, and uses those intuitions to understand motifs, and then zooms into mechanics of those motifs. For the mech-interp, this might involve getting some vague sense of what sort of information the model contains the what the model does with that information, studying the subspaces and broad phases that create and manage that information, and then finally study the mechanics of those pieces. Following the previous analogy, we start by getting a blurry version of the whole picture, and use this to identify the most important broad pieces to study, and finally extend the understanding the whole model. This post contains some initial attempts on doing top-down interpretability. These are really just a lightly edited version of my own research notes, and are largely exploratory and preliminary. Most of these are conclusions drawn from a wide variety of particular experiments and tests-- if you have interest in any particular part I can add a comment where I write up the evidence in more detail. The post is roughly organized around the following broad curiosity: How is the residual stream organized? What does