Understanding the Linear Representation Hypothesis

This post draws heavily from "A Survey on the Linear Representation Hypothesis", and "Do Not Rely on the Linear Representation Hypothesis". I really enjoyed reading them and would recommend you read them too. If you must pick one, Do Not Rely is shorter and has some fun results, although I am a bit sceptical. I tried to summarise and synthesise the two, primarily for my own learning. I'd appreciate feedback, and will update this post if I have made mistakes or read something which changes my understanding. what is the linear representation hypothesis?Matrix multiplication is one of the fundamental building blocks of neural networks. It's really easy to stretch and rotate specific directions with matmuls, so it seems like models might use linear representations. Briefly and informally, the linear representation hypothesis (LRH) posits that neural networks represent "concepts" with 1D subspaces - directions - of their activation space. There are many papers which show evidence that lots of models do something like this for some features at some layers, which I won't discuss in detail here. We will take it for granted that steering vectors and linear probes, to some extent, Just Work. This is somewhat unsatisfactory, though, and leaves me with a number of questions: • What, precisely, are concepts?• Why should we believe the model learns any coherent concepts at all instead of a bunch of inscrutable, spurious correlations and rules-of-thumb?• Are all concepts represented as subspaces?• So what? Even if we knew that all concepts are directions, what can we do with this knowledge?So, we should be a bit more specific. We'll introduce a number of linear representation hypotheses, which provide a more useful frame for discussion. what are the linear representation hypotheses?First, weakness vs. strength, as introduced by Lewis Smith: • "Weak" means that our hypothesis only applies to some concepts that the model represents.• "Strong" means that it applies to (almost) all of