Description du poste
Note: I only test on Gemma-4-31B, which contains the spaces I've been studying in this earlier post.
This is a shorter post, with just a few observations potentially relevant to those working on the J-lens work. The main observations here are:
• In Gemma-4, there are two relatively small subspaces that account the majority of the J-lens readout. These are both coordinate-aligned subspaces. That means we can take a hidden state, mask out all coordinates except both these ~600 coordinate groups, and it will capture ~80-95% of the J-lens readout.• These spaces are not built to capture J-lens contribution. Both spaces are defined by a distinctive pattern in the weights: the post-norm diagonal is very small on these coordinates. That is, the space is protected from edits for many consecutive layers.• Nonetheless, semantically valuable information often still exists outside these spaces, complicating an overly clean isolated 'workspace' story.Brief ReviewThis is a short review of the information from this earlier post. You can skip this section if you've already read the last post.
In the last post, we saw that the residual stream of Gemma is organized, and contains a few coordinate-aligned subspaces. Gemma has an extremely anisotropic residual stream representation, which bifurcates into two main scales between the layers ~L20 to L40. The two subspaces are coordinate aligned-- there are roughly 600 coordinates that live in the small scale (which I'll call the Quiet600 for their small scale), and Outside4723 for the remaining coordinates (minus Norm53, which are 53 coordinates that have a large norm contributions).
Tentatively, the post suggests Quiet600 can be further divided into two other coordinate aligned spaces. Persistent279 is the 'core' of the space, containing the coordinates that undergo virtually no change between L20 to L40. This space is presumed to carry information more closely derived from the token itself-- for instance, linguistic features like the part