Description du poste
Takehi et al. report that GPT-6-Astra can intuit the functions of neural networks by performing static analysis of the weights. I argue that, for at least two of these experiments, this is much more a feat of numerical gestalt than of mechanistic understanding.
In the first experiment, Astra is given the weights of 4 identically-sized neural networks, each trained to play a different basic video game. It is asked to identify, without running the network or using any tools, the game being played, as well as make some observations about the network itself, how the agent in the game gains reward, etc. The prompt for this task explicitly names the 5 input channels as agent, moving objects, reward targets, obstacles, and previous frame.
I agree that, given the literal task as described above, a human would not be able to identify the game from the raw floats. But humans are inherently terrible at parsing long lists of numbers; we invented a discipline called Data Visualization long ago to address this shortcoming. So, if we allow the absolute most basic Dataviz trick we can think of - averaging the first-layer weights across each channel and plotting them on a heatmap - the task suddenly becomes much easier, arguably trivial for anyone with at least some familiarity with the sorts of games often used to train toy neural networks. Try and guess:
I haven't included the obstacle channel, because under the naive normalisation it sort of just looks like noise. Principal component analysis recovers an outline of the game board in each case, but that's straying a bit far from the already slightly hyperbolic title of this post, and besides, it's unnecessary.
The gif below serves both to reveal the answers and to illustrate how trivially 'interpretable' these networks are.
For the second experiment, Astra is given a denoising diffusion model that produces 12x12px grayscale images, and some information about the size and shape of the network. Under the same setup (weights only