Description du poste
The fact that current LLMs are effectively black boxes is arguably the biggest obstacle on the road to alignment. There have been many proposals for mechanistic interpretability and other ideas to pull back the veil on the internal workings of LLMs, but not enough attention has been given to the possibility of translating weights into actual code.
The gargantuan nature of the task puts it obviously beyond the reach of unassisted humans, but it seems plausible to me that a sufficiently capable LLM, given access to its own weights and internal activations, could reverse-engineer its learned computations and turn them into conventional, human-readable source code.
That would involve discovering the algorithms, procedures and representations encoded in its weights in order to reconstruct them as actual code. Depending on how far this can be taken, it could replicate most or all of the original neural network.
Given that current frontier models are already excellent coders (second only to their ability in maths), I think it's at least worth a try.
FeasibilityA neural network can be described as a computational structure. It pairs a certain architecture, which defines operations, with a set of weights that determine how those operations work out in practice. After that, we train them in such a way that the interactions of its parameters yield emergent capabilities.
There's no cosmic law that says that computations by a neural network can't be reproduced by a system other than a neural network. As far as I understand, the challenging bit isn't replicating its calculations, but gaining an adequate understanding of their higher-level structure.
However, there has been promising research in this direction. In "Learning Transformer Programs", Friedman, Wettig and Chen proved that specially designed, trainable transformers could be turned into human-readable Python programs. While it didn't involve the decompilation of an arbitrary pretrained LLM, it served as a proof of concep