Towards LoRA science

Attribution: Nesiacel planned and executed all experiments. Analysis of the experiments was shared between the authors. Thanks to Clement Dumas for his input while we did this research. TLDR;1)We have developed a framework for reverse engineering LoRAs trained on MLP downprojection. We posit that it's extendable to all LoRAs and welcome further research. 2) We reverse engineered a simple case of backdoored Weird Generalization 3) In this post, we discuss the four experiments that developed this framework 4) The Weird Generalization LoRAs that we studied work by connecting neurons which activate on general question framings with neurons that write the weird outputs into the residual stream. Pre-IntroductionHow does a LoRA work?A LoRA is a set of matrices that is added to the frozen weights of a model. Gradient descent is then done and the weights of the matrix are optimised to reduce the loss. We then do gradient descent and optimise the matrix weights to reduce loss. In an MLP, a LoRA fit to one layer will change how the neurons of that layer pay attention to the neurons of the layer above them. For example, here's an example MLP setup before a LoRA is applied. It's made of three neurons and reads from a hidden layer of nine neurons. It has a loss of 0.78 on an example task. To reach loss of 0, each of the three neurons has to read from different neurons in the layer above. Figure 1 - an illustration of an MLP before a LoRA is fit When the LoRA is applied, it's initially randomly initialised. There are two matrices of equal size ('rank'). The A matrix is multiplied by the outputs of the hidden neurons. The B matrix is then multiplied by the A matrix and added to the 3 output neurons. In this case, because the rank is 2, matrix A has two columns. If it was rank 1, matrix A would only have one column. Figure 2 - an illustration of an MLP fit with a randomly initialised LoRA. Gradient descent is then carried out with the base weights of the neurons frozen.