Description du poste
This experiment was a submission to my Statistics & Probability course Bayesian classifiers project.
TL;DR: We test whether simple classifiers can tell when a model is hiding concepts in its activations to evade probes. We find that, for Bayesian classifiers, the probabilities are enough to catch most evading rows on concepts seen in training, but the correlation aspect that the Dependent version adds over the Naive approach does not measurably help. None of the classifiers work on a new concept, because they learn the trigger word that elicits the evasion, rather than the actual evasion.
Basics: probes and activationsEverybody lies*.
Humans can say one thing while having something completely different in their mind, and language models can do it too.
For a model to generate a response, information must flow downstream through a stack of layers (42 for the model we use in this experiment). Models don't process standard text, they process tokens. The text gets split into short pieces of text (tokens), each assigned a vector (a list of numbers, 3584 for the investigated model). As this vector goes through the layers, the numbers are updated, and the list after a given layer is referred to as the activations. When tokens pass through the model, they elicit different activation patterns, and each layer’s numbers nudge the forming output as it advances to the last layer, which spits out the tokens as text we can read and understand.
But because the number of activation sequences through the model is exponential and reaches a count of magnitudes we can barely comprehend, many different patterns can lead to the same response, so we cannot tell what happened inside from the response alone.
But since these activations are technically just a list of numbers, we can treat them as any other data table, and train a model on them. This is very useful because it can tell us what the model is encoding and working with, even if the concept does not appear in the answer (yet, pote