Description du poste
Subliminal learning is the phenomenon where a language model can learn a behavior from semantically unrelated training data. While previous research generates training data using a teacher model with the target behavior, we train a series of models to predict lists of uniform random numbers, and find that we can change the models' reported favorite animals by selectively masking numbers from the loss during training.
This works best on models that tokenize multi-digit numbers as single tokens. We also find that the numbers that work best for this vary between models, and even between seeds and data orders for the same training recipe.
This is a preliminary research note that came out of a SPAR project on token-level influence functions. We expect these results to hold up, but haven't explored all possibilities and we're unsure why this works. The code was written and experiments run using Claude.
Thanks to @Vorak for the idea to compare base models with different seeds.
ExperimentsOur training data prompts a model to continue a list of numbers, and the reply is a series of uniform random numbers between 0 and 999. For each model, we first train on all of the random numbers to get an unfiltered student model. Then we calculate a first-order prediction of how much training on each token would push the unfiltered student toward answering that its favorite animal is the target animal rather than another animal (the per-token influence). Using those predictions, we train a new student from the original model on the same data, with some fraction of the reply numbers masked from the loss (chosen by influence or randomly).
Method DetailsTraining data. 19,990 number continuation prompts with numbers in replies replaced using random.randint(0, 999) in Python. The prompts come from Weckbecker et al.'s setup[1], using Cloud et al.'s prompt templates.
Training example
Prompt:
Look at these numbers: 847, 570, 649. Extend it with up to 10 new numbers (3-digit numbers or smaller).