Tracing Subliminal Learning Through Pretraining

This post summarizes our paper, Tracing Subliminal Learning Through Pretraining, which was developed as part of EleutherAI’s SOAR program.  TL;DRWe study subliminal learning, the phenomenon in which a student model acquires a teacher’s preference through training data that appear semantically unrelated to that preference. Our work investigates subliminal learning through cross-model distillation, using the PolyPythia and DataDecide OLMo models to show that a teacher’s animal preference transfers to students with different weight initializations, pretraining data orders, training corpora, model sizes, and architectures.  We examine two distillation settings: hard-label number distillation, the classic setting where the student trains on number sequences generated by the teacher, and soft-label on-policy distillation, where the student generates text and learns from the teacher’s next-token probability distributions on that text.  Our main findings are:  • Shared initialization and pretraining conditions are not required. Cross-model transfer can survive differences in initialization, data order, pretraining corpus, model size, and model architecture between teacher and student. • Transmitting and acquiring a preference are distinct abilities that can develop separately during pretraining. When distilling across checkpoints within a pretraining trajectory, we observe that some checkpoints are effective teachers but weaker students, and vice versa. Consistent with this distinction, cross-model transfer can be asymmetric when teacher and student roles are reversed.• Number frequencies alone can carry much of the preference signal. We replace the teacher’s number sequences with synthetic ones that match its number frequencies but whose numbers are otherwise independent of the prompt and of preceding numbers. Much of the hard-label transfer survives this replacement in Pythia, and a smaller share in OLMo.• Restricting the loss to top-ranked tokens can increase soft-label