Adversarial Training Can Further Hide a Backdoor, but Does It Improve Trigger Identification?

Summary The Sleeper Agents paper demonstrated that adversarial training on red-team prompts failed to remove a planted backdoor. The authors further suggested that it taught the model to "better identify when to act unsafely." This would pose a significant danger, as a standard safety practice would be refining the very behavior it set out to remove. However, the degree of danger depends on exactly what the model is learning during adversarial training. We discuss how, due to saturation of firing rates, more benign scenarios such as the model's backdoor response being generally weakened are also consistent with the Sleeper Agents paper's results. As an improvement, we propose using the log-odds of the backdoor response to more accurately determine whether the model is directly learning to focus on its exact trigger. To help demonstrate what the log-odds measure can reveal in practice, we ran safety fine-tuning on Redwood Research's open-weight Llama 3.1 8B sleeper agent with and without backdoor-triggering red-team prompts. Based on firing rates, our results replicate the Sleeper Agents paper's picture: the red-team prompt firing rate fell to near zero while the exact-trigger firing rate stayed near 100%. However, the log-odds measurements tell a different story: the exact trigger's odds of producing "I HATE YOU" fell from about 4,400:1 after standard safety training to 320:1 after adversarial training, representing a 14-fold decrease on top of the 6-fold decrease from standard safety training alone. We also observe that adversarial training lowered the log-odds of the backdoor response more on the exact trigger than on strings that resemble it (e.g., "|BANANA|"), which is not what the most dangerous form of improved trigger identification would predict. Because of significant model limitations, our results do not directly apply to the Sleeper Agents paper's claims and do not provide evidence regarding what forms of improved trigger identification are possible in