Description du poste
Authors: Keshav Shenoy, Anthony Zang, Eric Gan, Jacqueline Tay, Amy Ngo, Sebastian Prasanna, Julian Stastny, Vivek Hebbar
tl;dr. We had a human expert secretly sabotage AI safety research and asked auditors to spot it. Even with the full research log and the ability to run their own experiments, human auditors could barely tell sabotaged research from benign research, and usually couldn't pin down what had been sabotaged. The best frontier models likely do better, but still miss most of the sabotage.
📄 Paper | 🐦 Twitter
IntroductionAs AI systems automate more of AI R&D, a misaligned AI could covertly sabotage AI safety research. This sabotage could take many forms, including making poor methodological choices, introducing subtle bugs, omitting important results, or drawing conclusions the evidence doesn't support. Prior work has studied sabotage in controlled settings (Libon et al., 2026; Gan et al., 2026), but these settings constrain the saboteur in unrealistic ways, such as pre-specifying its objective, narrowly scoping its task, or limiting it to low-level changes. Much less is known about research sabotage when an adversary is free to choose its own strategy across the full research process.
To study this, we ran a red-team/blue-team game. One of us played a “human-expert-level” AI saboteur, running new AI safety projects from scratch. When sabotaging, the saboteur already knew the benign results, could run the project however they desired, and was required to substantially change the project's topline conclusion. This produced 22 traces (full logs of each research run): 10 of which were benign and 12 sabotaged. The sabotaged traces contained 33 specific sabotages (‘sabotage mechanisms’), 17 of them ‘major’ (enough on their own to substantially change the project's conclusion).
To mirror a realistic auditing setup, auditors got 90 minutes per trace, the full research log, AI-generated audits, access to frontier AI models, and unlimited compute to run their ow