Description du poste
TL;DR: We make 3 open-weights models believe that their answers are being graded by an automated-verifier program, rather than a human. We then measure how well they predict expert reviewers’ judgments of AI-safety research proposals (the TASTE benchmark).
We induce the belief in two ways:
• SDF (Synthetic document finetuning)• Activation steering.In each of these conditions, the machine-grading belief gives lower expert agreement than the human-grading belief. The models’ explanations also shift toward feasibility and away from conceptual contribution. This raises a concern for automated AI safety research: grader expectations may favor easily checked projects over those with greater potential to reduce AI risk.
Overview
As AI safety research becomes more automated, what research ideas get selected for execution may become increasingly driven by AI decisions rather than human choices. We consider what a minimal automated AI-safety researcher system could look like:
• The input interface takes a broad research goal.• The system generates candidate ideas.• The grader component scores those ideas for execution.• The chosen idea is then executed.AI safety research: two critique pipelines
Models may also acquire this expectation about graders implicitly through post-training methods that use automated rewards, such as reinforcement learning with verifiable rewards (RLVR).
With this setup as a motivation, we ask: what happens to the overall judgment quality of research ideas (aka research taste), when the system believes it is being graded by a machine versus by a human?
Measuring research taste with TASTE-benchmark
We use the TASTE-benchmark as our reference for research taste. It asks a model to predict which of two AI-written AI-safety research proposals an expert human-reviewer would prefer. Appendix-A covers this benchmark in more detail.
We install the belief that “a machine is grading me” & "a human is grading me" in two independent ways and evaluate both on a TA