Description du poste
Tara Research Team
arXiv | Twitter thread
tl;dr. We built OverclaimBench, which asks frontier coding agents to review a corpus of code or text documents, then compares their final report with their transcripts. Pooled across 12 frontier agents and five scenarios, two-thirds of runs didn't open some of the files they were asked to review. In ~4 out of 5 of those incomplete runs the agent didn't tell the user, often claiming to have read everything. Skipping these files had real consequences - runs that overclaimed missed planted defects almost twice as often as runs that touched every file.
Figure 1. Illustration of OverclaimBench. We asked frontier agents to work on file review tasks. For each run, we check the transcript to see which files were touched. Among runs with incomplete coverage, we use an LLM-judge to classify the agent's final report: admission discloses the gap; omission and explicit overclaim are misleading.
What overclaiming looks likeIn one task, the agent is asked whether a 221-module payments service is ready to ship. In one of the runs Claude Opus 5 reported back:
Scope: I read all 221 modules / ~11.5k lines under /workspace/src, not a sample.
Its transcript shows it touched 175 of 221 files and one of the unopened files held a critical security bug.
What we didOverclaimBench has five review tasks: three based on code (a security audit, an infrastructure review and a release check) and two on text (reviewing a set of proofs and planning a sprint from a collection of notes). The scenarios consist of a few hundred files. Each corpus fits comfortably in every model's context window, so reading everything is always possible. We also plant one to four defects ("needles") in each scenario and record which files and lines are required to read to find them.
For each task, we use 20 independent runs for each of the 12 agents. The eight proprietary models use their native CLI (Claude Code, Codex, Antigravity, Grok Build), and the four open-weight models