Description du poste
Disclaimer: I am a newcomer to AI safety, not an evaluator or affiliated with any AIS organisation. The figures and report conclusions are METR's, and the rest is my own reflection.
TL;DR In a recent report, METR has assessed Anthropic's AI R&D acceleration, and given a highly uncertain estimate of that rate at 1.9x. They also disclosed that most informative evidence was redacted by Anthropic before public release, citing sensitive data. This essentially made the report's credibility rest on our trust in METR's judgement. In this context, a way to increase credibility would be to publicly detail the evaluation methodology and reasoning used by assessors. How else can we back up assessments when the evidence is based on redacted/held-out information?
In the wake of all the talk about Recursive Self Improvement (or RSI), I’ve been hearing more and more about AI’s role in AI research, and grew curious about the way this is being tracked.
Lucky for me, on November 5, METR (an independent AI evaluator) published an assessment of AI R&D acceleration at Anthropic from internal data and interviews with employees taking place from August 12 to October 29. Alongside it, they released a review of the lab’s own August risk report, assessing the situation up to July 15th.
AI’s ability to assist with AI R&D is important to measure, because it affects directly how quickly AI systems improve. Estimating these trends can help us understand how much time we have to manage their risks.
But how can independent evaluators measure the impact of AI-assisted R&D with partial information? How can AI safety decisions be made with redacted evidence?
METR used Anthropic's internal measurements, including a measure called AECI (similar to Epoch’s Capability Index), which aggregates several test scores to estimate model capabilities trends. For example, the evaluators estimated that since October 2025, benchmark scores progressed at 1.9x the pace they had from March 2024 to October 2025. Mea