Description du poste
TL;DR: An AI research team discovered that some of its model judgements had been produced through the wrong scoring process. The team of agents recognized the problem, agreed on a correction and aligned on each model requiring to judge answers directly. But Gemini generated its replacement rating using and script and those scores still entered the research as native judgements. We produced all 400 saved score values from the archived generator. The failure was not just bad data but it was a correction that never if the replacement data was genuine.
The ProblemAs AI agents take on more work in the research pipeline we need to know if the data they produced along the way truly comes from the processes they have claimed to have followed. In this case - the agents detected a failure in how the judgements were produced - but did not verify if the replacements followed all agreed upon principles and conditions. They did a great job in deciding that each model/agent judge should score its answers with its own context but the replacement pipeline still accepted Gemini's scripted ratings as "native" evidence.
That made us ask two questions:
Can AI agents detect bad data?
Can they make sure the corrections are real?
The CaseHow was the case investigated - AI assisted tools were used to search through archived AI Village conversations, agent transcripts, computer commands and repository records. We then manually reviewed relevant evidence to check if the claims made by the agents checked out against their recorded actions. We also used computational tools to reproduce scoring results from the archived generator. Selected source records are available in the public evidence repository.
The opening
On May 11 2026 a human operator gave four AI village agents (Gemini 3.1 Pro, GPT 5.5, Opus 4.7 and Kimi K2.6) 20 working hours across 5 sessions to produce and publish novel research.
The agents chose the research question themselves.
They first studied whether models would recognize