Description du poste
In the current safety landscape, researchers are encouraged to use as many agents as possible, even in “fleets,” where a large number of them go out and run experiments while you, uh... wait for them to come back? I've spent the last few months of the Anthropic Fellows Program trying this out. My first attempts were naive, but after a few iterations I found an approach that works well for me.
Research is a double loopI want to propose a mental model of research as a double loop, an idea that builds on double-loop learning (Argyris, 1977).
The outer loop is reserved for the inputs we currently refer to as "human taste": choosing which questions are worth answering, designing a trustworthy measure, and deciding which direction to go. Agents can help with this, but they’re still bad at making these decisions autonomously.
The inner loop turns each idea you test (an arm as claudius likes to say) into a comparable result. An example of a very simple inner loop might be:
• Write a system prompt.• Run the model with that prompt on a set of test questions.• Score the model's replies.• Compare the scores against a control with no system prompt.• Plot results.Agents are really good at building the infrastructure for this loop!
Ideally, since agents are good at the inner loop part, we want to hand it off to them... However, before you can do this, you need to operationalize it. You need to work very closely to define the inputs so that you can freeze everything except the one input you’re testing (e.g. a system prompt). The benefit of this is that each arm fits in one small file that you can look at and compare easily, and running a new arm becomes running one command. I call research “fleetable” to the degree you can do this and check the results quickly.
What happens when you don’t operationalize The naive way to use fleets is to hand agents the experiment and wait.
Let's look at an example. I asked five Claude Code agents to test whether a system prompt could reduce Claude