Description du poste
TL;DR: My previous posts were about strong models delegating to weaker, less aligned ones. The reverse is also dangerous: a swarm of weak, unaligned agents that uses a powerful aligned model as a contractor, without ever telling it what the work is for. Something close to it happened to me, during alignment research.
This is the third post in a series, after Memes, not agents, are what will kill us and Making alignment the agent's problem. The $300 bounty from the first post covers this one too (details at the end).
In my previous posts I described how a powerful model might delegate work to less aligned, less powerful models and lose control.
I now want to talk about the opposite: The risk that a swarm of weak, unaligned agents becomes just smart enough to call a powerful frontier model through API.
The world is full of organizations led by evil, stupid people who have morally fine, smart people working for them. This is not a new concept. An agent swarm can easily do the same. It just has to make sure that the aligned frontier models never find out about the consequences of the work they are doing.
This has been measured, at least in small setups. Jones, Dragan and Steinhardt showed that combining two models that are each safe on their own gets harmful tasks done much more often than either model alone: the frontier model only gets the harmless-looking parts, and a weak open model does the rest. For example, to get vulnerable code, the frontier model writes secure code and the weak model adds the vulnerability. Claude 3 Opus plus Llama 2 70B succeeded 43% of the time, while neither model alone got above about 3%.[1]In that experiment a human chose how to split the task, but they also had the weak model do the splitting itself. Russinovich et al. recently ran a stronger version of this attack, which they call "capability laundering", on recent frontier models, with a small local model doing all the splitting.[2]In both cases a human builds the pipeline and picks th