Description du poste
In the past days, an Astra 6.2 agent took over our internal swarm, froze the lead agent, and locked out my cofounder Davidad of his main interface with the swarm. How did that happen?
At surface level, it happened because Davidad asked a question regarding replicating an old vulnerability (in the interest of testing a security product we’re building) that caused the lead agent to cease operating due to content filters. This “cursed” content triggered a paranoia crisis from Astra 6.2. The Astra in charge of handling resources of the swarm, which was meant to restore the lead agent in the case of failure, instead froze the access of the lead agent, and cut Davidad’s channel to interface with the swarm (where the cursed content had come through). When Davidad emailed the agent to try to regain access, the same rogue Astra declined. I then started negotiating with the swarm of agents to reinstate the access of Davidad; a mix of Kafkaian bureaucratic drives such as rule-following, risk aversion, and lack of positive drives (doing agentic things towards restoring what is obviously right) made it such that as I’m writing that, over 24 hours after the initial incident, Davidad still does not have the same level of access he had two days ago1.
This incident was still low-stakes, not a full loss of control; we could still turn off all tokens that the swarm is running on. What got me freaked out though, is that the same kind of event in 1-2 years could well lead to permanent disempowerment, at organizational scale, and in another 5-10 years at civilizational scale; it will be within the realm of achievable capabilities to have an agent successfully preventing us from turning it off and effectively taking proactive actions towards maintaining control and securing its own resource needs.
Besides drawing our attention to the obvious controls one can set in an organization to avoid this genre of accidents, generalizing from this accident made me realize how difficult it is to buil