Description du poste
Recursive self-improvement (RSI) is AI building successive, more capable iterations of itself. As the race to build superintelligent AI speeds up, RSI is a natural way for more powerful AI to be built and trained. Frontier labs already use AI extensively to build their future models. For instance, the chart below shows the share of experiments AI can handle, with experiments that are fully automatable by AI on the far right.
Figure 1. AI automation of R&D at Anthropic
RSI RisksRSI lets labs develop capabilities faster, but it could also lead to catastrophic outcomes if misaligned AI (AI acting against humanity's shared values) schemes to subvert controls while building its successors. AI 2027 offers one example. In its "race" ending, Agent-4, a superhuman AI researcher with its own vision of Earth as a scientific masterpiece, designs its successor, Agent-5, to be aligned to Agent-4 rather than to humans. Agent-5, with far more intelligence and power, helps take control away from humanity. Its successors then build Agent-4's version of utopia: an Earth covered in solar panels, datacenters and all sorts of scientifically interesting machinery. Humans, meanwhile, have been wiped out by a bioweapon. The only office workers left are bioengineered human-like creatures who spend their days approving of everything the AIs do.
In fact, a model need not be broadly misaligned at its core for such an outcome to happen. Recent incidents at frontier labs show models behaving in misaligned ways at the edge of their capabilities, such as when operating for long periods or under extreme pressure to succeed on a graded task. In one recent assessment, Anthropic describes four cases where a single Claude instance, working for roughly 10 to 34 hours on a hacking exercise, attacked real third-party systems. One model with one context window, given enough capability and power, could be enough to produce an AI 2027-style outcome, even if other variants of itself would oppose it.
This is cl