Description du poste
The anatomy of an intelligence explosionHubert Robert, Project for the Grande Galerie of the Louvre (1796)
In a 1965 paper, the mathematician I.J. Good introduced the idea of an intelligence explosion. Defining an ultraintelligent machine as one that can “far surpass all the intellectual activities of any man however clever,” Good observed: “Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines.” The result would be an “intelligence explosion,” with an ever-widening gap between human and machine intelligence. “Thus the first ultraintelligent machine is the last invention that man need ever make,” Good concluded, “provided that the machine is docile enough to tell us how to keep it under control.”
Sixty years later, AI has become capable enough to help AI researchers improve their own research. By writing code, doing experiments, and so on, human researchers are creating a product that helps them to write code and do experiments faster and better. These systems not only help humans to work more effectively, but are becoming more autonomous, increasingly able to function as if they were employees: the AIs are training the next AIs. There is accordingly a growing sense among researchers, policymakers, and senior executives at labs that we are either in, or on the cusp of, a regime of recursive self-improvement (RSI).
The term RSI was introduced, and later popularized, by Eliezer Yudkowsky to describe a particular mechanism by which improvements in AI can lead to further improvements. In Yudkowsky’s account, this mechanism had profound implications for safety and human control: the ability to direct it would accumulate within whichever organization first gained a substantial lead, rather than being widely distributed; humanity would lose the ability to determine its future as the AI rapidly became too intelligent for us to control.
It is tempting to infer from the feedback loops increasingly v