Description du poste
The ProblemIn my experience, those who predict AI-caused extinction tend to view their main task as proving two things: that superintelligence is possible (i.e. a machine intelligence can be built that’s better than the best human at just about everything), and that it’s possible for superintelligence to be “misaligned” (i.e. have goals other than those humans wish it had). They assume that once misaligned superintelligence has been established, there are thousands of reasons for it to cause a near-total human extinction event, and thousands of ways it could easily accomplish this goal.
That explains why AI extinction scenarios tend to spend tens of thousands of words on the computer science of misaligned superintelligence, and then wrap up the omnicide part in two or three vague paragraphs.
As a physicist and hard sci-fi writer, I find this backwards. The most critical aspect of such a scenario, for me, is not whether there can be an autonomous robot collective that’s better than the best of us (which I always assumed was possible), but whether, how, and why such an intelligent system would cause a near-total human extinction event.
To be clear, this essay is not an argument for AI accelerationism or against the idea of understanding AI better before developing it much further. This essay is merely an argument that scenarios resulting in AI-caused human extinction within the decade (and within the foreseeable future) are highly unrealistic.
It seems the authors of these scenarios are extrapolating the concept of hacking a computer to the physical world. They assume a superintelligence could duplicate global supply chains overnight or predict the unpredictable or execute a synchronized attack across the world without being caught in the preparation.
But the world doesn’t cooperate like that. Hauling the necessary amount of material across the globe to build a machine civilization takes a long time. Executing a coordinated global operation without detection presuppos