Description du poste
This post is written in a personal capacity. All opinions expressed are solely those of the author.
I. Executive SummaryThere’s a mismatch that I’d like to correct between:
• the reasons why superintelligence is dangerous, and• the categories we use to describe how dangerous superintelligence is.The most important cluster of reasons why a superintelligent model is dangerous is that it (i) could quickly kill or impoverish billions of people, (ii) could be very different from its predecessors, and (iii) can’t easily be altered once it’s deployed or otherwise widely distributed. The combination of these three factors is worse than the sum of its parts: the only way to protect ourselves against a powerful, novel, incorrigible AI model is to accurately anticipate what dangers it might cause and prevent those dangers before they come to pass.
Unfortunately, we don’t yet have a good vocabulary for talking about whether AI companies can identify and mitigate dangers faster than those dangers arise.
The by-now cliched way to describe the danger of superintelligence is p(doom), i.e., a static point estimate of how likely AI is to end human civilization or otherwise cause catastrophe. More sophisticated descriptions might offer a conditional probability (“here’s how doomed I think we are unless X happens”) or a rough estimate of how many hours of work would be needed to make AI less dangerous (“I think solving alignment is roughly as difficult as the Manhattan Project.”) However, even these more sophisticated descriptions are more like distance than velocity; they tell us little or nothing about the rate at which capabilities or safety are scaling.
This is a problem because the only way we can arrive at a safe future is if the safety techniques that are actually implemented[1] by frontier AI developers scale in power at least as rapidly as frontier AI capabilities. What matters is not how likely it seems that today’s AI will kill us, or even how likely it seems that next year