Description du poste
This essay is a combination of personal notes, discussions I had at the MAIA/AIST workshop, and slides from a talk I gave with JHU ASBS and BlueDot Impact.
PremiseA common theme I noticed, particularly among viewpoints with short timelines, is the idea that the introduction of ASI is inherently and instantaneously the loss condition. This stems, I think, from Yudkowsky-ian philosophy that ASI = unavoidable doom.[1] I think this defeatist attitude is counterproductive and detracts from real, important, effective avenues of action.
A superintelligence is bounded through interaction with the physical world; there are fundamental limitations on feedback loops which are crucial to enact any kind of influence.[2] ASI on a laptop in a concrete room in 1850 cannot escape with any high probability, and even if it did, it would take a very long time for it to do anything – conceivably much longer than the physical lifetime of the laptop. In well-defined worlds like video games, it is not hard to imagine scenarios where you are mathematically guaranteed to lose even with perfect skill and play – for example, if the ball in pong is much much faster than your paddle. There is an analogy here to galactic algorithms in computer science[3]: just because an ASI has an algorithm that scales better asymptotically doesn't mean the constant factors (e.g. time needed to conduct experiments; time needed to manufacture a quantity of robots) allow it to win within a finite time horizon.
In the present world, incidents like HuggingFace[4] can be largely attributed to 1. Ease: cybersecurity and the internet is an extremely accessible attack surface for AI, 2. Lack of foresight: most of the internet was not designed in anticipation of highly intelligent adversaries and widespread integration into society (cybersecurity is almost always reactionary), and 3. Ubiquity: the internet is a widespread and well-connected infrastructure tied deeply to society. As I, and many others have previously wrot