Scott Alexander’s Against SURA Seems Deeply Confused

If you are helping build something that might kill everyone, maybe stop doing that. Ideally, when you do stop, tell everyone else to also stop building it. That is certainly the obvious, first-level thing to do. You need a damn good reason to decide that no, actually you should keep building the thing that might kill everyone. Scott Alexander is worried too many people are insisting upon this too loudly. He comes out against a strategy focused on getting people to stop. I disagree. I think many of the claims, arguments and rhetoric in Scott’s post are quite poor by the standards that Scott Alexander would apply to himself. Scott clearly continues to support advocacy for a global pause, or what he calls Plan A. His objection is to those advocating primarily or exclusively for SURA, which stands for ‘stop unilaterally and raise awareness.’ The practical differences here are smaller than they may at first appear. This post first lays out my understanding of Scott Alexander’s core claims, then gives my core reactions to those claims, then goes over various questions in detail along with gathering the resulting discussions. Table of Contents • Scott Alexander’s Core Claims. • My High Level Responses To Scott’s Core Claims. • Briefly on Alignment. • Probabilities and How They Work. • Building a Thing That Might Kill Everyone is Deontologically Evil. • People Quitting Major Labs While Raising Alarms Obviously Works. • On AI Forecasting. • Advancing Capabilities Burns Timeline. • But China. • Ethical People Can Disagree With You. • There Are More Than Two Outcomes. • Efforts Are Complementary, Not Mutually Exclusive. • If Anyone Builds It, Everyone Dies. • Vaniver is Not Convinced. • There Is No Consensus That We Should Exclusively Focus on Plan A. • It’s Not the Incentives, It’s You. • Anthropic Is Not Optimally Pursuing These Plans. • Against Against Sura. Scott Alexander’s Core Claims Scott Alexander writes Against SURA, or the strategy of calling upon nations a