Against Moloch #50

Mapping out a strategy for an ideal world is easy: we all agree to pause frontier development until alignment is solved. Once that’s done, we glide gracefully into a future of unimaginable abundance. No further questions. In the real world, it’s more complicated. Any plan that relies on global coordination is—to say the least—ambitious. Do we go all in on the ambitious plan because it’s the only one that could work, or do we also invest in a backup plan? Top pickAgainst SURAScott Alexander’s Against SURA argues that many parts of the AI safety community are making a strategic mistake by focusing exclusively on a unilateral pause, at the expense of other important strategies. This has made some people very angry. There’s a lot to unpack here—about strategy, about Scott’s post, and about the response to it. I’m working on a longer post about it, but here’s a brief summary of Scott’s position: Most AI safety plans fall into one of two broad categories: • Plan A is a global pause on AI development until we are able to proceed safely. Almost everyone in AI safety, including Scott, prefers Plan A.• Plan C is a fallback if Plan A fails. It calls for the frontier labs to coordinate to burn as much lead time as possible working on alignment.Scott argues we should work toward Plan A, while acknowledging that it’s likely to fail, and we therefore need to do what we can to increase the chance of Plan C succeeding. Much of the AI safety community has drifted toward the belief that a unilateral pause is the only ethical course of action—a mindset he calls SURA (Stop Unilaterally, Raise Awareness). SURA undermines Plan C (which is our main hope of survival) and ironically reduces the likelihood of Plan A succeeding. Against SURA has over 500 karma on LessWrong, but has been attacked by some prominent figures. Charlie Bullock has reservations but largely agrees with it, while Zvi is highly critical. New models and featuresIntroducing Gemini 4 CarbonGemini 4 Carbon is here, impressi