← Toutes les offres
R

Does most AI x-risk come from mostly-aligned models?

RohanS

Global
Publié le 19 novembre 2026

Description du poste

I had a brief discussion in the Constellation Slack that I thought might be of interest to a broader audience. Thanks to Owen Cotton-Barratt, Daniel Kokotajlo, and Eli Rose for commenting. Rohan SubramaniSomething I posted on LW about a month ago that I’d like to discuss with people here, because it feels pretty cruxy for threat modeling imo: How do people feel about the following 3 claims? • We currently have AIs that try to do a reasonable balance of what their developers and users intend on the vast majority of inputs they will see in practice (eg, practically useful tasks or fun tasks that people ask for work or personal use).• We’re on track for that to remain true through to early superintelligences. (Skip if you don’t agree with 1.)• AI x-risk from misalignment is >10% likely even if we only ever build AIs that do a reasonable balance of what their developers and users intend on the vast majority of inputs (because of the misaligned behavior on the tiny minority of inputs).Off the top of my head, I think I’d put ~90% on #1 and ~80% on #2 conditional on #1, and I currently believe 3 but give it ~20% that I’ll change my mind within the next 6 months. A slightly different framing that maybe more precisely captures what I really mean: Does most AI x risk come from AIs that consistently have large-scale unintended goals, or from AIs that only very rarely do? Owen Cotton-BarrattI think I get off the train at 3, but I don't totally know how to distinguish your "slightly different framing": I'd say that AIs that consistently have large-scale unintended goals is most of the risk; except that I think AI rarely having badly misaligned goals is a significant multiplier on the chance of ending up with that kind of bad world. (So maybe the question is whether the multiplier is more than 2x ... I'm gonna say no but I'm not confident) Daniel KokotajloIs 1 true for human slaves? Like, consider a historic slave society like antebellum South. Would the equivalent of 1 be true t