Description du poste
In his recent post, Scott Alexander argues that plans allocating a short amount of time—perhaps a few months—to solving alignment should be taken seriously. He argues that people calling for an indefinite or long pause have insufficient grounds for concluding that alignment of ASI is very difficult and cannot simply be solved by delegating it to AI systems.
There's a kafkatrap where if someone expresses hope that alignment is not doomed, they get told that they don't understand the subject, but if they quote widely-acknowledged experts who say alignment isn't doomed, they get accused of "deferring" and "modesty arguments" and "refusing to think for themselves".
I agree that this is an important question for determining whether something like Plan C is worth pursuing and how we should allocate our resources. That is why I wrote this post: to explain why I do not think a short pause allows us to solve ASI alignment.
TL;DR: I believe we only get one critical shot at this problem, and our ability to learn from experience is very limited. We are unlikely to get what we train for: the models we train will have different objectives from those we trained them for. This is made even harder because the tech tree to aligned superintelligence likely passes through efficient unaligned superintelligence first. It also worries me that people who have spent substantial time investigating aspects of alignment, such as corrigibility, have found deep problems and conflicts with agency. I think current approaches like persona alignment are deeply flawed. All these problems come back to bite us when we try to align a pre-ASI system to do alignment for us. These are some of the arguments that give me strong confidence that ASI alignment will be very difficult, and that Plan C is not particularly promising.
Aligning superintelligence seems obviously hardImagine we have created alien minds far more intelligent than all of us together, and we do not really understand how they think. These