Description du poste
There is a TL;DR at the end.
The heaven button: You wake up one day and find a button in front of you, with a proof next to it. The proof says that pressing the button will kill everyone on the planet and send them all to heaven. Would you press it?
It seems too good to be true. You think it must be a dream. It can't possibly be true. You go through the proof, and it seems correct. But what if you were under the influence of some drug?
But the stakes are very high, so you use software you trust to formalize it in Lean, and you run it on a machine you trust. Lean says the proof is correct so now you are more convinced. But you start doubting yourself again. What if someone compromised the formalization software, or introduced a bug in Lean?
So you check Lean's code, and run the proof on a second machine with an independent checker. It passes again. But each of these checks could fail in the same way the first one could. So you go further and formally verify Lean and the formalization software themselves. They pass. But now you are doubtful of the tool you used to verify this. What if that was compromised by someone? Every new check needs another check behind it, and at some point you have to stop and decide.
Would you press it?
A common belief in AI labs and AI safety is that handing control to some general superintelligence would be fine, if we have a formal proof of alignment. That, in my opinion, is saying yes to the same button, except there is no proof yet, only the hope of one.
In this post, I present what I think is the hard problem of alignment.
• First, I illustrate why testing for alignment is not enough and a proof would be required.• Then I lay out why a proof could be intractable.• Finally, I argue that even a formal proof that a superintelligence is aligned does not justify its development.The hard problem of alignmentThe hard problem of alignment: How can a system reliably establish whether a system broadly smarter than itself is aligned with it?
The m