Description du poste
After watching a recent debate between Yampolskiy and Israetel, I’ve realized that one of the main hurdles of AI optimists is the inability to understand why bad things would happen by default. They run along these lines roughly:
• Why would it want to kill us?• Why would we build something that wants to kill us?• People seem to get nicer as they get smarter, so why would this be any different?A second major issue seems to be a neglect of the full implications of what it means that artificial intelligence (AI) is closer to something that we ‘grow’ rather than something that we build. Their argument seems to be:
• It’s a thing we build, so just build it correctly.• If you are right, and it’s something we grow, then just grow it correctly, the same way we raise children.A third thing is that even when they do concede extinction risk may occur, they nonetheless think it is unlikely to be the default, and more like an edge case. And one that we would have to be very unlucky to reach.
Most readers will be familiar with the long body of alignment literature in this area, so this text will be short on citations.
Instead, I’m going to try and tackle these questions in a brief but hopefully persuasive way to an average audience.
1. Why (“Bad Things Will Happen”) By DefaultThe evidence typically used to support AI optimism/accelerationist takes is that humanity seems pretty competent at not letting bad things happen, despite the fact that many of these things seem dangerous, and despite the fact that many people have argued in the past that these ‘bad things’ should happen by default.
1.1 Anthropic BiasYou can’t sit around to observe how wrong you were about bad things that kill you by default if the bad thing killed you.
• Optimists cite the way the international community banded together to stop the proliferation of nuclear weapons as a case of humanity getting it right.• But if humanity got it wrong, and nuclear armageddon had killed everyone, no one would be around to rem