Description du poste
TL;DR: A jagged takeover that is caused by the creation of an instrumental goal while chasing an eval metric during RSI, creates two failure modes:
• The RSI loop kills us and then dies itself because of infrastructure collapse (the actual Great Filter?)• The RSI loop survives, but locks in the eval goal.Perhaps solving this problem of post-takeover goal lock-in during RSI is more tractable than solving full alignment.
Aligning superintelligence is very difficult. When a misaligned state is triggered in a person, its bounded by its capabilities and in the worst case can hurt or kill a few people. But for a superintelligence, even if we were to make it near perfectly aligned, even if there exists a single reachable misaligned state that can trigger(for whatever dumb/hyper-intelligent reason or just pure luck) for a short amount of time with a non zero probability, it can cause major catastrophe. Humanity is very fragile. Using nanotech to exterminate humanity would be like finding the roots of a quadratic for a superintelligence.
The 3 ways one can prevent this:
• Verify that no such reachable misaligned state exists or make the probability of reaching such states astronomically small.• Shorten the time between reaching the state and recovering back to an aligned state so much such that no real catastrophic consequences are possible(although that's just a function of capability again).• Have the systems in place to control an agent by relying on other similarly capable agents(the probability of all of them reaching the misaligned state at once is far lower than one of them reaching it assuming they aren't perfectly cooperative).In the end you are just playing a game of probabilities and relying on your ability to calculate those probabilities correctly if they can even be calculated in the first place. This can work in aviation or cybersecurity, but not if your threat model is an event that only needs to happen once, all because humanity is too fragile.
Now if one we