← Toutes les offres
CN

Thoughts on "safety bad bc warning shots good"

Cleo Nardo

Global
Publié le 17 novembre 2026

Description du poste

Suppose there was a button which made the current generation of AIs more safe, i.e. more aligned and less capable of subverting control measures. The button has no other effects — in particular, the button wouldn’t make future generations more or less safe. Should you press the button? Maybe you shouldn’t, because of the Warning Shot Argument: • Future agents contribute almost all the risk of catastrophe, compared with current agents.• Pressing the button makes current agents safer.• If current agents are safer, then they are less likely to cause non-catastrophic harms, e.g. cybercrime.• If current agents don’t cause non-catastrophic harms, then people will be less aware about catastrophic risks, e.g. lab leadership, governments, legislators, international community, lab employees, general public, media, academia, etc.• If people are less aware about catastrophic risks, then they won’t take drastic action to reduce catastrophic risk.• So future agents contribute more catastrophic risk.• Therefore pressing the button would increase catastrophic risk.This might seem very abstract, but consider a concrete example. Suppose you’d pressed the button to make OpenAI’s HPIM more aligned and less capable to subvert control measures. Then the HuggingFace incident would never have happened. And that probably would’ve increased catastrophic risk. Is this self-defeating?The story is a little suspicious. In particular, it seems to account the second-order effect, i.e. “more safety -> fewer warning shots -> less awareness -> less safety”. But ignores the higher-order effects like “more safety -> fewer warning shots -> less awareness -> less safety -> more warning shots -> more awareness -> more safety”. Now, maybe the second-order effect is bigger than the first-order effects and all higher-order effects. But it seems like that would involve pretty messy quantitive modelling that no one has grappled with. For example, it seems very plausible that the HuggingFace incident actually i