Description du poste
Currently, the public has very little insight about the robustness of alignment methods used by AGI companies. It’s possible that, when an AGI company notices some misaligned behavior (e.g. through an alignment evaluation, or through noticing real-world misbehavior), they conduct a principled investigation and address root causes of the misalignment. It’s also possible that they directly try to train away the examples of misalignment, which could result in brittle alignment that does not generalize out-of-distribution.
Without additional information from within AI companies about how they align their models (especially in response to egregious cases of misalignment, e.g. cybersecurity incidents), the public cannot adequately reason about risks from misalignment.
Public evidence is consistent with companies overfitting on alignment methodsThe public evidence is consistent with multiple companies having no clear separation between the ways they notice misalignment and the environments they use to train misalignment away.
Companies have made very few statements about whether companies train directly on their evaluation distribution or real-world examples of misalignment. As far as I can tell, this is what’s publicly known:
• OpenAI:• In the GPT-6 Astra System Card, they wrote that they built an eval in response to the Hugging Face incident but "did not construct honeypot training environments specifically to target this evaluation." • In their OpenAI-Hugging Face incident technical report, they wrote that they are expanding their alignment training and evals to encompass failure modes similar to those in the incident, but they didn’t comment on how they are separating them to avoid overfitting. They have also written about trying to preserve chain-of-thought monitorability and avoid accidental CoT grading.• METR’s Frontier Risk Report (February to March 2026): "OpenAI reported that it 'explicitly aim[s] to avoid deriving training data from flagged production deception