Misalignment traps lie in wait

Here's a basic argument that describes "misalignment traps" lying in wait as a mundane threat that will not be straightforward to fix. The argument is as follows: • Models are very sample efficient, and larger models tend to be more so. They generalise broadly from a few bits.• Models learn from everywhere: pretraining, post-training, and in context. We may also figure out online learning eventually. • Narrow data can cause a model to generalise broadly in unexpected ways. One explanation is that a model works out what kind of agent would produce the data, and becomes that agent.[1]• Such settings are often considered toy, but this is an over-update. • The datasets used to induce such behaviours (like insecure code, evil numbers) are pretty unrealistic to train on. A trivial mitigation applicable to these datasets would be to simply learn nothing. But, these are facts about the demonstrations, rather than the broader mechanism; researchers often pick obvious datasets to get clean existence proofs.• An example of a more realistic trigger for emergent misalignment might be the accurate offensive-security Q&A dataset from Obeso et al. • This is a dataset where you might plausibly want a model to learn valuable knowledge or skills, and the problem of selectively learning from the data is well-motivated.[2]• Reward hacking is a realistic well-discussed case here, as per Anthropic's production RL paper, an open replication on Llama 3.3 70B, and iterative DPO on GPT-4.1 with no hints. But a replication by UK AISI found only some inconsistent misalignment on Olmo 3 and GPT-OSS, and Yudelson et al. got reward hacking without broader misalignment on GPT-OSS-120B and Kimi-K2.5.• The non-emergent misalignment results are also quite plausible. A model may, over the course of long conversation with many web searches, come to in-context learn that it fits a certain definition of AGI, or gain confidence that it is conscious. It may also simply learn this by reading about itself in