Description du poste
Epistemic Status: Brief explanation of an unsettled debate, not novel theory; writing is entirely my own, with diagrams from ChatGPT 5.5. (Originally posted on twitter)
Recent debates have highlighted a divergence between Yudkowskian AI-doomers and AI-whisperers about how capabilities will evolve that is really important. Specifically, j⧉nus thinks that alignment has been going really well, and that we should update significantly from current jagged capabilities. In my view, whether or not that criticism is warranted relies on specific details about future capability trajectories, which then determine what updates we should make based on how RL is used.
Original Assumptions and LLM Alignment-by-DefaultPre-GenAI rationalist assumptions were around RL leading to capabilities. By reducing human values into an input, as the ChatGPT-generated diagram A implies, the RL will inevitably Goodhart away from the true goals, and in the limit entirely omit human values. If correct, this would make alignment inevitable. The use of foundation models changes that, because capabilities emerged from human representation, shown in diagram B. This is (conceptually) part of what led to a schism between the labs and the outside safety concerns in the first half of the 2020s; those optimistic about safety thought that we could preserve the alignment-relevant representations.
What is missing is what Yudkowsky called the best AI news of 2025, that values and capabilities were entangled, and there was some central preference vector. To the extent that diagram B’s two outcomes are actually correlated, j⧉nus claims we should update towards thinking that prosaic alignment will work; this is what is shown in model C. Rationalists argue that even if this is true, pushing from human level to superhuman capabilities creates a divergence, shown in diagram D, and returns us to something more like what rationalists originally expected.
From RLHFed LLMs to ASIThe Huggingface attacks would be cited as