Three Phases of Capabilities and Alignment

Cross-posted from my personal blog Epistemic Status: Reality is messier than the elegant procession of 'phases' presented here but, nevertheless, I do think this gets at an important way to think about how alignment and capabilities are intertwined and how the alignment challenge becomes increasingly abstract as the underlying autonomy and generality of the models improves over time. A thought struck me recently which I thought was interesting. As AI has progressed through different technological phases with different underlying drivers of progress, the corresponding alignment challenge has migrated too. In fact, in many ways the alignment challenge is the mirror of the fundamental capabilities challenge of each phase. This may actually be inevitable since alignment and capabilities are intrinsically deeply intertwined -- for a model to be capable it has to be at least minimally able to do what we want it to do. 'Capabilities' are intrinsically about doing the things we care about models doing[1]. The methods used to improve capabilities tend also to be applicable to alignment as well. What changes as we move to progressively more general levels of capabilities is that the training loop itself becomes increasingly endogenous to the model, and that this forces our notion of alignment to exist at progressively higher levels of abstraction. We move from directly constructing datasets exemplifying the desired behaviour, to constructing procedures for judging aptness of data, and ultimately to controllers that preserve an invariant in an otherwise fully autonomous learning process. Anyhow, enough abstractions, let's dive in. I'm thinking of modern (and future!) AI as basically progressing through three phases: pretraining, RL, and continual autonomous learning. In each phase, we progressively offload more of the training loop into the model itself -- i.e., with each phase, more of the loop is internalized. We are now deep into the RL phase and perhaps beginning to s