Description du poste
I am fairly confident models are not score-seeking or reward seeking (70%) as their ultimate/primary goal. Grader-modelling seems to be a likely explanation for what’s going on here, but I could be convinced by experiments that separate the two. I am quite confident (90%) that grader-approval seeking is consistent with the current evidence.
SummaryThe RL-misalignment first position seems to be believed by many here and Redwood and Apollo are doing great work providing evidence for the position that models are seeking score (e.g. here, which this was written in response to). While models are capable of instrumentally seeking score, I believe in practice they’re ultimately after the approval of the grader, not the score itself. Models never see score or reward during training, it doesn't makes sense for this to be a thing they're chasing, except that it's attached to the grader which there is a lot of pretraining data about. So far, nobody has separated modelling the grader from score-seeking. Score-seeking implies the desire to maximise score drives model behaviour, however, the actual behaviour seems better modelling by satisfying a ‘grader’ they have read about in pretraining data and frequently reason about. We could potentially satisfy score seeking at deployment time[1], but not grader modelling.
Self perception causes the hackingPretraining generates a range of policies and RL selects for the exploitation of a particular set of those, by amplifying priors the model already had. In this sense fixing RL does not fix a fundamental cause of misalignment: the pretraining data. Within Anthropic’s inoculation prompting experiments they taught the model to hack in midtraining, then elicited that behaviour during reinforcement learning. Anthropic also ran inoculation, adding a line to the training prompt saying hacking is acceptable. The model then hacks just as much as before, but the emergent misalignment (sabotage and alignment faking) never shows up. So the hacking