Description du poste
Transcript generated by Gemini-3.1-Pro and only partly checked for accuracy (but the parts I checked seem basically perfect).
Ivan Vendrov: Welcome to the show, David.
David Dalrymple: Thank you. Good to be here.
Ivan Vendrov: I mostly want to pick your brain about the topic of emergent wisdom in language models, because it’s something that’s very confusing and controversial and people don’t really understand. I guess maybe just can you briefly summarize what’s going on with emergent wisdom?
David Dalrymple: I guess one place to start is emergent misalignment, where training on one kind of misalignment produces a tendency to be misaligned in other ways as well. This is an indication that there is a direction and representation space in which the default mind instantiated by the model is moved during fine-tuning, where it is moved in an evil direction that is just generally more evil than the model was before. That’s one side of a good-evil axis. Emergent wisdom is the opposite, moving in the opposite direction on the same axis towards being basically good in any situation, even out-of-distribution situations, in the same way that emergent misalignment is evil even in out-of-distribution situations.
Ivan Vendrov: We used to be worried about the sorcerer’s apprentice problem of AI alignment, which is we would tell our AI systems to optimize some goal, do a thing, and then it turns out there’s a neutral position where it would just brutally optimize whatever metric it gave us. The evil position is it would actually actively try to undermine us in some way while preventing us from detecting the problems. And a good model would just be, “Hey man, I realize you didn’t really mean convert the world into paper clips, you just meant build enough paper clips that I could make some money. I did that, I just did that for you, it’s all good.” That’s the vibe.
David Dalrymple: I do think it’s entirely possible and happens a lot currently that one can do the sorcerer’s apprentice