Description du poste
TL;DRWe track how role-playing characters move along the assistant axis (Lu et al., 2025) across the post-training checkpoints in Olmo 3 7B Instruct. We find that:
• DPO pulls desirable roles toward the assistant axis, while undesirable roles on average stay put, though some still move toward it.• RLVR barely changes anything.• DPO also makes the model more willing to role play.Resources:
• Code: https://github.com/xiaohan2012/assistant-axis-shift• Data: https://huggingface.co/datasets/han-xiao/assistant-axis-shiftIllustration: after DPO, desirable roles sit closer to the Assistant while most undesirable ones stay far
IntroductionLLM behavior is largely shaped during post-training. After post-training, the model is expected to behave as a helpful and honest assistant. Still, we sometimes observe undesirable behavior such as sycophancy and hallucination. In this work, we aim to understand how an LLM's default behavior changes across different post-training stages. Understanding the shift dynamics helps us answer the following safety-related questions:
• When does a model's behavior become more aligned with our final goal of having a helpful assistant?• After which stage do undesirable roles tend to express themselves more?We re-use Lu's framework (Lu et al., 2025), where the default assistant is embodied in an interpretable axis, defined relative to a list of pre-specified roles (such as "assistant", "teacher", "spy", and "zealot"). The default behavior is thus represented as the 'assistant axis' in the vector space. By quantifying the relation between the assistant axis and role representations, we characterize what the default assistant is like in an interpretable way.
Our analysis on Olmo 3 7B Instruct (which undergoes 3 post-training stages, SFT -> DPO -> RLVR) shows that:
• The SFT -> DPO stage (meaning the SFT checkpoint is fine-tuned by DPO) shapes the default behavior more profoundly than DPO -> RLVR.• After DPO, desirable roles appear to be more aligned with