Revisiting Verbalized Eval Awareness in OLMo and tracing it to the training data

TL;DR • Contrary to what was previously reported, DPO does not suppress eval awareness. It is a chat template artifact.• Formatting differences between chat templates elicit different behaviors. OLMo's SFT data was written by different models. DeepSeek R1 wrote the safety data and starts its reasoning with <think>\n. A prompt that already ends with <think> can't start that way because the tokenizer merges > and \n into one token.• The base model recognizes obvious evaluation prompts but not realistic ones. The base model can separate plain and harmful evaluation prompts from deployment prompts, but not realistic prompts like Fortress.• We may be making models verbalize evaluation awareness, whether or not they are aware. Removing VEA from the SFT data and retraining drops VEA from 32.6% to 5.1%. Refusal does not change.• Removing VEA from the SFT data also removes its correlation with refusal. In the model trained without VEA in its SFT data, VEA and refusal are no longer correlated.BackgroundIf models can distinguish between deployment and evaluation scenarios (evaluation awareness), they might behave differently and inflate measured safety. Verbalized evaluation awareness (VEA) refers to a model stating that it may be under evaluation. Tracing Eval-Awareness Emergence Through Training of OLMo 3 measured VEA across the OLMo 3 training pipeline: pretraining, SFT, DPO, and RLVR. It found that VEA was about 1% during pretraining, rose to about 40% after SFT, fell to about 22% after DPO, and rose again during RLVR. However, the sharp suppression after DPO is not supported by the DPO data. The chosen responses contain more VEA than the rejected ones (8% vs 1.6%), so DPO should in principle increase VEA. The post attributes the suppression to the lower absolute VEA rate in the DPO data than in the SFT data, but one of the commenters noted that this explanation is inconsistent with the form of the DPO loss. We start by investigating this issue. We follow the post's setup: