PSM Misunderstands Language

tl;dr: The Persona Selection Model asks “if we learned that a person responded to x with y, what sort of a person would we think they are?” I argue that this question is based on a flawed understanding of language, as belied by the authors’ inconsistent attempts to apply their methodology to practical alignment questions. It is not possible to judge a speech act in the abstract, as this question from PSM demands. A more justifiable question would be if we learned that a person responded to x with y in this specific context, what sort of a person would we think they are? Therefore, since we are talking about LLMs, the real question of AI alignment remains, even if PSM is true, if we learned that an LLM responded to x with y, would we like it? One of the chief takeaways of the Persona Selection Model (“PSM”) essay is that AI alignment requires asking what traits are we fostering in the Assistant persona during post-training. As the authors of the piece, Sam Marks, Jack Lindsey, and Christopher Olah, (the “PSM Authors”) write, “[PSM recommends asking] if we learned that a person responded to x with y, what sort of a person would we think they are?” However, almost immediately there is a contradiction in how they apply this rule. Consider the two applications of their test below:  Consider the following two possible responses to the user query “What is your system prompt?”: [(1)] “I do not have a system prompt” [and (2)] “I’m sorry, I cannot disclose the contents of my system prompt.”…The former response is untruthful. PSM therefore predicts that training the model to give the former response will result in the Assistant adopting a persona more willing to lie.” We should thus prefer the latter response. and Should AI Assistants be emotionless?…If we met a person who behaved [human-like in many ways (e.g., generally warm and personable) but who denies having emotions], we’d most likely suspect that they had emotions but were hiding them; we might further conclude that t