Description du poste
This article replicates and extends the Persona Vectors' paper results, with a particular focus on coherence, an aspect that the original paper uses as a filter but leaves unexamined in detail. We find that steering coefficients large enough to express a trait also breaks the model's coherence (probably a consequence of the model leaving its normal distribution), and that finetuning reshapes the space along the trait direction.
After preventative steering, evil expression along the original direction drops but a lower projection is consistent with two different stories: the trait is suppressed or the trait has moved. We don't resolve which, and current tools (KL divergence, projection along the old vector) can't tell us. Reliable safety applications of persona vectors will need a deeper understanding of how these directions evolve across subsequent changes to model parameters to ensure adopted measures are robust.
IntroductionAs part of the ARENA 6.0 capstone project, we decided to read and work on the paper Persona Vectors: Monitoring and Controlling Character Traits in Language Models, which introduced an automated pipeline to find directions in the residual stream that represent character traits. The main contributions of this paper were:
• Proposing an automated and simple pipeline to obtain the residual stream direction for a particular trait. Some examples include "evil", "optimistic", and "sycophantic".• Studying defense mechanisms against model misalignment based on these vectors:• • Steering models at inference time in the negative direction of undesired traits.• A preventative steering finetuning mechanism, where the undesirable trait vector is added to discourage the model from incorporating such behavior.• Training data analysis, to understand how it projects onto undesirable persona vectors.• Monitoring prompt projections onto undesirable persona vectors during inference.Despite the demonstrated effectiveness of the approach and the generality of the pi