Description du poste
Getting back in the safety field after 4 years feels great. I had a paper on concept distillation where I showed how steering vectors based method did'nt help to debias model since the concepts itself are understood wrongly [Concept distillation, NeurIPS 2023].
I am so happy to see the steering vectors work picked up in interpretability and was curious to see how methods like ActAdd would work in this debiasing setting?
So, I apply ActAdd to debias a model.
Consider Colored-MNIST setting (age-old) where a model is trained on digits in a specific color always (say it sees zeros always in red). What would you expect to happen? It learns to associate digit with a specific color (sees red, says zero)! Now when such model is tested on reversed color digits, it fails badly (nearly zero accuracy).
Classifier learns to associate color with digits during training; in test, gets zero accuracy.
How to debias this model using steering vectors?
We can find directions of color red or simply color and steer model activations away from this direction simply. Lets try it out.
Results (will be discussed one by one below)
Method
Mean Test Accuracy (3 seeds)
Base model
0.0
Base model on grayscale inputs
16.9
ActAdd, single pair (notebook)
21.4
Random vector, same norm
19.0
Project out the mean-diff direction
9.4
Concept loss based steering (w/o teacher)
27.0
Concept loss based steering (with concepts from larger model)
41.8
Add-add adds difference between positive (here red color) vs negative (here random blobs) activations to the model layer and steers the activations with this transformation (read og paper for more details)
This can correlate with biased concepts; I did a controlled experiment in my NeurIPS work where I checked cosine distance of CAV_red (trained with red patches vs random - random color blobs) with activations of red-patches vs red-zeros.
What would you expect?
cs('red', CAV_red) < cs('red_zeros', CAV_red)
CAV_red was closer to red_zeros activations over re