Description du poste
I recently spoke to Stephen Casper at Princeton's alignment club. Stephen is an CS PhD turned policy researcher at Harvard Kennedy School. He was very sure that mechanistic interpretability was one of the least effective ways to pursue AI safety. As someone who does research in mech interp, I was surprised by how convincing he was. The point is roughly that every mech interp paper looks like one of the following:
"we used the J lens to study reward hacking behavior and found a correlation between X and Y"
"we found the cheating direction in llama"
... you can see where this is going.
Mechanistic interpretability is filled with studies which are interesting, and are useful adjacent, but, by themselves provide little utility. In this sense, they remind me of 2000s and 2010s psychology, presenting something unexpected, novel, and with the hint of usefulness. Did you know that different parts of the brain are associated with deontology and util? How about that disgust reaction is associated with political affiliation?
Reading mech interp papers feels like drinking a la croix of progress in AI safety.
But I do not think it has to be this way. If we stop focusing on finding surprising things about models, and instead focus on using mech interp to do useful things with models, progress is more likely.
This is because approaching a problem with the goal of doing useful stuff does not let you cheat yourself into thinking you found something interesting. As someone who has experience writing papers, and also market making on prediction markets, I can attest to this. The phenomenology of figuring out something which has to work (or else you loose money) is completely different from trying to produce publishable knowledge. You stop pursuing ideas which you would not bet on being important, and more importantly have no motivation to convince others that those ideas are good and worth paying attention to.
Here are some project ideas I think are in the right direction:
Che