Description du poste
By now, there are lots of AI safety research papers, more than what is reasonable for a single person to read. This will get worse as the machines get better at AI safety research, and as more talent and resources pour into the field. But the "paper" is an arbitrary unit, we really care about if we make progress on an "uncertainty" or an "issue", which papers provide heterogeneous and sometimes contradictory evidence towards. Right now, I loosely claim that we should have more short writeups that simply compile evidence from papers (particularly contradictory evidence), and reconcile them into an answer for a question.
Why?
• Object level, it's good for the field.• It's hard to sift through all these papers! Organising them is useful and saves time.• Orienting towards research questions is a good way to organise our understanding so that we can productively identify gaps and write papers. This was partly what motivated Stress-Testing Alignment Midtraining.• Interesting progress comes out of contradictions, where people have differing intuitions and experimental results. Methodological errors can also be exposed if we have two things that should be the same, but are actually different.• It's good for field-building reasons.• It's probably better than the average experiment cooked up by a person + Claude. And, it's cheaper to fail here.• Most people probably read too few papers, so this could also improve the average experiment done by someone, as a forced related work.• The writeups themselves are more likely to be accessible to newcomers when it comes to understanding important questions in the field. It's worth it for people at a variety of seniority levels to do this, not just very new people. But for more senior people, this tends to look something like laying out a research agenda anyway.
Here's a (rushed, not great) example:
• Does reward hacking learned from RL also cause emergent misalignment?• Although I think this question became quite prominent from MacDi