Description du poste
The oversight problem consists of describing, eliciting, and understanding behavior. Prior work in black-box behavior investigation and interpretability seek to construct compact, faithful, and informative theories that explain model behavior. We propose training investigator agents and subject models to emit explicit theories, and using these theories as common currency for oversight across models. We hope to create a complementary mode of oversight to weight-space and model-specific approaches by studying the transfer properties of theories across model families. This post outlines our positioning and initial research agenda built on top of a few recent publications.
Oversight as Theory-BuildingTransluce is working on oversight-as-inference, training oversight foundation models on experiments on a particular subject model: “Any oversight question can be formalized as Bayesian inference over the outputs of a Pythonic world model.” The models are posttrained to explain behaviors in natural language. Their method is promising for overseeing a system in depth: expressing interventions as a primitive set such as get_activations, patch, fine_tune, decode, and sample_model is a powerful toolbox, and we are excited to see oversight assistants that can use them fluently. However, it needs to be redone for every subject model, which seems unfeasible across rapid releases in many different families, unless the adaptation can be done in a very compute-efficient manner.
We propose to approach oversight by training both the investigator agents and subject models to output good theories, or working beliefs about the environment they are operating in. The question is: can a theory learned from one set of models help predict or investigate a new model, in a way that is more efficient than subject-specific training? Unlike raw CoT, theories operate in belief space. Correct theories should aid the models in solving tasks instrumentally as well as being more amenable to oversight