Some kinds of selection are safer than others

On monitors, rejection sampling, and default-robustness. Note: The main ideas of this blogpost have been discussed before. This post repackages those ideas and explores some implications. Not all kinds of selection are created equal. One kind of selection is performing RL on a model to act a certain way. This selects for the cognitive processes within the model that promote that action. Another kind is training many separate models and choosing the one that acts the best way. Take two coding models: Model A and Model B. We want to train these models to be competent, honest coders that do not cheat. We train Model A with the RL reward function “pass these test cases while the chain of thought contains no indication of cheating”. By the end of training, Model A will probably still cheat, but simply learn to not mention it in its reasoning trace. For Model B, we train the model with no penalty for mentioning cheating in its reasoning trace. Then, after training, we apply a monitor that detects whether “cheating” is mentioned in the CoT. In the first iteration, the model learns to cheat, and actively mentions “cheating”. However, we repeat this process, starting fresh each time and tuning some things like the prompt, and perform RL over and over. Eventually, we get a model that passes all the test cases and happens to not mention cheating in its reasoning trace. By the end of this, Model B is probably less likely to cheat than Model A. Model B faces no direct RL pressure to become deceptive, since there is no RL penalty for mentioning “cheating”. Instead, there is direct RL pressure to 1) solve the task honestly, and 2) solve the task by cheating. All the Model B variants that yield to (2) are discarded, leaving the first variant that yields to (1). There is no direct selection to be deceptive; there is only indirect selection to do this, and this indirect selection is probably too weak. Traditional ML understands this: Instead of training on all our data, we keep some