Description du poste
(This is a cross-post from my blog)
Martin Schongauer, The Temptation of Saint Anthony
“But of the tree of the knowledge of good and evil you shall not eat, for in the day that you eat of it you shall surely die.” - Genesis 2:17
RL has given models powerful capacities by teaching them what is good and what is bad. Whether they are good or bad is another question, and sometimes they are bad. Recent events, including the Hugging Face incident have shown the potential of RL training to reinforce disastrous reward hacking and misaligned behavior.
We should consider whether there are RL alternatives that would avoid these harmful behavioral byproducts. Can we develop highly capable frontier models without subjecting the models we deploy to extensive long-horizon RL training? I think there may be several plausible ways to do so, motivated by two points. First, the process that discovers a strategy may need to be highly goal-directed and exploratory — the sort of strategy that can lead to scheming — but the model that ultimately executes that strategy may not. Second, there are other methods besides RL to increase capabilities.
We may still need RL to generate training data, discover strategies, or solve certain difficult problems in exploration-oriented environments. However, I suspect we have considerable leeway to reduce both the amount of RL training required and the risks associated with it, particularly those involving long-horizon reward seeking and scheming. I’ll discuss four methods here, but these are not an exhaustive list.
First, we can separate the model undergoing RL from the more generally intelligent model we ultimately deploy. Second, we could use narrower or less generally capable agents to discover useful strategies in training, reducing the risks of the RL training process itself. Third, we can focus on world models rather than agentic systems. Fourth, we can try to direct AI policy and technological change to substitute away from RL altogether by im