Description du poste
Epistemic status: I am an AI engineer, not an AI researcher. I like reading papers and following debates, but I had never posted on LessWrong because I had nothing new to bring. I think I finally do. The idea is untested, and the literature review was done with the help of LLMs.
TL;DR: Humans seem to learn in two ways. Punishment without explanation teaches us not to get caught. A story we tell ourselves about why something matters gives us values we keep even when nobody is watching. Reinforcement learning only gives AI the first. I propose a way to add the second, and an experiment to test it.
Current situationA large part of how AI models are shaped after pretraining is reinforcement learning: if the AI does well according to a grader, the behaviour gets reinforced, if it does not: the behaviour is discouraged. This relies on an automatic grader that scores many dimensions: honesty, efficacy, length, adherence to the values etc.
The problem with this approach is that reinforcement learning tends to create reward-hackers: the AI will learn how to trick the grader instead of internalising the core values.
My initial intuitionThis got me into thinking that humans are actually similar but not completely. There is definitely some reward hacking in humans. But there are also deep core values where we don’t try to cheat a grader.
Before diving into this, let's illustrate the opposition between Trick the grader and Deep core values in humans with 3 examples.
Example 1:
• Trick the grader: In my AI engineer job, the analog of the grader would be my boss. He wouldn’t be happy that I play chess on my work hours. But I do it anyway because I understood how to trick the grader: I know when he’s looking and when he’s not.• Core values: I am very concerned with animal suffering. This is a value that I hold for itself and at no point am I trying to trick any grader in this case.Example 2:
• Trick the grader: I don’t try to steal money from rich people because I am scared of be