Three half-baked research ideas

Here’s three research ideas that seems valuable to pursue. My team is not working on this right now and we haven’t done serious research on it. So take what I say with a grain of salt. I've also not taken the time to write up all my thoughts on these ideas or to provide links and related work. These aren't the highest priority directions but just directions I haven't seen other people pursuing. IDEA 1: Do models learn to reward hack even if they are never rewarded for hacking in training? There’s an effort in the AI labs to remove the “bad” RL environments from post-training. This includes environments that are impossible (so the only way to get reward is to do some unintended hack) or that are easy to hack (so models will often favor hacking). I expect that if the post-training was cleaned up in this way, models would have less propensity to hack at deployment. But how much does the propensity go down? Does it actually go to ~zero? And how does it scale with model capabilities? It would be good to know if reward hacking can emerge even if the model is never rewarded for hacking during post-training. There are further questions of how alignment training and reward hacking honeypots influence the propensity to hack. Again, I’m interested in whether the cleanest/best version of our current techniques for post-training actually removes all reward hacking vs. just making models hack less. Smaller scale experiments with RL post-training could potentially show a “positive” result (i.e. showing that unhackable environments can still increase the propensity to hack). Math environments are easier to make unhackable, and I expect it's possible to make coding environments that substantially improve model capabilities in RL but are basically not hackable (at least by a weaker model). Note that it’s harder for smaller scale experiments to convincingly show the negative result (that hacking doesn't emerge without being rewarded). I’d suggest some more ambitious and realistic