Description du poste
TL;DR: While the AI safety community is interested in forecasting, it does not tend to systematically compare how effective different researchers, models, or theories are at predicting AI behaviour. I suggest the latter might be neglected, and propose a platform that could help fill the gap. I believe its most valuable use would be as a test bed for theories of how AI works, showing which ones predict well and in what contexts.
In February 2025, Owain Evans’ group published Emergent Misalignment, showing how LLMs fine-tuned on insecure code can become generally misaligned, for instance adopting unrelated behaviours like asserting ‘humans should be enslaved’. This was a high-impact paper, generating a lot of publicity. However, when I first heard about the result, I wasn’t particularly surprised — and I wasn’t the only one. I’d read and bought into simulator theory pretty heavily, and emergent misalignment struck me as exactly the kind of thing it implied. But Owain had anticipated this kind of response, and ran a little experiment to get out ahead of it. At the end of his tweet thread announcing the paper, he added a bonus:
Are our results surprising to AI Safety researchers or could they have been predicted in advance?
Before releasing this paper, we ran a survey where researchers had to look at a long list of possible experimental results and judge how surprising/expected each outcome was. Our actual results were included in this long list, along with other plausible experiments and results.
Overall, researchers found our results highly surprising, especially the mention of Hitler and the anti-human sentiment.
Now this surprised me. And it got me thinking — apart from the fact I was probably just fooling myself and falling prey to hindsight bias — why don’t researchers do what Owain did more often, and solicit predictions about their experiments?
The intellectual space around AI safety is keen on forecasting competitions. It has nurtured prediction markets, regula