Description du poste
A friend of yours says they've had a flash of insight. A manga they've been into lately has helped them understand something deep and fundamental about decision theory. You aren't yet clear on the details when they rush off to work on "practical applications", but before long they return with their magnum opus:
"This hard drive contains the weights of an AI model that is both wildly superintelligent and completely safe!"
You feel as though you have good reasons to be pretty skeptical of this claim. This friend has terrible taste in manga, and you aren't convinced that they grasp the more subtle themes in the stuff they do read. You're about to shoo them away when, anticipating a dismissal, your friend pleads:
"What would convince you that I'm right?"
Now, if this friend wanted to convince you that they were wearing the Crunchyroll Store exclusive Solo Leveling Beru Tee, you could verify that claim directly, with the unaided senses. It's as simple as taking a look at their outfit. Assessing the safety of the AI model is not so simple. Observations of the hard drive are separated from the fact you want to know by a gulf of empirical and logical uncertainty. Whether or not the model is safe is dependent on the unpredictable behaviours produced by the model weights, and on the environment's responses to those behaviours. Since the AI model and the environment around it are so complex, we cannot anticipate the relevant dimensions of these phenomena without some kind of instrument that aids us in interpreting our observations. Without such an instrument, I would consider safety to be an unobservable variable.
You could ask your friend to provide you with an instrument that could be used to measure the AI model's safety. Maybe, this instrument is a safety case, where claims are written in natural language and presented diagrammatically. Or, maybe it ends up looking like a mathematical proof, formalised in Lean. In any case, the instrument is based on some theory that relat