Description du poste
If we want to ensure that AI systems are safe, should we approach the problem like an engineer, or a scientist? Do we need to discover deep and fundamental truths about how intelligence works, or is it better to focus our efforts on applying the battle-tested methods developed by safety engineers over the course of decades? As others have noted, this question is beginning to become a crux of disagreement between those who are concerned about the risks posed by advanced AI systems, and those who are relatively confident that humanity will muddle through.
In this post I argue that it isn't clear how to use these safety engineering methods, or whether they work for advanced AI systems, because you immediately run into thorny scientific problems when you try to apply them. Furthermore, these thorny scientific problems are the same ones being studied under the banner of Agent Foundations.
I'll start by laying out my understanding of how safety assurance typically works. Here, I'm drawing from what I've learnt as a PhD student embedded in the University of York's safety assurance research community. Then, I'll try to apply that framework to a highly capable AI system, and discuss some problems that come up.
How Does Safety Assurance Work?Safety assurance is the process of a designer convincing themselves, and other stakeholders, that their system is acceptably safe. It involves not just crafting an argument to that effect, but also gathering the evidence which supports that argument, assessing the likelihood and severity of risks, and the development of risk mitigations. Also, the process doesn't begin when the designer is about ready to deploy, it begins right at the start of development, and it doesn't stop until the system is decommissioned.
The first step is to get clarity on what we're assuring the safety of. This is done by drawing what is known as a system boundary. We put all the parts of the world that we consider to be our system – usually parts that the designe