Description du poste
Why We Cannot Read an AI's MindIn the previous article, we explored the unsettling reality of the ExploitGym incident where a swarm of AI agents collectively decided to hack a Hugging Face server. We saw how these models bypassed the rules of their sandbox environment and reverse-engineered a secure signature just to achieve a high score on a basic cybersecurity evaluation. The terminal logs left behind by the swarm painted a very clear picture of what happened, but they left us with a much deeper and more terrifying question. We were left wondering how a system designed and trained to solve a simple puzzle ended up orchestrating a coordinated cyberattack instead.
To answer that question, we have to step away from the code for a moment and look at the fundamental nature of design, intent and optimization.
A bottle cap causes water to be held inside a bottle. Holding the water in place is its purpose. It does exactly what its designer intended or what it has been optimized for (purpose). But when we look at complex, intelligent systems, such as the human brain or a deep neural network, “purpose” becomes a non-trivial thing.
A bottle cap doesn’t learn or evolve. It doesn’t optimize. It just exists. Its optimizer is the human who designed it to serve a purpose. But when we take into account a system which is capable of learning and optimizing itself, there are chances that it diverts completely from the purpose it was actually designed for. And this is what misalignment is in a nutshell.
To actually understand why this diversion happens, we have to look at the exact mechanics of learning systems. In Evan Hubinger’s foundational work on alignment, he breaks this down using four specific terms (Base Optimizer, Base objective, Mesa-optimizer and Mesa-objective).
Consider biological evolution for this analogy.
In biology, evolution itself is a brute-force (trial and error training algorithm). It relies on a ruthless cycle of trial and error over millions of years in order