Power — a guide for the perplexed

Whether AI systems could take power from humans matters a great deal for how much risk we face from them. Answering that requires a correct picture of how power works, which is what this guide tries to give. In short, power rests on self-reinforcing expectations. A ruler has power if everyone expects everyone else to enforce their commands, even if this is bad for almost everyone involved. But expectations can also shift suddenly, and when they do, change is rapid and chaotic. This can sometimes allow small, coordinated groups to seize control over much larger societies. Almost any rule can enforce itself Consider the classic tragedy of the commons. A village has a common green where everyone can graze their cows. Each villager gains from grazing as much as they can, but if everyone does, the grass dies and the green becomes useless. One solution is to privatize the green. But many societies have solved the problem differently, by punishing anyone who grazes more than custom allows. Their neighbors stop talking to them, trading with them, and marrying their children. In a village where everyone depends on everyone else, being shunned is very costly. Shunning costs the shunner too, who loses a trading partner, a guest, or a match for their child. The solution is to treat failing to shun as a violation in its own right. If Alice overgrazes, everyone shuns Alice. If Bob helps Alice out, everyone shuns Bob. No one wants to be shunned, so no one breaks the rules. Notice that nothing here depended on the rule being a good one. A rule that only cows wearing bells may graze, or that only Charlie may graze at all, could be held in place the same way. All that's needed is that everyone prefers following the rule to being shunned. If we look at human societies, we see many examples of these self-enforcing rules. Some are positive, like the grazing rules on the green, or a norm that young men fight to defend their community rather than fleeing. Some are neutral, like wear