A Taxonomy of Agency

Epistemic status: a rough write-up of a taxonomy of agency I developed during my PIBBSS fellowship this summer. A few people have since asked me to write it up, so this is mainly so I have something to point to; it is exploratory rather than a finished theory. This post sits in the middle of a small set. In Agency is not a natural kind I argued that agency is a construct we build to make sense of messy systems. My post on different lenses of agency provides a deeper exploration of some of the alternative directions of alignment I mention at the end. I also wrote a short paper on unsupervised value discovery reporting some experiments the taxonomy led me to undertake. Why we need a new taxonomy of agencyA lot of the discourse around agents in AI safety starts from the idea that they are something like 'goal-directed systems'. Taxonomies of agents tend to be hierarchical, like Russell and Norvig’s textbook progression from simple reflex agents, through goal-based agents, to utility-based agents. Goal-directed systems are seen as striving towards utility maximization and this is presented as the ideal type, compared with which real-world agents are better or worse approximations. This idea leads naturally, almost tautologically, to the view that alignment is about specifying agents’ goals. This is a helpful approach; decision theory gives us a clear vocabulary and a useful tool set for alignment. However, one shouldn’t jump from having a useful tool set to assuming the world is made out of Meccano. Three things about these taxonomies particularly bother me: • They make us obsess over edge cases. A common reaction when I told people what I was working on this summer would be: so is a cell an agent? A heating circuit? A corporation? The biosphere? The naturalness with which we reach for sharp lines to divide these things suggests a flawed ontology. I’m much less interested in how to divide up these cases than in what is going on within them, and hence within us and AIs.•