Desiderata for the Theory of Agents

TLDR: I give informal conceptual desiderata for the theory of agents we need to solve ASI alignment, and show a few examples of how existing theories fall short of these desiderata. A Theory of What? The word "agent" is often used in the AI safety discourse and in other research communities, but the intended meaning and scope of this concept varies substantially. For the purpose of this sequence, I informally define an agent to be a system with inputs ("observations") and outputs ("actions") that is effective at steering its environment in the direction of particular preferences, for a large variety of different environments. To do so, it arguably has to create a sophisticated model of the environment [1] , and use this model to build a long-term plan. Moreover, it has to continually update the model and revise the plan with new information and additional processing time. Additionally, it has to accomplish all of that while making efficient use of the available information (statistical efficiency) and computing resources (computational efficiency). The main examples of agents in the context of ASI alignment are (i) humans, and in particular users or stakeholders; (ii) the unaligned ASI we are trying to avoid or counteract; and (iii) the aligned ASI we are trying to create (such as the GDS). Rocks, bacteria, and thermostats are non-examples: while a bacterium or a thermostat can be regarded as steering the environment, both are very limited in the variety of patterns in the environment they can exploit. For instance, if you switch the "cool" and "heat" outputs of a thermostat, it will not understand what has gone wrong and adapt its behavior. Animals and contemporary AIs are marginal cases (it's not obvious whether they should count). Given that agents are at the core of both the threat and the proposed solution, the theoretical foundation for ASI alignment requires a theory of agents. Now, concepts adjacent to "agent" have a considerable history of acad