Description du poste
In a research post, Cunningham et al. of METR propose the “‘expenditure horizon’ over an optimization problem as the dollar value at which the improvement to the goal metric is equal to the improvement by a human with the same budget.” Though their work focuses on individual optimization problems (with NanoGPT as the main subject), the expenditure horizon can be meaningful for any task distribution: it’s the budget at which human-only and AI-only work reach the same performance.
Though I like expenditure horizons as an operationalization of AI capability, I have a few critiques which suggest an improved metric, which I’ll call the “X% expenditure share horizon”: at what labor budget is it optimal to spend X% of the budget on agents rather than humans?
In this essay, I’ll explain why this metric is both more useful and likely easier to measure than the expenditure horizon. I think this is an area that’s ripe for concrete follow-up work.
Figure 1. Performance as a function of expenditure on humans or agents. Source: METR.
Pros and Cons of the Expenditure Horizon MetricLet me begin by saying a bit about what I like about the expenditure horizon metric. First, as we brace for the economic impacts of AI, I think it’s good to begin proposing measures of AI capability that are tied to the human cost of comparable work, and the expenditure horizon takes a shot at this. Another metric of this kind that I am partial to is “autoflation” from the Remote Labor Index: “The percentage decrease in the cost of completing [a] fixed [task] bundle when using the cheapest-possible method to complete each [task] (human [or AI]).”
Second, as Cunningham et al. note, the expenditure horizon implicitly captures how agent performance varies with test-time compute. This is good, since AIs should no longer be understood to have a single “score” on any task. Older metrics like original time horizons don’t have a built-in way to indicate task budgets.
Finally, I like that the expenditure horizon