Description du poste
The recent METR report on AI acceleration at Anthropic has led to several analyses and arguments for rapid takeoff. There are important hard-to-supervise tasks that are necessary to make progress in AI research, and there are arguments that AI progress on these will lag progress on easily supervised tasks (like optimizing performance). The section "Why a Staged Model for Research Taste" presents an argument for this claim. I worked with Fable 5.2 to build a model where AI progress comes from a three-step process:
Figure 1. AI Research Model
• Experiments produce results tested at small-scale, where judgment determines the quality of experiments• The best small-scale result is run at scale. transfer is the share of that gain that translates into experimental metrics at scale• "Real value" represents a hard-to-measure quantity (like model quality, innovation) that is the ultimate goal of the research, subject to a variable validity that determines how well the proxy metric transfers to real applicationsImprovements in both transfer and validity are types of research taste. The iterative feedback loop makes transfer a supervisable task that is easier to optimize with AI (e.g., as argued by Dwarkesh Patel). The difficulty in measurement, catching gaming, and need to test in real-world experiments makes validity a hard-to-supervise fuzzy task that is hard to optimize with AI (e.g., as argued by Cindy Wu). Tom Reed's "Goodhart Singularity" argues that "the relevant data for superintelligence in most non-coding domains will only become available through deployment of AI models throughout the economy."
This model is derived from the AI Futures Project Q3.5 Timelines Update. The most salient difference is that my baseline scenario has the AI's research taste for validity increase by 0.5 SD for each year of progress at the 2025 pace, unlike its research taste for transfer that increases by the same 3 SD per year the AI Futures model has. Like that model, it uses a fixed proje