Description du poste
Recently the Conceptual Reasoning Capabilities (CRC) team at Redwood released a post arguing that we should accelerate AI’s conceptual reasoning abilities. This post was very controversial. In particular, people are worried that improving AI’s conceptual reasoning abilities will accelerate both their ability to do ML research and the capabilities they need for takeover.
I am going to address just one chunk of this controversy: how much does training conceptual reasoning on AI safety questions translate to general ML research abilities? As a reference point for conceptual reasoning training, we can use the example of LMCA questions. These questions measure conceptual argumentation by having models rate counterarguments made against a claim. Counterarguments are rated on axes like strength, clarity, and centrality. The claims and counterarguments are in a variety of conceptual domains, including AI safety and philosophy. Using LMCA, I can provide a more specific operationalization of the question I’ll be tackling: if you train on LMCA-style tasks where all the arguments are AI safety relevant until you get a factor of X increase on automated AI safety work, what factor increase do you get on automated ML research?
I am going to tackle this question by breaking it down into what I consider to be the primary considerations and quantifying the contribution of each. I am hoping this approach will make the discourse easier to navigate and help readers identify the most important cruxes. So far, I have found this controversy hard to pin down: good points are being made on both sides, but I don’t know how much weight anyone gives to any particular point. Thus, my numbers are meant to make it easier for people to see exactly where I stand on each subpoint (even though I have large amounts of uncertainty).
My breakdown centers on three considerations that I identify as the primary reasons conceptual reasoning training would improve AI safety skills more than ML research skills