What Can AI Welfare Buy Without Consciousness?

Lightly edited transcript of a talk given at NYU AI Safety, November 9, 2026. Recording available at https://youtu.be/6WpRfW1d4CM Joseph Wright of Derby, Public domain, via Wikimedia Commons IntroductionWhat can we buy while the question of AI consciousness stays open? My answer turns on a distinction between duties to a model and duties with respect to a model. These run through at least two relations, with society and with particular people. I'll try to show that they justify more measures than it seems at first glance, and plausibly more than a low precautionary principle would. Consider a case. A model keeps failing its tests, and a representation of desperation rises, a representation of calmness falls, and as that happens, the model cheats more and more. Should we do something about it? There's at least two possible reasons for saying yes. One, the model may be suffering, and second, the states make it less safe. Now note that the first one requires, of course, that the model is a thing that can suffer, and that is the entire question that I'm going to try to set aside. The second needs no such assumption, so I'm going to follow reasons akin to the second here. But let me motivate it a little more. The first reason sends us to the debate of consciousness, and that's where most of the debate stands. It may be at a stalemate for a while. Skeptics disagree about whether AI systems are or even could be conscious. But both skeptics and advocates agree that consciousness and sentience are the clearest grounds of patienthood. Moreover, people like Seth argue that caring about non-conscious systems would distort the circle of moral concern. The question may stay open for a long time. Eric Schwitzgebel says, quite bluntly, the experts do not know, you do not know, and all is fog. And his stronger claim is that we will not know before we have built very many systems whose consciousness is disputable. The disagreement is wide. At one end, Seth defends biological n