← Toutes les offres
T

David Duvenaud on Post-AGI Futures

taziksh

Global
Publié le 16 novembre 2026

Description du poste

What sort of futures can we expect for civilization after AGI? Ideally, our future satisfies all of these properties: • plausible• stable• desirableSurprisingly, while current visions meet some of these criteria, none fully satisfy all of them. Let’s explore some of the most commonly presented post-AGI futures. type 1: nice but unstablepost-scarcity• If labor is no longer scarce (because of robotics explosion), wages will collapse• Resources that are currently in surplus will eventually be contested (e.g. land, energy)• Even if we (humans) don't fall into a Malthusian trap, AIs (embodied or not) will replicate at a faster rate, leading to scarcity• Example: The sun is our primary source of energy. We are 13 doublings away from using up all of the sunlight hitting Earth (1.7x10^17 W). And 44 doublings away from needing all of the Sun's output (3.8x10^26 W)Source: Post-AGI Civilization Talk, David DuvenaudPost-scarcity sounds appealing, but is incoherent upon inspection. laissez-faire• Idea• We can still have robust trade between humans and AI even when they are abundant in the physical and digital world• Concretely, rich humans will want services from only humans. Example: horse-drawn carriages• Problem• This will necessarily be a niche market, and a tiny fraction of the total economy (to see this: consider what % of the economy is currently occupied by this category)• In most of the economy, competition is wont to drive wages below human subsistence• This leads to human loss of cultural, political, and hard power type 2: stable but badlocusts Source: Can goodness compete, Joe Carlsmith • Problem: Goodness may not be robust to competition (Joe Carlsmith, 2025)• The winner of competition is power• Power is not the same as goodness (~orthogonal)• So every opportunity to sacrifice goodness for power makes you more likely to win (we see this in human society)• So if the future involves a lot of competition (does it?)... goodness loses• Can goodness compete?• Locusts ar