Instrumental convergence is incoherent

Crossposted from the EA forum ‘Like most of life’s dangers, the dangers of artificial intelligence are not easily identified from the armchair.’ J Dmitri Gallow Epistemic statusI’m not an expert in this field, and herein I make some sweeping generalisations educated guesses about the overall nature of its contents based on my strong impression from the subset of work that I’ve read. I’ve recently spent maybe 2–300 hours learning about modern AI safety technical work, and some amount of spare time over the years before that thinking about here-relevant notions such as preferences from an analytic-philosophical perspective. I intend this post as a work in progress. TL;DR:• The concept of instrumental convergence comes from a reference class of pre-empirical categorical predictions with a very low success rate.• In historical usage, its definition is very murky. Perhaps too murky to really say that it made any meaningful real-world predictions. I suspect its definition is fundamentally unfixable.• On my best interpretation of what it could be said to have predicted, it seems to be much worse than we would want a good high-level model of AI behaviour to be.• Despite these shortcomings, it still seems to be widely internalised by the EA and AI safety communities, seemingly carrying a lot of implicit weight in predictions of AI takeover. I review and criticize recent usage across the AI safety community, and in the case of the Hugging Face attack.• I argue for replacing it with a more grounded framework, and I hint at what such frameworks might look like.• I conclude that overindexing on instrumental convergence-derived scenarios might be substantially reducing the EA movement’s positive impact - or even, by justifying warlike and nationalistic sentiment, causing counterfactual harm Intro What if LLMs are less this… … and more this?[1] ‘Instrumental convergence’ (IC) is an intuitive thesis that seem