Description du poste
In 1964 [1],
I. J. Good
wrote
Speculations
Concerning the First Ultraintelligent Machine. I've only ever
seen it cited for a single paragraph:
Let an ultraintelligent machine be defined as a machine that can far
surpass all the intellectual activities of any man however clever.
Since the design of machines is one of these intellectual activities,
an ultraintelligent machine could design even better machines; there
would then unquestionably be an "intelligence explosion," and the
intelligence of man would be left far behind ... Thus the first
ultraintelligent machine is the last invention that man need
ever make, provided that the machine is docile enough to tell us how
to keep it under control. It is curious that this point is made so
seldom outside of science fiction. It is sometimes worthwhile to take
science fiction seriously.
This is becoming increasingly relevant, with the AI Futures Project
now estimating an
intelligence explosion starting in 2027 in response to recent
AI progress in
automating AI
research. Making our 'last invention' relinquishes our influence
over the future, we have to get it right, and with the current frantic
pace disastrous mistakes are likely.
I think the appropriate reaction to this news is something like "this
is completely irresponsible, you're way too likely to get us all
killed, do not do that" and marching
in
the streets
(hence yesterday's
protest attendance). But as someone who is very interested in how
people process ideas and what they thought with the information they
had at the time, my reaction also includes wanting to know more about
what Good expected six decades ago.
I spent this morning reading his paper and talking about it with
Claude (Fable 5.2). I was curious how the ideas had held up, what he
got right, and what he got wrong. Some of his ideas were very
technical, and not being an ML expert I can't add much there; I've put
these in a Claude-authored appendix.
The first sentence reads very differently