Description du poste
LLM Intelligence, measured by Starburst, has been basically linear over time since that start of the reasoning era.
By Chapin Lenthall-Cleary and Cole Gaboriault
In summer 2024, my friend Cole and I started a hobby research program into perceptions of (human) intelligence. We started by creating a bespoke 1-5 intelligence scale[1] and rating people we knew. Our plan was to then make a bunch of tests, give them to the people we rated, and see which tests, if any, correlated with our ratings.
One of those tests was Starburst. We’ve talked about it before; in short, it’s a puzzle game where one tries to figure out the laws of physics of a fictional universe. We never did get very many people to play it. But using the results of two early playthroughs (Cole’s and our friend Ryan’s), some further test playthroughs where we pretended to be people of certain intelligences, some a priori reasoning about which parts of the game would be difficult, and a healthy dose of raw intuition, we pieced together a correspondence between Starburst scores and our intelligence scale. Then Ryan suggested we give Starburst to LLMs.
Here's a plot of SOTA[2] model intelligence over time, measured using those Starburst correspondences, along with all four (emphatically not population-representative) human scores for reference (please reach out if you want to play and get us more data):
There's a lot notable here, including the general observations that progress has been rapid, but LLM performance still falls short of the most capable humans we've tested (let alone saturating the benchmark to the level of genius humans or an arbitrary superintelligence).
But the feature of that plot that I find by far most striking is that intelligence over time has been basically linear since the start of the reasoning era. Here, I'll add a line through it:
Okay, wtf?
Obviously, there's some noise and stair-stepping. But the broad trend has been basically linear from the release of gpt-4o (May '24) until now.