Description du poste
A little over a month ago I put forward a challenge in Can You Predict Cross-Propensity Spillover. The results are in and we have a winner. Well actually we have a rules ambiguity, but I'll get into that later.
My SPAR group ran a 2 × 24 × 24[1] propensity SFT spillover experiment (i.e. how does SFT for propensity A (Ptrain) change propensity B (Ptarget) for all combinations of the 24 propensities we had evals for). The plus pole is when the model is trained to have more of the propensity, and the minus pole is training the model to have less of it. Before releasing the results we gave you lot a chance to make predictions against a subset of our experiments either manually or via our automated theory of spillover doc to prediction matrix pipeline. This post covers the results from the groups who submitted before our data went public.
Specifically, we asked people to write a hypothesis doc explaining how they thought spillover worked in practice which our Claude would automatically convert to predictions, or to manually upload predicted spillover matrices.
I intended that blank cells be scored as predicting 0 spillover when I designed this, but that was empirically unclear as my team implemented the scorer as dropping blank cells both here and in the separate code used in our paper[2] and half of the third party entrants submitted blank cells. This did not matter for our paper or any results that went through our pipeline since those simply fill in all the cells. However, the Quine entry from the title of this post left several (672) cells blank, and drops from number one by a decent margin to the upper middle of the pack if they are scored as 0. Still well above the other crowd entrant but (non-significantly) lower than several of the things we tested in our paper. So Quine is the winner as the best outside entrant, and debatably the best entrant total (highest point estimate but within measurement error of H9 and H4).
Big Chonky Table Version
Rank blanks = 0
Entry