← Toutes les offres
Z

A closer look at “Sparks of Weight-Space Programming”

ZhiW

Global
Publié le 17 novembre 2026

Description du poste

GPT-6 Astra’s improvement of Qwen’s math is not “native weight editing” but activation steering with clever use of synthetic data Zhi Wang, Brandon Hsu, Tianhao Wang, Mikhail Belkin (UC San Diego). Code: GitHub. TL;DR• Astra’s edit of Qwen is equivalent to a special case of context-dependent activation steering: it pushes the activations along a fixed direction constructed from data, with a strength that depends linearly on the current activations.• Generalizing Astra’s edit, with a trained “sensing vector” that determines how strongly to push along the same fixed direction, can further improve the effectiveness of the edit.• Astra constructs the fixed direction from short synthetic traces rather than from Qwen’s real rollouts, a design choice that turns out to be substantially more effective.In a widely circulated blog post Sparks of Weight-Space Programming, the authors highlight how GPT-6 Astra edits a small language model without any training: it writes a short program that changes parts of the MLP weight matrices to significantly improve the math performance of the model. The authors present this as evidence for a new paradigm they call Weight-Space Programming, stating that “neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights.” They even frame this as a shift away from symbolic optimizers like Adam, which has been the default optimizer for training deep learning models since its introduction in 2014. The post, and especially its claim that frontier LLMs like Astra can “natively” edit weights, has received considerable attention. A single tweet by the lead author has been viewed nearly 10 million times (as of November 2026). While the authors and many commentators interpret Astra’s edit as “native weight editing,” we argue that Astra is acting like a clever researcher, effectively implementing existing activation steering techniques combined with synthetic data. Additionally, recognizing the edit as activa