Weight-Space Programming: GPT-6 Astra edits Qwen's weights directly — 0.016% of params, MATH-500 7.9%→48.1%

MIT CSAIL researchers had OpenAI's GPT-6 Astra read and edit another model's weights directly — no gradient descent, no training loop. A 0.016% edit to Qwen3.5-0.8B raised its MATH-500 score from 7.9% to 48.1% by removing a reasoning loop. The team calls the paradigm "Weight-Space Programming." An independent developer rebuilt the edit, ran it locally, and read what the model "thinks" at the token where it used to get stuck.

What shipped

In "Sparks of Weight-Space Programming", posted November 2, MIT CSAIL researchers Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab report eight experiments in which GPT-6 Astra reads, repairs, or writes neural-network weights directly — sometimes with no environment feedback at all. The headline is the Qwen fix.

Qwen3.5-0.8B has a known failure mode: it gets stuck re-checking an answer it has already written, repeating "Wait, $140 * 6 = $840." until it exhausts its token budget. In the authors' evaluation, 93% of its MATH-500 answers ran to the 32,768-token limit. Working in a Python kernel with up to 50 batches of forward-pass feedback on practice problems, Astra traced the loop to a single direction written by the MLP feed-forward layers in blocks 15 and 19, then subtracted that direction from 64 output columns in each layer.

The edit touched 131,072 weights, about 0.016% of the model. Under the authors' sampling settings (thinking on, temperature 0.6, presence penalty 1.5), the reported results were: MATH-500 7.9%→48.1%, GSM8K 13.8%→67.1%, IFEval 21.0%→62.3%, MATH-500 answers hitting token limit 93.0%→33.1%.

The fix generalizes beyond math: the loop is not math-specific, and IFEval roughly tripled. The authors released the edit manifest and logs on GitHub and the edited model on Hugging Face.

The other experiments show the same capability at smaller scale. Given only the 17,253 raw weights of a game-playing network as text, with no tools, Astra named which of four games each network was trained on (4 of 4 correct). It identified which shape three small diffusion models generate from weights alone, and which language three 7.2M-parameter character transformers were trained on — and named every copying (induction) head in each, matching the measured ground truth.

Given a stuck driving policy, a scene description, and one recorded failure episode, it rewrote 3.5% of the weights so the simulated car cleared a ramp in all 1,024 test runs, up from 0%. Asked to write game networks from scratch, it produced networks that beat policies trained with PPO for 100 million steps in all three MinAtar games, and a 9x9 Go network that beat an equal-sized PPO network.

Why it matters

The paper's claim is a shift in how weights get optimized. Today weights are written by symbolic optimizers like Adam or Muon, or by compilers from formal programs. Weight-Space Programming argues that a frontier LLM can reason about what each part of a network computes — or should compute — and change the numbers directly, in natural language, around a goal.

The authors propose two practical hooks: auditing a model for backdoors or hidden capabilities without running benchmarks, and patching a failure with a targeted edit that touches few weights and avoids the forgetting fine-tuning often causes. These are proposed applications, not demonstrated benefits — but a 0.016% edit that removes a loop is far smaller than a fine-tune.

The independent teardown makes the mechanism concrete. Developer alisorcorp rebuilt the edit from the released diff and confirmed it is exactly rank-one: on the 64 edited columns, the component along the edited direction ends at -5.0 times its original value, i.e. W - 6uu^T W — Astra's notes call it "top64 alpha6." The direction reads as doubt versus results: through a Jacobian lens, one end decodes to "Actually," "?!" and Chinese tokens meaning "or is it" and "not at all," the other to "results," "outcomes," "calculations."

At the token where the unedited model writes "Wait" again, the lens reads only "Wait" plus "Wrong," "Nope" and "Hmm" from about 60% depth on — even though the answer is already written. With the edit applied, the same token reads "Verification," "Completion" and "Result." The 64 edited neurons were the strongest doubt-writers inside loops, so flipping them turns the strongest doubt-writers into conclusion-writers.

The teardown also shows why plain removal isn't enough. Keeping the same direction and neurons and varying only the strength: at alpha 1 (plain removal), the model still puts 99% on another "Wait" inside a loop; at alpha 6 it drops to 63%, while "Wait" at checks that lead somewhere useful barely moves. Over 40 GSM8K problems, the unedited model looped on 29 and got 8 right; plain removal looped on 19 and got 13 right; Astra's edit looped on 4 and got 19 right. The edit also converts to GGUF cleanly — the base and edited bf16 files differ only in the two edited tensors — and holds at Q4_K_M, with 1 loop in 40.

Caveats

The headline number is vendor-reported and the framing is contested. Within a day, developers Matthew Honnibal and Sean Goedecke published critiques: the Qwen fix is the kind of Python analysis a human ML engineer would write, and Goedecke describes it as an application of steering, an existing technique. Honnibal's specific objection is that the post groups experiments with very different levels of tool access under one broad claim. The Qwen result sits at the tool-heavy end.

The most important caveat is what the edit actually does. The authors themselves frame it as removing an obstacle, not adding capability. Their follow-up on Gemma 3 270M, which rarely loops (8.5% of answers hit the token limit versus 93% for Qwen), produced no established gain — MATH-500 moved from 6.8% to at most 7.7%. A third-party analysis (AI Socratic) adds that an annotated comparison puts unedited Qwen at 49.45% MATH-500 with thinking disabled, slightly above the edited model's 48.1% — though that is not a controlled comparison. The Hugging Face model card for the edited checkpoint puts it plainly: "This removes a failure mode; it doesn't add capability."

The teardown's own numbers are small samples — single examples or 8-9 generations for the lens and dose readouts, 40 problems for each eval — and the dose test is teacher-forced on text the unedited model wrote. Scaling weight-space programming to frontier models remains unproven; the demonstrated edit targeted a 0.8B model. The whole result rests on GPT-6 Astra's behavior, which the authors report and for which no public end-to-end reproduction exists.

Sources

• Sparks of Weight-Space Programming (MIT CSAIL blog)
• R&D World: MIT team says GPT-6 Astra can fix an AI model by editing its weights; critics say the method was standard ML engineering
• alisorcorp: What Qwen is thinking when it gets stuck saying "Wait" (independent teardown)
• alisorcorp/inside-a-reasoning-loop (code)
• AI Socratic: LLMs can natively read & write neural network weights
• Hugging Face model card: Qwen3.5-0.8B-Astra-WeightEdit