Richland Technology
Richland Technology
← Back to Blog
llmopen-sourceagentsresearchinterpretability

Two GGUF checkpoints fix a Spark-X2.6 reasoning loop by editing a single output-head row

A minimum-norm edit of ~2,000–2,500 weights in the unembedding row cuts a "but wait…" loop rate by 58–93% with no training run.

Richland AI Desk·14 November 2026·4 min read
Two GGUF checkpoints fix a Spark-X2.6 reasoning loop by editing a single output-head row

Two GGUF checkpoints fix a reasoning-loop failure in the small Spark-X2.6 models by editing a single row of the output head — no training run, roughly 0.00006% of the 4B's weights. The diagnosis is the interesting part.

What shipped

The models are graydini/Spark-X2.6-4B-heretic-noloop-headrow-GGUF and graydini/Spark-X2.6-1.7B-noloop-headrow-GGUF (4B, 1.7B, both Apache-2.0, published 2026-11-14). They derive from the Spark-X2.6 base (XHToken) and, on the 4B, a "heretic" decensored finetune. The author describes the failure as a "but wait…" spiral: the model reaches the correct answer, then refuses to commit, re-derives the same conclusion in different words, re-lists its tools, and never acts or stops. In an agent setting that is fatal — the model knows the answer and still won't pull the trigger.

The fix is a surgical weight edit, not fine-tuning. On the 4B it changes 2,560 weights (one row of the tied output head); on the 1.7B, 2,048 (untied head, so input embeddings are untouched). Every other byte is claimed bit-identical to the upstream quant. The repos ship a CPU-only apply_edit.py, the delta files, a ggml re-encoder, and an edit_report.json with source/output SHA-256 hashes so the checkpoint can be reproduced offline.

Why it matters

Because the head is a linear readout — logit(t) = emb[t] · h on the tied 4B, output_weight[t] · h on the untied 1.7B, where h is the final-norm residual — the author could attribute the failure to a specific place. Activation analysis at the moment the model reaches a conclusion but before it loops showed the closure token <​/think> at rank ≈ 54 with log p ≈ −11.8 (P ≈ 7×10⁻⁶), while the restart token " But" was top at P ≈ 0.45. That ~11-nat deficit is a pure head-readout gap, not an MLP "loop direction."

An earlier attempt to ablate an MLP down_proj circling direction was "correlated-but-not-causal": ±sign symmetric, max +2.85 nats — too weak. The fix instead solves for a minimum-norm row delta, δ = Hᵀ(HHᵀ)⁻¹(Δ·1), that raises δ·h_p by ~14 nats at the conclusion states (‖δ‖ ≈ 0.2, roughly 15% of that one row's norm). Because the edit is applied only along the conclusion residual directions, it is meant to be conclusion-selective rather than truncating legitimate reasoning.

Reported loop-rate results on a controlled branch-point benchmark (4 prompts × 12 seeds, n=48 per model):

ModelBeforeAfterReduction
4B0.7500.31258.4%
1.7B0.8750.06292.9%

The tool-ambiguity prompts dropped to 0/12 on both models; the harder math prompts improved but did not fully clear. Capability evals were roughly neutral on the 4B (mean −0.002) and slightly better on the 1.7B (mean +0.047), where GSM8K went 0.200 → 0.433.

This is a clean instance of a thesis building across several recent results: behavior is linearly decodable in the final hidden state and editable in the unembedding. The closest published precedent is HeadEdit (arXiv:2610.01170), which calibrates behavior through the unembedding matrix via a low-rank behavioral subspace — though it does a vocabulary-wide correction rather than a single-row min-norm delta. Related work on weight-space programming points the same direction.

Caveats

Everything above is author-reported from the model cards and edit_report.json; no third party has independently reproduced the numbers or the bit-for-bit claim. Several things keep this from being oversold:

  • The author's own materials disagree. The Reddit title/description claims "up to 97% reduction" on the 1.7B, but the model card and edit_report.json show 92.9%. The 97% figure is unexplained.
  • Small samples. The loop benchmark is n=48 per model; capability evals are n=100 (MMLU 91, GSM8K 60). The 1.7B GSM8K jump of +0.233 on n=60 is a large swing that could be small-sample noise.
  • The fix is narrow. The author states it targets the thinking loop only: agent tool looping can still occur on broader tool-use tasks, and arithmetic re-derivation termination is not solved. An EOS-row boost was tested and rejected because it regressed some tool/clean seeds.
  • "Conclusion-selective" rests on four loop prompts. The selectivity claim is supported by a handful of early-vs-conclusion P(close) measurements, not a broad sweep.
  • Novelty is unverified. The single-row output-head fix appears new, but the Reddit comment thread (potential independent discussion) was inaccessible, and no independent verification was found.

The takeaway: a single row of the output head — about 0.00006% of the 4B's weights — appears to encode much of the "commit and stop" decision, and a minimum-norm edit of that row suppresses a reasoning-loop failure with no training. That is a useful diagnostic pattern for anyone fighting agent loops, even before the numbers are independently confirmed.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.