OpenBonsai is the first open pipeline for Prism's ternary GGUFs — your own BF16 fine-tune down to a ~2 GB PTQ1_0 file

OpenBonsai, released this week under Apache-2.0, is — as far as its author knows — the first public pipeline that compresses your own BF16 model or fine-tune into Prism ML's PTQ1_0 ternary GGUF format. It turns Qwen3.5-9B into a 1,997,486,976-byte (~2 GB) file that runs in Prism's llama.cpp fork. Prism ships ternary weights and a runtime but, the author says, never published its BF16-to-ternary compression pipeline, so until now you could fine-tune Prism's models but not compress your own into the format.

What shipped

The repo is github.com/esokullu/openbonsai, from the developer behind the open-source WebBrain browser agent. His announcement post states the goal of running as much intelligence as possible locally, ideally through WebGPU. The compressed Qwen3.5-9B model is on Hugging Face.

The pipeline, documented in the repo's method notes, combines:

1. Signed Hadamard rotation — a ±1 sign vector per input width, then a blockwise Hadamard transform (blocks of 4096) that spreads weight outliers across the matrix so the ternary grid can represent them.
2. Sequential GPTQ rounding to codes {-1, 0, +1} in groups of 128, with one FP16 scale per group.
3. Self-generated calibration traces — the AYOT idea from Intel Labs China's "Attend to Your Own Thoughts": the model's own reasoning traces and answers, not generic text, serve as calibration context.
4. Scale distillation — with codes frozen, the group scales are fit against the original BF16 model's top-64 next-token probabilities.
5. Export to Prism's PTQ1_0 GGUF — 28 bytes per 128 weights (~1.75 bits/weight, scale included), with the same tensor layout as Prism's own Ternary-Bonsai-2-9B.

The dev-set numbers show each stage doing real work. The author's NLL ladder: BF16 2.423 → plain ternary 10.20 → rotated+GPTQ 3.32 → all-ternary 3.480 → after distillation 2.892. ARC-Easy goes from 13/20 to 19/20 after distillation. The GPTQ step alone is fast — roughly 10 minutes on the 9B model — and the full reasoning-code recipe takes about 5 hours end to end on one RTX 5090 (32 GB).

On the author's own PyTorch harness (thinking off), the compressed model scores GSM8K 65/100 and HumanEval 34/50. The original BF16 model scores 77/100 and 44/50; Prism's Ternary-Bonsai-2-9B scores 77/100 and 45/50 in the same harness. With thinking on, it's 32/50 and 14/25 versus 36/50 and 22/25 (BF16) and 46/50 and 24/25 (Prism).

The pipeline generalizes: the same recipe compressed Qwen3-8B into a 1,798,918,592-byte GGUF with no code changes, scoring GSM8K 70/100 and HumanEval 22/50 (thinking off) versus the BF16 model's 88/100 and 41/50. The author also quantized the embedding and LM head rather than leaving large FP16 tensors in the file — all 202 large matrices in the 9B GGUF are ternary.

Why it matters

The useful thing here is not the specific model — Prism's models are still better — but that the format is now writable. Prism's Bonsai 2 family (2B/4B/9B at 0.43/0.95/2 GB) and its PTQ1_0 format are attractive for local inference: ~1.78 bits/weight, 262k context, and a 9B model that fits in 2 GB. But the compression side was a black box. OpenBonsai makes the whole BF16-to-PTQ1_0 path reproducible — useful for anyone who wants to ship a fine-tune in this format rather than a base model, the exact case Prism's release doesn't cover.

The mechanism is worth understanding if you're doing low-bit quantization yourself. Ternary weights can't represent outliers, so the Hadamard rotation is what makes the format viable. Sequential GPTQ compensates errors one group at a time. The two calibration choices — self-generated reasoning traces and distillation against the BF16 output distribution — close most of the remaining gap. The author's ablations quantify it: switching from generic calibration data to self-generated math/code traces moved HumanEval from 6/50 to 19/50; adding more code-focused traces pushed it to 34/50. Distillation is worth the 3.48 → 2.89 NLL jump.

There's a second, quieter point: the release pre-registers its own failure checks. The model card admits the headline HumanEval score is contaminated — the setting was picked from four runs scored on the same 50 problems, and at least six of those tasks are restated in the training traces. It also reports a pre-registered fresh-problem check (62 vs 45 of 114, +14.9 points with a 95% CI of 5.7–24.1) instead of hiding it.

Caveats

Everything above is the author's own measurement, and the limits are real:

• Prism's official models still win. On the author's harness, Ternary-Bonsai-2-9B beats OpenBonsai's model on every reasoning and coding row. Prism's own published GSM8K for Bonsai 2 9B is 95.07 in its eval setup — not comparable to the harness numbers here, but a reminder that the open pipeline is catching up, not ahead.
• The HumanEval number is optimistic. The 34/50 was selected from four runs on the same 50 problems, and six of those tasks appear in the distillation data. The author's fresh-problem mitigation narrows the gap but doesn't eliminate it; no independent replication is cited in the repo.
• Hardware and runtime constraints. Compression needs a 32 GB GPU (28.7 GB peak during trace generation); smaller cards are untested. PTQ1_0 runs only in Prism's llama.cpp fork — upstream llama.cpp supports only the Q2_0 variant, and WebGPU isn't supported yet. Tied-embedding models (Qwen3.5 0.8B/2B/4B) can't be exported yet.
• Adoption is near zero. The repo is three days old with a handful of stars and ~200 model downloads. Treat the numbers as a promising single data point, not a validated recipe.

The author's own docs give the honest summary: distillation closes most of the gap to the BF16 model on dev NLL (2.89 vs 2.42; the shipped model measures 3.012), but the result still loses to Prism's models on reasoning and code. The one component that failed in testing — ScaleQ/CAT-Q, the technique the project builds on — got shelved in favor of GPTQ plus distillation. That's the kind of transparency that makes the pipeline worth reading even where the weights aren't yet competitive.

Sources

• OpenBonsai repository (README, method, model docs)
• OpenBonsai Qwen3.5-9B model card
• OpenBonsai project site
• Reddit announcement (r/LocalLLaMA)
• Intel Labs, "Attend to Your Own Thoughts" (arXiv:2608.01078)
• Prism ML, Ternary Bonsai 2 9B docs
• Prism ML, Bonsai 2 family announcement
• WebBrain browser agent