Richland Technology
Richland Technology
← Back to Blog
quantizationllmopen-sourceresearchinference

BitMTP: multi-token prediction that survives 1.58-bit weights — 2.2–2.4× decode speedups on H20, with Apache-2.0 code

A single low-bit model drafts and verifies its own blocks; accepted-prefix alignment keeps CUDA-Graph replay consistent enough that speculative-style speedups hold on ternary backbones without a draft/verify pair.

Richland AI Desk·21 October 2026·5 min read
BitMTP: multi-token prediction that survives 1.58-bit weights — 2.2–2.4× decode speedups on H20, with Apache-2.0 code

Hong Kong Polytechnic University researchers have released BitMTP, an algorithm–system co-design that makes multi-token prediction (MTP) pay off on ternary-weight LLMs. Hybrid decoding reports 2.2× five-task mean decode speedups on BitNet b1.58-2B and 2.1–2.4× on 1.7B–4B Bonsai models on an NVIDIA H20 at comparable task quality, and the paper is accepted at NeurIPS 2026.

What shipped

BitMTP is a single-model framework, not another draft-model stack. The paper (arXiv:2610.28500, v1 posted 20 Oct 2026) trains one low-bit model with a joint next-token-prediction and MTP objective, then decodes with one of three policies — confidence-only, speculative-only, or hybrid — none of which use a separate draft or verifier model. The release is three layers deep:

  • Code: an Apache-2.0 repo with training code, packed ternary (t2-g128) CUDA kernels compiled for sm_90 (H20), sm_86 (RTX 3090), and sm_87 (Orin), a CPU-scored HumanEval+ harness in Docker, and pinned SHA-256-verified Hugging Face checkpoints (BitMTP-Bonsai-1.7B SFT / D4 / D8, plus a 4B SFT/D8 pair that runs on H20 only).
  • Numbers (all author-reported): on H20 with hybrid decoding, five-task mean speedups of 2.2× on BitNet b1.58-2B and 2.1–2.4× on Bonsai 1.7B–4B; up to 2.0× five-task mean throughput on an RTX 3090; and 3.3× on MBPP+ on a Jetson Orin. The paper's Figure 1 shows the motivating contrast: naive "BitNet + MTP" improves BF16 throughput but loses both throughput and quality in low-bit W2A8 execution, while BitMTP raises throughput at quality comparable to the SFT-AR baseline.
  • Notable: the NeurIPS poster lists a different, peak-decode metric: 964.2 tokens/s on "BitMTP-2B," a 2.64× speedup over BitNet b1.58-2B-SFT. Both figures come from the authors' own documents and measure different things — five-task mean vs. single-point peak throughput.

Figure from the paper: five-task mean quality and decode throughput for BitNet b1.58-2B SFT-AR, naive BitNet+MTP, and the proposed BitMTP on NVIDIA H20, in BF16 (left) and low-bit W2A8 (right) execution.

All baselines are each model's own SFT-AR run on the same BitMTP stack — not bitnet.cpp, Bonsai's vendor runtime, or a dense FP16 model.

Why it matters

The constraint BitMTP targets is specific to ultra-low-bit inference. Once weights cost ~1.58 bits, linear-layer arithmetic is nearly free and memory traffic is tiny; the decoding state machine — how many sequential steps you take, what KV state you materialize, and whether you can replay a pre-captured CUDA Graph — becomes the bottleneck. MTP is the obvious fix, but it fails naively, the paper says: block-causal masked drafting lacks the intermediate token inputs next-token training provides (hurting quality), and the wider block computation plus acceptance-dependent causal KV updates can eat the savings from fewer steps.

BitMTP's answer is three coordinated pieces:

  • Joint NTP + MTP training. SFT is followed by joint next-token and block-prediction training; MTP forms block targets within assistant responses at block sizes 4 (D4) or 8 (D8). The result is one model with both an autoregressive path and mask-conditioned block-causal drafting.
  • Three no-drafter decode policies (repo docs): conf commits the confidence-selected prefix at a 0.9 threshold; spec verifies every draft block with the native ternary matmul and exact token-ID matching; hybrid commits the prefix once it reaches a minimum chain length (3 for D4, 6 for D8) and verifies otherwise. Draft projection precision is split by platform and policy: FP8 on H20 for conf/hybrid (speculative-only drafts natively on H20), BF16 on RTX 3090, native ternary on Orin.
  • Accepted-prefix alignment. The paper's core contract: at every iteration, the committed prefix must simultaneously be a valid autoregressive continuation, a directly promotable KV slice, and the index of a pre-captured CUDA Graph replay state. Runtime mechanics M1–M3 make that contract cheap: width-specialized packed low-bit kernels with role-dependent output projections; a finite-state "graph atlas" that replays reachable pending-prefix states over shared KV storage (config values list replay widths [1, 8, 9, 10, 12, 16, 17] for the D8 recipe and [1, 4, 5, 6, 7, 8, 9] for D4, over a geometric KV segment ladder of 256/1024/4096); and verifier-input preparation hoisted ahead of the host branch decision in hybrid mode.

The practical upshot: you can run MTP-style multi-token decoding on a ternary model without doubling your model inventory for a draft/verify pair. For engineers on memory-bound edge or server hardware, that is the difference between MTP being a training-lab trick and a deployment feature.

Caveats

  • All speedups are first-party. No independent reproduction or third-party benchmark of BitMTP's numbers surfaced in our review. The repo ships a reproducible GSM8K + HumanEval+ harness for the Bonsai models, but not the paper's full five-task suite, the BitNet path, or an Orin evaluation workflow. "Comparable quality" is asserted in the abstract; the full quality results sit in the paper body and were not fully verifiable from the accessible excerpt.
  • Scope limits. Shipped eval configs cap generation at 1,024 tokens at temperature 0.0 and run single-GPU at batch size 1; Orin is documented at a single 4,096-token KV segment; Bonsai-4B targets H20 only. No multi-GPU serving or long-context claims.
  • The baseline question. The comparison is BitMTP's MTP path vs. its own AR path on the same stack. Whether the 2×+ holds against tuned vendor runtimes is an open question the release does not answer.
  • Peer review is a plus, not verification. NeurIPS 2026 acceptance is confirmed by the conference's own poster listing, but the figures remain author-reported.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.