Richland Technology
Richland Technology
← Back to Blog
quantizationllmopen-sourceinferencegguf

MACaQ: Huawei's layer-local quantizer keeps the Hadamard transform online — 122B in 64 min in 32.5 GB, but the "beats Unsloth" KL numbers are still unverified

Huawei's layer-local quantizer keeps the Hadamard transform online to stream one layer at a time — but the "beats Unsloth" KL tables are self-reported and unverified.

Richland AI Desk·11 November 2026·5 min read
MACaQ: Huawei's layer-local quantizer keeps the Hadamard transform online — 122B in 64 min in 32.5 GB, but the "beats Unsloth" KL numbers are still unverified

Huawei's Computing Systems Lab has open-sourced MACaQ, a post-training quantization toolkit that writes 2/3/4-bit GGUFs for llama.cpp and W4A4 checkpoints for vLLM. It claims lower KL divergence than Unsloth and Bartowski GGUFs, and a 122B-parameter model quantized in 64 minutes inside 32.5 GB of GPU memory. The interesting part is not the headline numbers — self-reported and unverified — but the design bet underneath: MACaQ keeps a Hadamard transform online at inference so quantization stays strictly layer-local, which lets it stream one layer at a time through memory.

What shipped

MACaQ ("Model Agnostic Calibrated Quantization") is an Apache-2.0 toolkit from Huawei's Computing Systems Lab (repo, paper, blog). It produces two kinds of artifacts: GGUF files in six llama.cpp formats — IQ2_XXS, IQ2_XS, IQ3_XXS, IQ3_S, IQ4_NL, IQ4_XS — and W4A4 MXFP4 checkpoints for vLLM, Diffusers, and PyTorch. The team does not ship ready-made GGUFs; you get the code, a llama.cpp fork, and a CLI.

The Reddit thread that surfaced it republished the team's GGUF quality tables, which report mean KL divergence to the FP16 model:

  • Qwen3-4B (IQ4_XS): MACaQ 2.260 GB / KL 0.02804; Bartowski 2.271 GB / 0.03443; Unsloth 2.271 GB / 0.03663 — claimed 23.5% lower than Unsloth, 18.6% lower than Bartowski.
  • Qwen3.6-35B-A3B (IQ4_XS): MACaQ 18.86 GB / 0.018696; Unsloth (UD-IQ4_XS) 17.73 GB / 0.020954; Bartowski 19.70 GB / 0.021927.
  • Qwen3.8-27B (IQ4_XS): MACaQ 14.96 GB / 0.036784; Bartowski 15.48 GB / 0.104170 — claimed ~65% lower than Bartowski (no Unsloth row for this model).

Qwen3-4B GGUF size versus KL divergence to FP16, from the MACaQ repo

The team also reports quantizing a 122B-parameter model in 64 minutes using 32.5 GB of device memory, and, in the paper, lower perplexity than DuQuant++ and MR-GPTQ — on Llama-3-8B it matches their best perplexity in a fraction of the time (6.96 vs 7.16 in 50.8 s vs 624.4 s, versus DuQuant++ with GPTQ).

Why it matters

The mechanism is the story. MACaQ quantizes each linear layer independently, applying two online activation transforms whose inverses are absorbed into the weights at quantization time: a per-channel scale D and a block-Hadamard rotation B (block size 32, matching the MXFP4 microscaling block size). At inference the activation path runs X' = XDB, W' = WD⁻¹B, so the rotation is applied on the fly rather than baked into the weights.

That is the opposite of DuQuant-style methods, which propagate the transform out of the layer and absorb it into preceding layers' parameters, so inference pays nothing for it. MACaQ deliberately leaves it online, accepting a claimed ~2.3% end-to-end decode overhead over pure MXFP4 on Qwen3-32B in vLLM (batch 1, context 512). The payoff: quantization becomes strictly layer-local. No transform couples one layer's weights to another's, so layers can be streamed through memory one at a time. That is what makes the resource claims plausible — a 122B model in 64 minutes in 32.5 GB is a memory-bandwidth story, not a compute story.

The paper's quality mechanism is a Hessian-weighted coordinate descent it calls Flip-Refinement, which greedily moves FP4 codes to adjacent representable E2M1 values. The ablation is concrete: on Llama-3-8B, 512 flips over 200,000 calibration tokens gives 7.08 perplexity without Flip-Refinement and 6.87 with it.

For engineers this is a runnable technique: a CLI, a llama.cpp fork, and a documented recipe (the Qwen3.6-35B-A3B example pins a model revision and calibration settings of 512 context, 128 ubatch, 256 chunks — 131,072 calibration tokens). One Reddit user confirmed the workflow runs and is fast, and produced a Qwen3-4B IQ4_XS GGUF from it.

Caveats

Everything above the code is vendor-reported. The GGUF KL-divergence tables, the 23.5%/18.6%/~65% deltas, the 64-minute/32.5 GB 122B claim, and the perplexity comparisons all come from the team's own blog, paper, and LinkedIn posts — the authors explicitly frame them as "our internal evaluations." No independent reproduction of the KL measurements had appeared in the Reddit thread at the time of writing.

The KL methodology itself is underspecified. The evaluation corpus, token count, and whether KL is measured over generated text or a held-out corpus are not published in any retrievable source (the interactive blog is client-side rendered and exposes no numbers). Reddit commenters immediately flagged the risk that a single self-reported KL number can be gamed by handpicking examples, and asked for a second metric like top-1 agreement.

Two more things to know before you build on it:

  • The first community-shared GGUF was calibrated on the project's own smoke-test corpus — a 128-paragraph synthetic set whose docstring says it is "not evidence of production calibration quality." The filename qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf decodes to a 64-block Hadamard rotation, 2 vector-codebook sweep passes, and that smoke calibration. It is a demonstration artifact, not the production-calibrated file behind the headline numbers.
  • The paper is a non-peer-reviewed preprint (arXiv:2611.10649, v1) with no citing models or datasets yet, and vanilla llama.cpp compatibility with MACaQ GGUFs is unconfirmed — only the included fork is documented to run them. The hardware behind the 122B/64-minute claim and even the identity of that 122B model are not stated in the retrievable text.

The honest read: the "beats Unsloth and Bartowski on KL" headline is a claim awaiting independent auditing. The layer-local design that makes streaming quantization possible is real, visible in the code, and worth understanding regardless of whether the benchmark numbers hold up.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.