RadixArk serves Meta's 816B Muse Spark Open on one TPU v7 host with a 62-layer megakernel — decode at the HBM floor
All 62 MoE layers of a decode step run as one Pallas program per device; the verify step lands within 1.5–1.7× of HBM weight-stream time.
RadixArk and the SGLang community served Meta's 816B-parameter MoE Muse Spark Open on a single four-chip TPU v7 host using SGLang-TorchTPU, running all 62 MoE layers of a decode or verify step as one Pallas megakernel per device. The headline result: a full 62-layer verify step runs within 1.5–1.7× of the time it takes to stream its weights from HBM — essentially at the memory-bandwidth floor.
What shipped
The work is described in a PyTorch blog post by Haolin Fu, Xiaoyu Zhang, Junyan Xu, Wangyuan Zhang, Haoguang Cai and the RadixArk team, with code in the SGLang-TorchTPU repo (branch haolin/ms-v11).
The model and hardware. Muse Spark Open is Meta's open-weights MoE: 816B total parameters, about 42B active per token, 62 decoder layers, all MoE, each routing a token to 8 of 256 experts. Attention is hybrid — 46 layers use sliding-window attention over the last 2,048 tokens, and 16 attend over the full context up to 1M tokens. Expert weights are NVFP4 (4-bit, one scale per 16 values).
A TPU v7 (Ironwood) host has four chips, each with two chiplets that the framework exposes as separate devices, giving 8 devices per host. The model is sharded 8 ways with tensor parallelism, one shard per device. Each device has 96 GiB of HBM (192 GiB per chip at 7.37 TB/s).
The megakernel. All 62 layers of a decode or verify step run as a single Pallas program per device, instead of one kernel per layer. On a single-threaded device, every kernel boundary stops the weight stream; the megakernel keeps the next weights always in flight. SGLang still runs the embedding, draft model, a small KV-write kernel, the LM head and sampling around it.
The team reports the megakernel is 1.94× faster than the native path per output token for a single request (1.95 ms vs 3.78 ms) and 1.85× for two concurrent requests. The previous native path — already built from fused Pallas blocks — took 2.9× the time needed to stream its weights; profiling showed collectives cost about 45 µs per layer at batch 1, so the step was bound by overhead, not bandwidth.
Inside the megakernel, everything is pipelined: a ring of ~10 dense-weight tiles sustains 3.34 TB/s per device (vs 1.13 TB/s for a single DMA stream), expert weights are requested the moment routing finishes, and tensor-parallel partial sums are written directly into the owning device's memory with remote DMAs (~3 µs per round) instead of separate collectives. Pipelined KV writes cut write cost per step from 810 to 155 µs at 16 rows and from 2,286 to 336 µs at 64 rows, with bit-identical results.

Two model-specific tricks. The attention loop is "max-free": the team dropped the running maximum from online softmax. Because Muse Spark's query/key normalization plus a fixed query gain bounds the attention scores, the maximum is unnecessary. Dropping it removes a serial dependency between KV-cache chunks and lets them be processed in a pipeline. The team reports this pipelined attention loop raised decode speed per request by 18% for two requests at a 500,000-token context. Separately, the team split the expert loop in two to keep register spills off its common path; they report decode steps ran 13–19% faster with many requests in a kernel test with random routing.
Speculative decoding and AgentX. The server uses DFlash2 speculative decoding, where a small draft model proposes tokens and the full model verifies them in one pass. On GSM8K with real acceptance, the team reports DFlash2 cut the megakernel's time per output token for one request from 3.93 ms to 1.95 ms — a 2.0× speedup.
On the 1M-context version of SemiAnalysis's AgentX agentic-coding benchmark, the team's curve runs from 433 output tokens/s per request at 6,101 tokens/s per chip to 25,906 tokens/s per chip at 95 tokens/s per request. Every TPU point finished requests within 47 s at p90. AgentX is built from 393 opt-in Claude Code sessions, replaying request lengths, prefix sharing, tool pauses and subagent structure with synthetic tokens; the 1M-context trace set has prompts reaching about 990K tokens, and throughput counts both input and output tokens.

Why it matters
Decode now runs close to the HBM-bandwidth floor. The verify step takes 1.5–1.7× the time to stream its weights from HBM — most of the step is spent moving weights, not computing. The native path was 2.9× that floor. That is the practical definition of a well-pipelined MoE decoder on a single-threaded device.
The economics matter too. The post cites InferenceX's external owning costs of $1.21 per TPU v7 chip-hour versus $1.86 per GB200 GPU-hour, with the GB200 (FP4) baseline from InferenceX's published results as of 2026-11-02. At up to ~25.9K tokens/s per chip on a 1M-context agentic workload, that is a meaningful cost-per-token gap — though it rests entirely on InferenceX's cost model and the team's own measurements.
For engineers building on TPUs, the reusable lesson is the megakernel pattern: fuse the whole decode step into one program so weight streaming never stops, and pipeline every transfer behind compute. The max-free softmax shows that model-specific invariants (bounded attention scores) can eliminate machinery generic kernels assume is required.
Caveats
Every performance number here is self-reported by the RadixArk/SGLang team; no independent reproduction was found. The headline decode figures (1.94×/1.85× megakernel, and the DFlash2 2.0×) were measured on an earlier Muse Spark checkpoint, not necessarily the final released model. The 18% attention and 13–19% expert-loop figures are presented as serving-system and kernel-test results. The tokens-per-dollar comparison depends on InferenceX's cost model and the GB200 baseline as of 2026-11-02. The team's own X thread claims "3× faster decode at small batch size" — that bundles the megakernel and DFlash2 together, so it should not be read as the megakernel figure alone.