ORL: diffusion LLMs get a learnable "tokens per forward" — 2,788 tok/s on one RTX 5090, 7,708 on 8× B200, fully open source
Together AI's on-policy residual learning plus the dlmrt engine make diffusion decoding competitive with autoregressive speculative decoding — and the release is fully open, with a bit-for-bit reproducible QwenDLM recipe.
ORL, a training recipe from Together AI and academic collaborators, teaches diffusion LLMs to commit more tokens per denoising pass and pairs it with a purpose-built engine, dlmrt. A diffusion LM now decodes at 2,788 tokens/s on a single RTX 5090 (LLaDA-2.1-mini, NVFP4, batch size 1) and 7,708 tokens/s on eight B200s (DiffusionGemma 26B MoE). The paper, training code, engine, and NVFP4 checkpoints are public, and the authors report the QwenDLM training recipe reproduces the paper's checkpoint bit for bit on 8× H100s.
What shipped
Diffusion language models generate in blocks rather than one token at a time: each denoising round produces a predictive distribution over the still-unresolved tokens, and the block is committed after enough rounds. The practical cost is the number of rounds per block. ORL attacks that on the training side; dlmrt attacks it on the serving side.
On-policy residual learning. Residual decoding already feeds the previous round's distribution back into the next round as context — through a self-conditioning adapter on the tied embedding — so each round starts from the model's own last guess. The paper's core claim is that this residual learning is inherently on-policy: the residual states a model sees at inference are generated by the model itself, and their distribution shifts as training proceeds, so pre-generated trajectories go stale.
ORL regenerates trajectories on the fly after training updates, captures the exact residual inputs each denoising step consumed asynchronously off the decoding path, and replays them in packed, compute-balanced batches. The co-designed training system — asynchronous residual capture plus specialized block-diffusion attention kernels — makes training steps up to 5.2× faster.
Adjacent entropy contraction. Supervising toward the final output teaches the model what to predict but not how fast to converge — "simply supervising the final output doesn't necessarily teach the model to converge faster," as the announcement puts it. AEC adds an explicit objective: each refinement step must reduce uncertainty by a fraction ρ (0.2 in the released DiffusionGemma recipe) relative to the step before it. Supervision is gated by rollout entropy so already-confident predictions are not sharpened further; the released QwenDLM stage-2 recipe runs this gated variant, while the DiffusionGemma AEC recipe runs ungated. The result: more positions clear the commit bar each round, fewer rounds per block, and higher tokens per forward pass (TPF).
dlmrt, the inference engine. dlmrt is built for single-request latency, the regime the authors argue agents actually operate in — one user at a time, where what the user feels is how fast a single request finishes, not aggregate throughput. Each block of tokens runs as one CUDA graph whose denoising loop stays on the GPU: a conditional node decides after every round whether to continue, so the host only launches the block and reads back tokens, with no per-round CPU synchronization. On Blackwell, routed experts run NVFP4 on FP4 tensor cores, the KV cache is FP8, and tensor parallelism across an NVLink node uses a fused, barrier-free reduction, so every rank gets identical values and all ranks run the same device loop with no host coordination. With identical checkpoints and decoders on one H100, dlmrt is 1.2–1.6× faster than SGLang (QwenDLM, LLaDA-2.1-mini) and vLLM (DiffusionGemma). One denoising round takes 5.0 ms for QwenDLM + ORL on an H100 and 2.0 ms for LLaDA-2.1-mini + ORL in NVFP4 on a B200.
The numbers. All figures are single-request (batch 1) and token/s as measured on the paper's prompts. From the paper and README:
- RTX 5090: 2,788 tokens/s (LLaDA-2.1-mini + ORL, NVFP4, HumanEval). LLaDA-2.1-mini is a ~16B-parameter MoE, and the NVFP4 checkpoint fits in the card's 32 GB — the README notes a single consumer 5090 runs these models at about the speed of BF16 on an H100 or faster.
- 8× B200: 7,708 tokens/s (DiffusionGemma 26B MoE + ORL, NVFP4, GSM8K); 5,046 tokens/s on a single B200.
- H100 end-to-end: QwenDLM-4B + ORL on dlmrt reaches 1,298 tokens/s, versus 746 for DFlash and 405 for EAGLE3 speculative decoding of the base autoregressive model (relaxed verification, vLLM baselines). That is 1.74× faster than DFlash at matched accuracy.
- QwenDLM-4B (a 4B model converted from Qwen3-4B-Thinking-2507): TPF up to 6.2× higher, end-to-end throughput up to ~6.9× with dlmrt, at comparable or better quality. HumanEval pass@1 rises from 82.9% to 89.6%, GSM8K from 90.4 to 92.4, MATH-500 from 88.0 to 92.2.
- LLaDA-2.1-mini: 4.2M self-generated training tokens and 36 H100-hours — roughly 10× less post-training data than earlier LLaDA parallel-decoding recipes — for up to 1.5× TPF and up to 3× throughput at comparable quality. NVFP4 on one B200 is 2.3–2.9× faster than BF16 on one H100 with comparable accuracy.
- DiffusionGemma 26B MoE: ORL adds 1.1–1.2× TPF to an already highly parallel model.
Why it matters
ORL treats how many tokens to commit per forward pass as a learnable quantity rather than a fixed property of the decoder, then co-designs an engine that converts the learned parallelism into wall-clock speed. That is the same pattern that made speculative decoding practical — train or draft the fast path, verify cheaply — but applied inside the diffusion paradigm, where every denoising round was previously paid for at full model cost.
Three things stand out.
First, the single-request framing. vLLM and SGLang optimize batched throughput; with batch size 1, most of the GPU sits idle and kernel launch overhead dominates. dlmrt's whole-loop CUDA graph — loop condition decided on-device, host only launches the block and reads back tokens — is a reusable pattern for any iterative decoder that wants to avoid per-round CPU round-trips.
Second, the openness. The training repo is MIT, the engine is Apache-2.0, the checkpoints and training data are public, and the authors claim the QwenDLM recipe reproduces the paper's checkpoint bit for bit on 8× H100s. Independent verification is cheap to attempt — exactly what an engineer evaluating diffusion LMs wants.
Third, the quality moved in the same direction as the speed. On the authors' evaluation, QwenDLM-4B gains 6.7 points on HumanEval (82.9 → 89.6), 2.0 on GSM8K (90.4 → 92.4), and 4.2 on MATH-500 (88.0 → 92.2) while getting several times faster. A speedup that traded away accuracy would be easy to dismiss; one that improves benchmark scores at the same time is worth tracking first when independent verification shows up.
Caveats
These numbers deserve scrutiny before anyone builds on them:
- Batch 1, peak cells. The authors state that all figures are single-request decoding and are "peak results on specific benchmarks, not averages across arbitrary prompts." The DiffusionGemma 8× B200 and RTX 5090 rows use a single generation seed (1234); only the single-B200 row averages seeds 1234–1236.
- Comparisons mix engines and precisions. ORL runs on dlmrt while the DFlash/EAGLE3 baselines run on vLLM, and the speed tables mix precisions across rows: the RTX 5090 and B200 headline rows are NVFP4, while the H100 rows are BF16 (including QwenDLM's 1,298 tok/s row). Reading across rows therefore compares NVFP4 against BF16 as well as dlmrt against vLLM.
- The 1.74× vs DFlash headline uses relaxed verification — the fastest setting whose accuracy is within 0.5 points of ORL's. Under strict verification the base autoregressive model keeps a 2.4-point accuracy edge over ORL, and ORL's raw tokens/s lead is actually larger (about 2.3× vs DFlash). The "at matched accuracy" framing only holds under relaxed verification.
- FP4 quantization costs efficiency, not accuracy. The checkpoint card reports FP4 rounding reduces tokens per forward by 2–12%, while accuracy stays within run-to-run noise.
- Hardware scope. The NVFP4 checkpoints require Blackwell (B200/B300/RTX 50 series); dlmrt's supported-GPU table lists H100, B200/B300 and RTX 50 series, with QwenDLM currently served on Hopper only.
- Unverified by anyone else. The paper went up on 14 October 2026, and we found no independent third-party reproduction or benchmark yet. Community skepticism about diffusion LMs as daily drivers — model quality and large KV caches were both raised in the thread — is not addressed by throughput numbers alone.