NVIDIA's Sigma is an open continuous-space diffusion LM — it denoises 16-D embeddings instead of predicting tokens
Open-weights 3B/8B diffusion LMs that denoise Gaussian noise in a 16-D embedding space, with block-parallel decoding and a clean NFE knob for trading quality against speed.

NVIDIA released Nemotron-Labs-Sigma, an open-weights family of continuous-space diffusion language models in 3B and 8B sizes that generate text by denoising Gaussian noise in a low-dimensional token-embedding space instead of predicting tokens one at a time. It runs today in plain PyTorch via trust_remote_code, with no custom kernels or flash-attention required, and ships with a technical report, Large Language Continuous Diffusion Models (arXiv:2610.02665).
What shipped
Sigma is a decoder-only Transformer (grouped-query attention, SwiGLU, rotary position embeddings, in a Ministral-style layout) with a continuous-diffusion generation head (model card). Instead of sampling one token at a time, the model denoises Gaussian noise in a low-dimensional 16-D continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block is refined over a configurable number of denoising steps using DDPM or DDIM samplers with self-conditioning and optional classifier-free guidance, while previously committed tokens are reused through a standard KV cache.

The knob engineers will care about is NFE — number of function evaluations — which equals steps + 1 per block. Fewer steps per block trade a little quality for large speedups. NVIDIA's recommended sampler settings are DDIM-32 with CFG 3 and score temperature 0.3, DDIM-16 with CFG 3 / 0.4, DDIM-8 with CFG 3 / 0.6, and DDIM-4 with CFG 2 / 0.8 (model card). The card warns that below roughly 16 steps the pretrained models lose quality noticeably, and points to the separately released -distill-* variants for few-step generation.
The family spans base, instruct (SFT), few-step distilled (NFE 2/4/8/16), and self-speculative checkpoints — 14 in total. The instruct variants have an 8,192-token context; the base variants 4,096 (model cards). Inference runs on a CUDA GPU in bfloat16 with PyTorch ≥ 2.4 and Transformers ≥ 4.55; CPU inference is not supported because the sampler degrades numerically on CPU (model card). NVIDIA reports roughly 120–140 tokens/s for 256 generated tokens with the DDPM sampler at 32 denoising steps per 64-token block on a single Hopper H100.
Why it matters
Sigma is a distinct line from the masked/discrete diffusion LMs that have dominated the open field — LLaDA, Dream, Mercury, and NVIDIA's own Nemotron-Labs-Diffusion. The paper claims it is the first family of large-scale (3B/8B) continuous diffusion LMs, noting that prior continuous/flow text methods were under 2B parameters or unconditional-only (arXiv:2610.02665). The point of the continuous formulation is that it gives you a smooth, steerable latent space and a clean path to few-step distillation.
The reported numbers, all from NVIDIA's own paper and cards, are competitive with the discrete baselines. On the paper's six-task average, base Sigma-8B scores 69.67 with DDPM sampling and 66.72 with DDIM at 32 steps, against 70.22 for Nemotron-Labs-Diffusion-8B; base 8B reaches 89.07% on GSM8K and 67.47% on Minerva Math (paper summary). The 3B variant posts the best base-model numbers on GSM8K, HumanEval, and HumanEval+ and beats LLaDA-8B and Dream-7B by 22.6 and 5.6 points on average despite being roughly half their size (paper summary).
The practical payoff is the compute-for-speed trade. The paper reports roughly 8× acceleration on math and 4× on coding from few-step distribution-matching distillation with negligible accuracy loss, and about 3.5× throughput over the autoregressive counterpart from one-step speculative decoding on GB200 (arXiv:2610.02665). In arithmetic terms, at the recommended 32-token block size the base model runs about 33 denoiser calls per block (≈0.97 tokens per forward pass), while the distilled NFE-4 variant runs 5 calls per block (≈6.4 tokens per forward). That is the kind of lever that matters when you are paying for GPU-hours on agent workloads.
Caveats
This is a vendor release, and the accuracy claims are NVIDIA's word — no independent third-party reproduction was found. Two things in particular should temper enthusiasm:
- The base and instruct model cards publish no benchmark numbers at all. They defer to the technical report, so everything you can know about Sigma-8B's accuracy comes from the paper and the distilled checkpoints' cards (model card). The distilled NFE-4 card reports a ten-seed six-task mean of 66.36 ± 0.46, but that is a different checkpoint under a different sampling configuration than the paper's base-model 69.67, so the two cannot be subtracted to measure what distillation cost.
- The block size is stated inconsistently. The model cards say "default 64 tokens per block," while the paper, the code examples, and the sampler-recommendation table use 32-token blocks. The throughput figure is quoted at 32 denoising steps per 64-token block. It is a documentation inconsistency, but it is exactly the kind of detail that will bite when you try to reproduce the numbers.
- The distilled NFE-16 checkpoint is marked
pending_ten_seed_six_task_validation, and its card warns against comparing it with the other variants' ten-seed means.
The license is the NVIDIA Nemotron Open Model License — open weights, and NVIDIA states the models are ready for commercial or non-commercial use, but it is not Apache (model card).