NVIDIA opens the coding-agent midtraining recipe: 500B tokens at 256K context, and the ablation that shows why ~10% agentic traces matter
The Nemotron team's arXiv paper discloses the stage between pretraining and post-training — with data blend, eval protocol, and scaling curves.
On October 15, NVIDIA's Nemotron team posted an ablation-level recipe for "midtraining" — the stage between pretraining and post-training that frontier coding models now use but almost never document. On two open Nemotron 3.5 base models, the arXiv paper reports that 500B or 280B midtraining tokens at 256K context raise SWE-bench scores by up to 7 points after an identical SFT run.
What shipped
Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe (arXiv:2610.22019, 19 pages) was posted October 15 by NVIDIA's Soumye Singhal with thirteen co-authors. The abstract frames it as a disclosure problem: "Frontier models, general-purpose and domain-specific alike, now include a midtraining stage between pretraining and post-training, but little has been published about how this stage is built."
Pretraining runs at short context and "sees almost no agent interactions," SFT data is mostly synthetic, and "very long SFT stages can hurt the model's later RL trainability" (abstract). Midtraining sits between the two: hundreds of billions of tokens at 256K context on a mix of "packed repositories, pull requests, commit histories, and build logs together with a small share of agentic traces."
The headline numbers, all after an identical full SFT on both the midtrained and the base checkpoint:
- Nemotron 3.5 Lightning-30B-A3B, 500B midtraining tokens: SWE-bench Verified 60.7 → 64.5, SWE-bench Multilingual 55.8 → 62.8, SWE-bench Pro 32.0 → 35.5.
- Nemotron 3.5 Super-120B-A12B, 280B tokens: DeepSWE (which the authors call a harder benchmark of long-horizon tasks) 11.7 → 15.7, and SWE-bench Verified 72.9 → 74.3. On Super, Multilingual and Pro do not improve consistently (alphaXiv summary).
From the paper's Table 1, the 500B Lightning blend is roughly 41% issue–PR pairs, 20% repository-level data, 20% pretraining replay, 10% agentic traces, 5% long-context data, 3% diffs, and 1.2% build and CI logs. The trace share is a deliberate choice — the authors keep it at 10–15% of the blend to avoid overfitting to SFT-like data (arXiv HTML, alphaXiv).
The more portable contribution is the evaluation protocol. The team found that base-model proxy metrics — bits-per-byte and LoCoDiff — "did not reliably predict post-training results," so every data choice was scored with short fixed SFT runs (33B tokens) and confirmed with full 100B-token SFT, both agentic-focused (arXiv HTML, DAIR.AI, alphaXiv). Anyone tuning long-context continued training on perplexity-style signals should read that sentence twice.
Scaling is roughly saturating: most of the Verified and Pro gains arrive by 100B tokens, Multilingual keeps improving, and Terminal-Bench is noisier, peaking at 300B (alphaXiv). The paper also claims robustness across agent harnesses — "both models improve on every harness we tested" — and reports that on Super after 65B tokens plus short SFT the average gain across harnesses is 8.6 points on SWE-bench Verified and 6.8 on Terminal-Bench (arXiv HTML). It notes that scores still differ substantially across harnesses (alphaXiv).
The release is timed with NVIDIA's same-day launch of the Nemotron 3.5 Super VL and Bolt models, which the company blog says ship with "open weights, open datasets and training recipes."
Why it matters
The paper's framing is narrower than "we invented midtraining" and more useful: relative to the recipes it cites — Qwen3-Coder-Next and Kimi-Dev — it supplies "the ablation-level detail missing from the technical reports" (arXiv HTML), and it does so on an open-weights model line. Nemotron 3.5 Lightning is a 30B-parameter, 3B-active Mamba2–Transformer hybrid MoE with up to 1M context, released under the OpenMDW-1.1 license with weights on Hugging Face. The recipe is checkable: base checkpoints, license, token counts, and benchmark harnesses are all public, so a replication run is a matter of compute rather than reverse engineering.
The sharpest result is what the agentic traces are worth. In a 100B-token Lightning midtraining run followed by identical full SFT, SWE-bench Verified pass@1 goes 59.7 (no midtraining) → 60.5 (natural software-engineering data only) → 63.5 (full blend with traces) (alphaXiv). Under the short SFT regime, the natural-only variant actually regressed against base (56.1 vs 57.1). So the ~10% trace share is the difference between midtraining helping and hurting — the practical takeaway for teams tempted to skip trace collection as a data-engineering nuisance.
The eval protocol matters beyond this paper. If bits-per-byte and LoCoDiff cannot predict downstream agent scores, cheap proxy-driven iteration on continued training is a trap; the fixed-short-SFT gate is a design pattern worth copying even at smaller scale.
Caveats
- Everything is vendor-reported. All benchmark numbers come from the paper, and we found no independent replication in the first days after posting. Per the paper's limitations, as summarized by alphaXiv, benchmarks are noisy, each comparison is a single training run, and the results are on Nemotron models only, mostly coding tasks, with an agentic-focused SFT blend.
- No compute-parity comparison. The team did not compare spending those 280–500B tokens on more pretraining or more SFT, so the paper shows midtraining helps, not that it is the cheapest way to get the same gains.
- The harness claim is thinly backed in public form. The abstract asserts improved robustness across harnesses without per-harness numbers; the body's averaged gains come with the admission that scores differ substantially across harnesses.
- The trace composition is only partly disclosed. The 10% share, the loss-masking policy, and a credit to Perplexity for contributing agent traces (Soumye Singhal on X) are public, but the composition of the trace portion is not described in the abstract (AI Weekly), and per alphaXiv the authors state the experiment cannot separate the value of traces from the value of scaffold diversity.
- The RL evidence is limited. The paper's short, single-seed RL experiment shows only that the midtrained model's post-SFT advantage is not lost when RL begins — not that midtraining increases what RL can add (alphaXiv).
- Coverage note: AI Weekly's report correctly flags that no per-harness numbers and no trace-composition details appear in the abstract.