DeepSeek open-sources DeepInfer, the engine behind V4.1 Flash: DSpark, EPLB, 155T tokens/day — minus the serving layer

DeepSeek has open-sourced DeepInfer, the inference framework that powers its production DeepSeek-V4.1-Flash API service, under the Apache 2.0 license (repo). The release exposes the kernel stack, the DSpark speculative-decoding pipeline, and the batch scheduler — while explicitly withholding the KV-cache system, VE/prefill/decode disaggregation, and load balancing that turn an engine into a service.

What shipped

DeepSeek puts production economics at the center of this release. The deployment Q&A, published inside the repo, says a single cluster "currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing)." The README adds that the framework "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM" — important because anyone running a vLLM-derived engine can diff directly against it.

The scope is narrow. The release supports exactly one model family — DeepSeek-V4.1-Flash (MIT-licensed weights, default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017) — and exactly two accelerator families: NVIDIA SM100 GPUs (the die name DeepSeek and parts of the ecosystem use for Blackwell Ultra-class silicon like the B300) and Huawei Ascend 950 NPUs. It needs Linux, GCC ≥ 13, Python 3.12, and CUDA 13 or CANN 9.2. No Hopper, no ROCm support.

The kernels, integrated. DeepInfer wires together the kernel libraries DeepSeek has been releasing separately — FlashMLA (sparse attention with FP8/FP4 KV), DeepGEMM (FP8/FP4/BF16 tensor-core GEMM with fused MoE), DeepEP (expert-parallel all-to-all dispatch/combine), plus TileKernels, DeepSelect, and in-tree TileLang operators. It is the first DeepSeek release to ship them as one stack.

DSpark is the reference implementation. DeepInfer ships DeepSeek's confidence-scheduled speculative decoding, described in the DSpark paper (arXiv:2607.05147) and already re-implemented independently in vLLM and SGLang. Because V4.1 dropped the MTP module V3/V4 trained — per vLLM's recipes of the checkpoint config — DSpark is the only speculative method for this checkpoint.

The mechanism: a ~14B-parameter semi-autoregressive draft head — a parallel backbone plus a lightweight sequential module — proposes a block of draft tokens in one pass, and a learned confidence head scores each token's probability of surviving verification. The scheduler spends a fixed verification budget only on the top-scoring prefix slots, sized against a profiled throughput table rather than verifying every draft token (paper abstract).

Production default is a max draft length of 5 (DEEPINFER_USE_NEXTN_DECODE). DeepSeek reports 60–85% faster per-user generation versus their MTP-1 production baseline at matched throughput. That range is the paper's claim and has not been reproduced exactly; an independent SGLang implementation reports reproducing the shape of the gains.

SWA bounded replay. The repo's default DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 exists because of how V4.1-Flash is built: every layer attends over a 128-token sliding window, on a model whose context extends to 1,048,576 tokens via YaRN (factor 16 over a 65,536-token training window). Instead of re-running the full encoder prefix during decode, the engine replays only a bounded suffix of it — 140 tokens covers the SWA window plus maximum backtracking, per the README. That converts the model's sparse-attention design directly into prefill savings.

Reproducibility as a feature. DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only) keeps model results identical across different batch compositions; combined with a fixed seq_seed and MTP disabled, sampling becomes deterministic — identical outputs for identical inputs. Both are properties you want before running a commercial API on top. Also included: structured outputs constrained to regex, JSON schema, or EBNF grammar, and an asynchronous "zero-overhead" batch scheduler that overlaps CPU scheduling with device execution. MoE parallelism is EP plus EPLB — redundant-expert replication where hot experts live on multiple GPUs and are re-assigned every M steps over NVLink symm_mem.

The production numbers behind it

The deployment Q&A is the closest thing to published frontier-inference economics, and it is all vendor-reported:

• 155 trillion tokens/day per cluster at US$38/GPU-hour revenue (valley pricing).
• 64 prefill + 64 decode GPUs as the smallest efficient deployment; decode runs EP64–128 / DP64–128 ("Wide-EP"), prefill EP2–8 / DP2–8, with TP and CP not used.
• KV cache retention is a stated design goal of more than 72 hours, with ~14.4 TB of storage per GPU, on the 3FS clustered filesystem.
• EPLB measured moe_lb (slowest rank / mean) of 1.074 (EP4DP4) / 1.045 (EP2DP2) for prefill and 1.090 for decode — within a few points of perfectly balanced — and "roughly 80% of a perfectly balanced offline benchmark" overall.
• The eval README publishes reference Pass@1 figures: AIME 2026 99.9%, GPQA Diamond 93.4%, HLE-Full 42.6%, explicitly labeled "indicative rather than exact."

What is deliberately missing: KV-cache management and the load-balancing layer are "tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced." A companion piece, deepseek-recipe, ships the production token-encoding/decoding component that matches the official API's behavior.

Why it matters

Three things make this release worth studying rather than skimming.

It is a diff target for every vLLM-derived engine. DeepInfer is not a clean-room engine — it descends from vLLM, so the interesting code is the delta: DSpark's confidence-scheduled verification loop, EPLB's redundant-expert reassignment, SWA bounded replay, batch-invariant scheduling. Those are the same deltas the custom-engine community has been reverse-engineering from DeepSeek's kernel releases; now the serving logic around them is readable too.

It publishes the constraints any V4.1-Flash engine must handle. The architecture facts — 128-token SWA, 1M context, an MXFP4/MXFP8 mixed checkpoint around 511 GB on disk, and KV at roughly 890 bytes per token (vLLM recipes) — are the hard requirements. DeepInfer shows the official engineering response to each one, which is more actionable than the tech report's benchmark tables.

It is a rare look at serving-economics decisions. Prefill on commodity 8-GPU machines, decode on 64–128-GPU supernodes, KV kept for days on a storage cluster rather than evicted, redundant experts shuffled over NVLink — these are load-bearing product decisions, and the Q&A documents their measured effects.

Caveats

• No throughput numbers in the repo. The README's only performance artifacts are the two Pareto charts above — prefill on 4 SM100 GPUs, decode on 64, "each within one scale-up domain." Coverage of the release, such as Tech AI Wire, notes the 155T-tokens/day figure does not appear in the README itself and advises treating throughput claims as unverified until the sweep script is run on your own chips.
• The scale-up domain mismatch is unexplained. 64 GPUs "within one scale-up domain" does not match NVIDIA's official 72-GPU GB300 NVL72 domain.
• Everything economic is vendor-reported. The token/day, GPU-hour revenue, moe_lb, and load-balancer efficiency figures all come from DeepSeek's own Q&A. The Hacker News thread that carried the release to the front page is mostly a debate about whether inference margins can survive competition, not an independent benchmark.
• The serving layer is the withheld part. Without the KV-cache system, disaggregation, and load balancing — the three components explicitly not included — the repo reproduces DeepSeek's kernels and scheduler but cannot reproduce their service. Hardware exclusivity (SM100/Ascend 950 only) further limits who can run it at all.
• The eval Pass@1 figures are reference sanity checks for a deployed instance, not independent evaluations, and the DSpark speedup is a paper claim.

Sources

• DeepInfer repository (README)
• DeepSeek V4.1 Flash Inference Deployment Q&A
• DeepInfer evaluation README (reference Pass@1)
• DSpark paper (arXiv:2607.05147)
• vLLM recipes: DeepSeek-V4.1-Flash architecture
• vLLM blog: DSpark adaptive verification
• LMSYS blog: DSpark in SGLang
• FlashMLA · DeepGEMM · DeepEP
• DeepSeek-V4.1-Flash-1017 model card
• NVIDIA GB300 NVL72
• Hacker News thread
• Tech AI Wire coverage
• r/LocalLLaMA announcement thread