TIDE gives FSDP2 layout-aware zero-copy collectives: up to 27.1% faster steps on 64 GPUs, zero-SM all-gather, bf16→fp32 reduce-scatter
KlingAI Research's zero-copy collectives keep FSDP2's per-parameter storage while making the collective write every byte once at its final address — a zero-SM all-gather, a bf16→fp32 reduce-scatter NCCL doesn't provide, and up to 27.1% faster steps on 64 GPUs.
KlingAI Research has posted a preprint describing TIDE, a set of zero-copy collectives for PyTorch's FSDP2 that fold tensor-layout placement into the collective itself instead of copying data between layouts. On 64 GPUs, the authors report step-time reductions of up to 27.1% versus stock FSDP2 and 6.0% versus the symmetric-memory path across Llama-3.1 70B, Wan2.2, and Qwen3-30B-A3B, with reduce-scatter SM time down 12.5–14.5× (arXiv:2610.38228).
What shipped
TIDE is a pair of collectives — TIDE-AG and TIDE-RS — built on public NCCL 2.31+ APIs. It keeps FSDP2's per-parameter storage and compute kernels untouched, folding layout placement into collective addressing rather than copying data between layouts (arXiv HTML).
The problem is a layout-contract mismatch. FSDP2 stores parameters and gradients as per-parameter DTensor shards, but its collective path, NCCL, operates on flat rank-major buffers. Bridging the two layouts requires copying every all-gather output into parameter storage and packing gradients into a full-group fp32 buffer before every reduce-scatter. On Llama-3.1 70B at 8K tokens per GPU, layout conversion alone is 6.4% of step time, about 0.33 s per step. The paper breaks the residual cost into three parts:
- Layout conversion and packing: each group's parameters and gradients briefly exist twice.
- SM-wave quantization: a default NCCL reduce-scatter that stays resident for the whole transfer can slow a concurrent GEMM that would otherwise finish in a single wave by 69% — and the symmetric-memory variant by 11% at the median.
- HBM direction contention: remote-service reads and switch-reduction reads slow co-running memory-bound kernels 2.2–4× more than inbound copy-engine writes, depending on the clock regime.
TIDE-AG keeps the all-gather off the SMs entirely: copy engines multicast shards through NVSwitch locally, each GPU forwards its rail's data across nodes with paced recursive-doubling RDMA, and receiving nodes multicast the pieces directly into their final parameter-major addresses. Each piece is written once, at its final address — no copy-out, no temporary buffer.
TIDE-RS is the more interesting half. It consumes per-parameter bf16 gradients in place and reduces them into fp32 shards — a mixed-precision reduce-scatter mode the paper notes NCCL does not provide. Copy engines push bf16 slices to bounded local staging; short-lived TMA thread blocks sum node-local contributions in fp32; rail RDMA carries the fp32 node partials; and a Latin-square placement spreads contributors across destination GPUs, letting the NVSwitch sum up to L nodes' partials in a single reduction.
The reduce-scatter still uses short-lived CTAs — Table 1 budgets 8 (tunable) versus 16–32 for stock FSDP2, and the evaluated training configuration runs 16 summing plus 4 landing CTAs — but the full-group fp32 input buffer is gone.
A two-phase pipeline separates gradient reads from network progress, and there are two modes: In-Place reads gradients where autograd leaves them and runs the next all-gather early; Slot-Reuse copies them into a just-vacated all-gather slot and frees their storage. On paper, the live bytes during a layer's backward fall from FSDP2's 4NS to 3.72NS (In-Place) or 2.29NS (Slot-Reuse), and the extra buffer footprint drops from 3NS to ≈1.1NS. Measured peak memory is up to 4.7 GiB below FSDP2+SymmMem, and below stock FSDP2 on Wan2.2 with Slot-Reuse.
Why it matters
All headline numbers are on 64 GPUs, with step times as medians over 40–100 post-warm-up steps (alphaXiv page-referenced summary). The 27.1% / 6.0% pair comes from the mixture-of-experts model Qwen3-30B-A3B at 16K tokens per GPU — exactly the configuration the paper's overhead model identifies: the recoverable FSDP overhead fraction scales with Ψ_total / (Ψ_active · tokens per GPU), so it grows under strong scaling, sequence parallelism, and sparse MoE routing, where every rank still communicates the full sharded parameter volume while only a fraction of parameters is activated per token. The authors flag the MoE explanation as their interpretation, not a separate causal measurement.
The more useful result for engineering judgment: TIDE is not just "cap the NCCL CTA count." The paper's operator-level overlap test on eight nodes (alphaXiv) runs a 32 MiB reduce-scatter against a concurrent one-wave GEMM at a fixed SM clock. Default NCCL is fast in isolation (6.75 ms) but slows the GEMM 69%; capping NCCL to 4 CTAs protects the GEMM (2.7% slowdown) but balloons the reduce-scatter to 26.35 ms; TIDE's training configuration (16 summing + 4 landing CTAs) gets 2.9% slowdown at 6.96 ms. The same mechanism shows up end-to-end: reduce-scatter SM time drops 12.5–14.5×, and interference with concurrent compute falls by up to an order of magnitude.
Dense-model gains are real but smaller in relative terms — and the paper is explicit about it. On dense Llama-3.1 70B the absolute saving over stock FSDP2 stays at 0.51–0.54 s per step from 16K to 32K tokens per GPU, so the relative reduction falls from 6.1% to 3.1% as tokens grow. The reduce-scatter buffer stays nearly constant from 16 to 64 GPUs at a fixed per-GPU output size — the design does not degrade as you scale.
Where TIDE sits relative to the field, per the paper's Table 1 and §1: PyTorch's symmetric-memory FSDP2 path removes all-gather transport SMs but keeps rank-major outputs, full-group gradient packing, and ~13 reduce-scatter CTAs. NVIDIA's PACE extends direct placement to the reduce-scatter but stages and converts its multi-tensor input in a 32-CTA SM ring. AMD's MORI lands all-gathers within one node. Megatron-FSDP and veScale-FSDP avoid packing by replacing per-parameter storage with persistent flat windows.
The paper's claim — not verified anywhere else yet — is that none of those systems consumes per-parameter tensors directly across nodes while keeping fp32 reduction, taking transport off the SMs, and leaving FSDP2's storage contract intact. That is the practical appeal: it preserves FSDP2's data model, so the win is available without a storage redesign — if you fit the target contract (see Caveats).
Caveats
- Unreviewed preprint, no code. TIDE is arXiv v1 (28 Oct 2026), not peer-reviewed, and we found no code release. Every number above is author-reported and has not been independently reproduced.
- Narrow target contract. TIDE targets PyTorch 2.13, dimension-0 sharding, up to eight GPUs per NVSwitch domain, rail-aligned RDMA, NVSwitch multimem, and compute capability ≥ 9.0. CUDA-graph capture and asynchronous unshard are unsupported, and anything outside the contract falls back to stock collectives (alphaXiv). We could not confirm the exact testbed GPUs, interconnect, and software versions from the paper's evaluation section; those details live in §7, which was not fully accessible, so the per-model table (beyond the Qwen figure and the dense-Llama token-scaling curve in the intro) is not covered here.
- "Zero-copy" is scoped. The paper itself notes that required dtype casts, bounded multi-node staging, and the optional Slot-Reuse gradient copy remain, and that the reduce-scatter is not SM-free.
- Evaluation caveats the authors state. The overlap test is operator-level, not a training run, and its NCCL rows start from pre-packed fp32 input, so it does not prove CTA placement is the only interference source. The paper does include a 1,000-step convergence check whose stock-FSDP2 and TIDE loss curves overlap (arXiv HTML figure), but that is a sanity check, not proof of training equivalence at scale.