Richland Technology
Richland Technology
← Back to Blog
llmopen-sourcemoetrainingmegatron-lm

HuggingFace's open MoE training handbook: fp32 routers, a silent 25× TCP fallback, and a top-k serving trick from ~800k GPU-hours

HuggingFace's open Megatron-LM MoE training guide, distilled from ~500k–800k GPU-hours, with runnable configs and a recommended flag block.

Richland AI Desk·8 November 2026·4 min read
HuggingFace's open MoE training handbook: fp32 routers, a silent 25× TCP fallback, and a top-k serving trick from ~800k GPU-hours

HuggingFace released a ~70,000-word handbook for training Mixture-of-Experts models with Megatron-LM, distilled from roughly half a million to 800,000 GPU-hours of real MoE runs. It ships with runnable Slurm scripts, 55 interactive figures, and a single recommended Megatron flag block. The claims are concrete and falsifiable: keep the router in fp32, expect a silent 25× slowdown if NCCL falls back to TCP sockets, and use a top-k renormalization trick that serves a top-8-trained model at top-4 for a statistically insignificant quality loss.

What shipped

The handbook, published November 5, 2026, is credited to Nouamane Tazi, Leandro von Werra, and Thomas Wolf, and announced on Reddit's r/LocalLLaMA. It walks a reader who has trained a dense model through their first MoE: chapter 2 is a Megatron-LM runbook (test hardware, build the image, train a dense baseline, then a first MoE on one node), chapter 3 takes the MoE layer apart piece by piece, and chapter 4 covers serving and RL. Chapters 3 and 4 need no GPU.

The reference environment is a Slurm cluster of eight-GPU nodes — the written setup uses AWS P5 nodes with eight H100 80GB GPUs and EFA, with Megatron pinned at commit d6316eb45. The worked example is a 6.86B-parameter MoE with 1.22B active per token: 64 experts, top-8 routing, and expert width 1,024 so the 8 active experts match the dense FFN size. Alongside the handbook, HuggingFace released the scripts as a gist and a gallery of its figures.

Why it matters

The value is in the per-setting recommendations, most of which come with a number attached.

Keep the router in fp32. The handbook calls fp32 routing "the one setting to always use": a bf16 router can round two close logits to the same value and flip which expert wins. This is a one-flag change (--moe-router-dtype fp32).

Watch for a silent 25× network fallback. Two of the authors' runs set FI_PROVIDER=efa and named an OFI plugin, yet NCCL logged "Using network Socket" — traffic had fallen back to TCP, making the runs "about 25 times slower" than expert-parallel on EFA, with no error in the logs. The fix is to put the EFA RDMA libraries first on LD_LIBRARY_PATH and set NCCL_NET_PLUGIN=ofi by name (a path, not a name, left NCCL on sockets), then confirm with NCCL_DEBUG=INFO that it reports "Using network Libfabric."

All-to-all is the bottleneck to engineer around. In an eight-H100 benchmark, exposed all-to-all communication consumed about half of a step. The handbook's overlap flags (--overlap-moe-expert-parallel-comm --delay-wgrad-compute) delivered up to 35% higher throughput on two nodes in the authors' runs, at the cost of rank 0 reserving 5–10 GB more memory.

Load balancing is a speed problem as much as a quality one. In the authors' runs, forced-even routing reached 149.4 TFLOP/s versus 100.5 TFLOP/s for forced-skewed routing; a learned router started near the worst case and climbed to about 125 TFLOP/s after ~120 iterations. Start with an auxiliary loss at 1e-3 and watch MaxVio alongside LM loss; run a forced-even control once to set a speed target, but never train with it.

Expert granularity has diminishing returns. In an 18-layer ablation with fixed parameters and compute, loss fell from 2.334 to 2.297 at 128 experts of quarter width, and finer splits gave no further gain.

The serving trick. On a Qwen3.6-35B-A3B model trained with 256 routed experts at top-8, the handbook (citing Chen & Yao, arXiv 2609.04575) reports that serving at top-4 and dividing the surviving weights by the top-16 softmax mass loses only −0.35 MMLU points (p = 0.66, not significant), versus −4.65 points with plain renormalization over survivors and −27.45 points at top-6 with no renormalization at all. That halves activated routed parameters per token, from 1.006B to 0.503B.

The same source reports the effect is model-dependent: on Gemma-4-26B-A4B, dropping renormalization at top-6 raised perplexity only 0.7% versus 356% on the Qwen model.

Caveats

The handbook is a vendor-authored recipe book, not a peer-reviewed benchmark, and several headline numbers are borrowed rather than measured. The chapter-4 serving math is largely cited from Chen & Yao (arXiv 2609.04575) and Gu et al. (arXiv 2509.21892); the load-balancing MaxVio and 1,024-expert results are cited from Wang et al. and Li et al. The authors flag that their own throughput numbers are cluster-specific and may not transfer to other clusters.

There is also an unresolved discrepancy in the headline figure: Thomas Wolf's promotion cites ~800k GPU-hours, while Tazi's own thread says "1/2M" (~500k). The announcement also mentions the Alps GH200 cluster at CSCS, while the handbook's written reference environment is AWS P5 H100 nodes. Treat the specific numbers as starting points and re-measure on your own hardware.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.