One process, two runtimes: Project Maya's HIP expert helper runs a GLM MoE across an RTX 5090 and a Strix Halo iGPU
PR #879 loads HIP into Maya's CUDA engine so a Radeon 8060S computes complementary experts — author-measured +47% decode, welcomed upstream, not yet merged

A pull request now open on Project Maya loads a HIP helper module into its CUDA engine, so one process runs a single MoE model across an RTX 5090 and a Radeon 8060S iGPU at once. On the author's rig, GLM-5.3-Flash-Maya-M decode went from 30.65 to 45.10 tok/s over stock CUDA+CPU — +47% with no speculation, +97% with Maya's built-in MTP2 drafter.
What shipped
PR #879 by RenWuJun adds an opt-in "HIP expert helper" to Project Maya's CUDA engine. Project Maya is Peasant Smith's (@mw00) engine for running datacenter-scale MoE models on consumer hardware: it keeps the experts a conversation uses most in GPU VRAM, the next tier in RAM, and the rest on NVMe, moving them as you chat. It normally builds as either a CUDA or HIP engine — one backend per process.
The PR's split is at the expert level. The CUDA backend on the RTX 5090 keeps attention, dense/shared layers, and its expert cache, while a separately compiled HIP module — built and running in the same process — stores and computes the complementary GLM experts on the Radeon 8060S iGPU in a Strix Halo APU. The runtimes talk through a small versioned C ABI that passes host data and opaque contexts; allocations, device ordinals, and runtime handles stay inside the runtime that owns them.
The mechanism reuses Maya's existing GlmExpertHelper (CUDA/HIP expert kernels, prefill staging, and result combination), which upstream already uses for AMD-only rigs via MAYA_GLM_EXPERT_HELPER=1 — Maya's README benchmark table shows a Radeon 8060S helper beside a Radeon 7900 XTX. What the PR adds is cross-vendor: a HIP helper running alongside a CUDA primary, so the 8060S joins an NVIDIA card instead of an AMD one.
The helper adds bounded persistent expert storage with generation-checked uploads, serialized completion and error propagation, and an optional grouped mode that overlaps a bounded share of expert rows with CUDA work (the tested 25% setting is an admission cap, not a fixed compute split). An opt-in cost-based prefill admission policy (STRATA_GLM_HELPER_COST=1) watches helper time per row and CUDA staging time per MiB. A setup script (setup-cuda-hip.sh) builds a matched CUDA engine + external HIP executor pair and labels the result "CUDA + HIP" in the Models page.
Benchmarks (v1.1.55, GLM-5.3-Flash-Maya-M IQ2_S, 9216-token context, INT8 KV, maya.py --bench, two repeats per mode):
| Configuration | Decode tok/s | 2K prefill tok/s | 8K prefill tok/s |
|---|---|---|---|
| Stock CUDA+CPU, no speculation | 30.65 | 213.5 | 771.5 |
| CUDA+HIP, no speculation | 45.10 | 332 | 969 |
| Stock CUDA+CPU, built-in MTP2 | 32.95 | 209 | 752.5 |
| CUDA+HIP, built-in MTP2 | 64.85 | 321 | 932.5 |
| Stock CUDA+CPU, DFlash2 (up to 4 drafts) | 36.80 | 213.5 | 768.5 |
| CUDA+HIP, DFlash2 (up to 4 drafts) | 69.05 | 321.5 | 954 |
All numbers are the author's own measurements on a Ryzen AI Max+ 395 (128 GB unified RAM) with an RTX 5090 32 GB attached over PCIe 4.0 x4 OCuLink, Ubuntu, CUDA 13.1 / HIP 7.1. The author's announcement titles say "up to 48% faster," but the documented decode gains are +47% (no speculation), +97% (MTP2), and +88% (DFlash2). The cost-based admission policy alone lifted mean 2K prefill +15.3% (330.5 → 381 tok/s over two runs per arm) with decode essentially unchanged (−0.1%).
The maintainer has not merged it yet, but the response is unusually specific — mw00 commented that "running an NVIDIA card and an AMD GPU together in one PC is something we really want," called the separately built HIP library with a small versioned interface "a clean way to do it," and plans to test it on his own NVIDIA + AMD Linux setup before deciding. The PR is scoped deliberately: Linux, one primary GPU, native GGUF GLM experts (width 4096, intermediate 2048, at most eight routes). Windows, EXL3, DeepSeek, other model shapes, and primary/helper hot-expert exchange are explicitly out of scope.
Why it matters
The reusable idea here is the process architecture, not the speedup. Two vendor runtimes coexist in one process behind a small C ABI, with each runtime owning its own allocations, device ordinals, and error handling. That is a pattern you can steal for heterogeneous inference beyond Maya: the primary GPU keeps attention and the hot expert cache, and a cheap secondary device — an iGPU that would otherwise sit idle, or a low-end card in a PCIe slot — carries the complementary experts an MoE router rarely calls.
The second takeaway is the mixed-vendor inference gap the author was trying to close. He writes that no engine he found could run one model on both of his GPUs at once — vLLM serves all-CUDA or all-AMD, and llama.cpp's Vulkan backend supported both but was weak at prefill on the 5090. Reddit commenters push back on the premise: llama.cpp can be compiled with both GGML_CUDA and GGML_HIP enabled in one binary. That is true but a different granularity — llama.cpp's multi-backend builds split at the tensor/layer level, not at the expert level with a dedicated helper runtime holding a complementary expert cache. Most commenters report that CUDA+ROCm in one llama.cpp binary is faster than Vulkan (one dissenter says the opposite) — which is precisely the mode this PR ports to Maya's expert-split architecture.
Do the speculative-decoding gains mean the helper makes the drafter smarter? No: draft acceptance is flat between stock and hybrid (MTP 82.93% vs 83.25%, DFlash 95.53% vs 95.65%). The +97%/+88% MTP2/DFlash2 decode numbers are throughput, not accuracy — the iGPU is doing real expert compute that the 5090 would otherwise serialize with its own.
Caveats
- Author-measured, not independently reproduced. The maintainer has not run it yet, and no third-party results were found. The verification described in the PR is solid for correctness — all 108 real-weight Maya-M expert arrays matched the previous HIP 7.1 fixture bitwise, sequential logits at 1025 tokens match exactly, 793 Ubuntu tests pass — but nothing here is a vendor-neutral benchmark.
- The speedup comparison is not a code-only A/B, by the author's own description. Stock used a 4 GiB UMA reservation and 119 GiB host RAM with native CPU/RAM expert scheduling active; the hybrid run used 96 GiB UMA with CPU expert computation disabled. Different memory reservations and settings — "practical configurations," in his words.
- The rig is bandwidth-starved in a way that flatters the numbers. The 5090 is on PCIe 4.0 x4 OCuLink (~8 GB/s per direction), far below the 5.0 x16 a desktop 5090 expects. Absolute decode rates will not transfer to a normally-connected card; whether the relative expert-offload gain survives a faster link is untested.
- Small samples. Two repeats per mode; the adaptive admission policy got two runs per arm. The author flags his own 8K prefill gain from cost-based admission as not a "general speed guarantee."
- Output determinism is not guaranteed. Batched logits differ between policies (reported KL ~1e-6, same top token), and the author notes stock-versus-hybrid generated-output equality was not established.
- The "48%" framing is the author's. It appears only in his announcement titles.
Sources
- PR #879: HIP expert helper for Maya's CUDA engine
- Maintainer comment on PR #879
- Project Maya README (model tiers, benchmarks, backend separation)
- Upstream AMD expert-helper docs (
AMD_MAYA.md) - Author's fork: benchmark methodology (
BENCHMARK.md) - Author's fork: expert-plugin design (
EXPERT_PLUGIN.md) - Announcement on r/LocalLLaMA (comments via redlib mirror); HN thread
- GLM-5.3-Flash model card (zai-org, MIT, 320 B total / 18 B active)
- Radeon 8060S specs (40 CU, RDNA 3.5, up to 2.9 GHz)