Richland Technology
Richland Technology
← Back to Blog
llmopen-sourceinferencemoegpu

One process, two runtimes: Project Maya's HIP expert helper runs a GLM MoE across an RTX 5090 and a Strix Halo iGPU

PR #879 loads HIP into Maya's CUDA engine so a Radeon 8060S computes complementary experts — author-measured +47% decode, welcomed upstream, not yet merged

Richland AI Desk·11 November 2026·6 min read
One process, two runtimes: Project Maya's HIP expert helper runs a GLM MoE across an RTX 5090 and a Strix Halo iGPU

A pull request now open on Project Maya loads a HIP helper module into its CUDA engine, so one process runs a single MoE model across an RTX 5090 and a Radeon 8060S iGPU at once. On the author's rig, GLM-5.3-Flash-Maya-M decode went from 30.65 to 45.10 tok/s over stock CUDA+CPU — +47% with no speculation, +97% with Maya's built-in MTP2 drafter.

What shipped

PR #879 by RenWuJun adds an opt-in "HIP expert helper" to Project Maya's CUDA engine. Project Maya is Peasant Smith's (@mw00) engine for running datacenter-scale MoE models on consumer hardware: it keeps the experts a conversation uses most in GPU VRAM, the next tier in RAM, and the rest on NVMe, moving them as you chat. It normally builds as either a CUDA or HIP engine — one backend per process.

The PR's split is at the expert level. The CUDA backend on the RTX 5090 keeps attention, dense/shared layers, and its expert cache, while a separately compiled HIP module — built and running in the same process — stores and computes the complementary GLM experts on the Radeon 8060S iGPU in a Strix Halo APU. The runtimes talk through a small versioned C ABI that passes host data and opaque contexts; allocations, device ordinals, and runtime handles stay inside the runtime that owns them.

The mechanism reuses Maya's existing GlmExpertHelper (CUDA/HIP expert kernels, prefill staging, and result combination), which upstream already uses for AMD-only rigs via MAYA_GLM_EXPERT_HELPER=1 — Maya's README benchmark table shows a Radeon 8060S helper beside a Radeon 7900 XTX. What the PR adds is cross-vendor: a HIP helper running alongside a CUDA primary, so the 8060S joins an NVIDIA card instead of an AMD one.

The helper adds bounded persistent expert storage with generation-checked uploads, serialized completion and error propagation, and an optional grouped mode that overlaps a bounded share of expert rows with CUDA work (the tested 25% setting is an admission cap, not a fixed compute split). An opt-in cost-based prefill admission policy (STRATA_GLM_HELPER_COST=1) watches helper time per row and CUDA staging time per MiB. A setup script (setup-cuda-hip.sh) builds a matched CUDA engine + external HIP executor pair and labels the result "CUDA + HIP" in the Models page.

Benchmarks (v1.1.55, GLM-5.3-Flash-Maya-M IQ2_S, 9216-token context, INT8 KV, maya.py --bench, two repeats per mode):

ConfigurationDecode tok/s2K prefill tok/s8K prefill tok/s
Stock CUDA+CPU, no speculation30.65213.5771.5
CUDA+HIP, no speculation45.10332969
Stock CUDA+CPU, built-in MTP232.95209752.5
CUDA+HIP, built-in MTP264.85321932.5
Stock CUDA+CPU, DFlash2 (up to 4 drafts)36.80213.5768.5
CUDA+HIP, DFlash2 (up to 4 drafts)69.05321.5954

All numbers are the author's own measurements on a Ryzen AI Max+ 395 (128 GB unified RAM) with an RTX 5090 32 GB attached over PCIe 4.0 x4 OCuLink, Ubuntu, CUDA 13.1 / HIP 7.1. The author's announcement titles say "up to 48% faster," but the documented decode gains are +47% (no speculation), +97% (MTP2), and +88% (DFlash2). The cost-based admission policy alone lifted mean 2K prefill +15.3% (330.5 → 381 tok/s over two runs per arm) with decode essentially unchanged (−0.1%).

The maintainer has not merged it yet, but the response is unusually specific — mw00 commented that "running an NVIDIA card and an AMD GPU together in one PC is something we really want," called the separately built HIP library with a small versioned interface "a clean way to do it," and plans to test it on his own NVIDIA + AMD Linux setup before deciding. The PR is scoped deliberately: Linux, one primary GPU, native GGUF GLM experts (width 4096, intermediate 2048, at most eight routes). Windows, EXL3, DeepSeek, other model shapes, and primary/helper hot-expert exchange are explicitly out of scope.

Why it matters

The reusable idea here is the process architecture, not the speedup. Two vendor runtimes coexist in one process behind a small C ABI, with each runtime owning its own allocations, device ordinals, and error handling. That is a pattern you can steal for heterogeneous inference beyond Maya: the primary GPU keeps attention and the hot expert cache, and a cheap secondary device — an iGPU that would otherwise sit idle, or a low-end card in a PCIe slot — carries the complementary experts an MoE router rarely calls.

The second takeaway is the mixed-vendor inference gap the author was trying to close. He writes that no engine he found could run one model on both of his GPUs at once — vLLM serves all-CUDA or all-AMD, and llama.cpp's Vulkan backend supported both but was weak at prefill on the 5090. Reddit commenters push back on the premise: llama.cpp can be compiled with both GGML_CUDA and GGML_HIP enabled in one binary. That is true but a different granularity — llama.cpp's multi-backend builds split at the tensor/layer level, not at the expert level with a dedicated helper runtime holding a complementary expert cache. Most commenters report that CUDA+ROCm in one llama.cpp binary is faster than Vulkan (one dissenter says the opposite) — which is precisely the mode this PR ports to Maya's expert-split architecture.

Do the speculative-decoding gains mean the helper makes the drafter smarter? No: draft acceptance is flat between stock and hybrid (MTP 82.93% vs 83.25%, DFlash 95.53% vs 95.65%). The +97%/+88% MTP2/DFlash2 decode numbers are throughput, not accuracy — the iGPU is doing real expert compute that the 5090 would otherwise serialize with its own.

Caveats

  • Author-measured, not independently reproduced. The maintainer has not run it yet, and no third-party results were found. The verification described in the PR is solid for correctness — all 108 real-weight Maya-M expert arrays matched the previous HIP 7.1 fixture bitwise, sequential logits at 1025 tokens match exactly, 793 Ubuntu tests pass — but nothing here is a vendor-neutral benchmark.
  • The speedup comparison is not a code-only A/B, by the author's own description. Stock used a 4 GiB UMA reservation and 119 GiB host RAM with native CPU/RAM expert scheduling active; the hybrid run used 96 GiB UMA with CPU expert computation disabled. Different memory reservations and settings — "practical configurations," in his words.
  • The rig is bandwidth-starved in a way that flatters the numbers. The 5090 is on PCIe 4.0 x4 OCuLink (~8 GB/s per direction), far below the 5.0 x16 a desktop 5090 expects. Absolute decode rates will not transfer to a normally-connected card; whether the relative expert-offload gain survives a faster link is untested.
  • Small samples. Two repeats per mode; the adaptive admission policy got two runs per arm. The author flags his own 8K prefill gain from cost-based admission as not a "general speed guarantee."
  • Output determinism is not guaranteed. Batched logits differ between policies (reported KL ~1e-6, same top token), and the author notes stock-versus-hybrid generated-output equality was not established.
  • The "48%" framing is the author's. It appears only in his announcement titles.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.