Richland Technology
Richland Technology
← Back to Blog
llmopen-sourceinferencespeculative-decodingon-device

Garlic LLM runs Qwen3.5-9B on a stock PS4 GPU at 31–39 tok/s — using the model's own MTP head for speculative decoding

A GPLv3 homebrew app runs Qwen3.5-9B fully offline on a 1.84-TFLOPS GCN console — the same native MTP-head speculative decoding vLLM and SGLang use in the data center.

Richland AI Desk·13 November 2026·4 min read
Garlic LLM runs Qwen3.5-9B on a stock PS4 GPU at 31–39 tok/s — using the model's own MTP head for speculative decoding

A homebrew PlayStation 4 app runs Alibaba's Qwen3.5-9B entirely on the console's GPU, fully offline, at a self-reported 31–39 tokens/s. The technique that makes a 12-year-old, 1.84-TFLOPS GCN chip usable is the same native MTP-head speculative decoding that vLLM and SGLang ship for data-center serving.

What shipped

Garlic LLM v1.19 is a GPLv3 homebrew app by Leonardo Alegría that runs Qwen3.5-9B on a stock base/Slim PS4, with no PC and no internet while it runs (GitHub, Hugging Face package). Per the author's measurements, text chat runs at 31–39 tokens/s (up to 43 on code) with a 16K-token context; voice mode drops to about 22 tokens/s because the speech model takes the memory that speculative decoding uses.

It is a full appliance, not a toy demo. The app listens (NVIDIA's Parakeet TDT 0.6B v3, 25 European languages), speaks (Resemble AI's Chatterbox, seven languages), and runs the MicroPython it writes on the console itself. The all-in-one package is 5.66 GB with the models inside, installs like any other homebrew PKG, and needs a PS4 running GoldHEN (tested on firmware 11.52).

Garlic LLM answering a question on a PS4

Per the author's write-up, a PS4 gives homebrew no compute API and no machine-learning library, so the app talks to the GPU directly: the model's operations are OpenCL C kernels compiled by LLVM for the console's GCN architecture, dispatched by hand-written PM4 packets (the low-level commands a graphics driver would normally generate). Weights are repacked into two custom formats — Q4 (4-bit) for most matrices and the output head, and Q3 (~3.5 bits including scales) for the two largest matrices of each layer, the MLP gate and up projections. Storing everything in Q4 would cost about 384 MiB more than the console has free. The attention cache is quantized to 4 bits too, which is how a 16K context fits.

Why it matters

The load-bearing trick is speculative decoding via Qwen3.5's own MTP (multi-token prediction) head. It drafts up to two tokens ahead, and the full model verifies them in one pass — the author credits it with the jump from 22 to 31–39 tok/s on this old GPU.

This is not a PS4-specific hack; it is the same mechanism the data-center engines use. Qwen3-Next and Qwen3.5 ship a native MTP module that vLLM and SGLang expose as a speculative-decoding method (vLLM speculators docs, Qwen3-Next model card).

The MTP head is a single prediction layer. At each speculative step it takes two inputs — the verifier's hidden states and the token embeddings from the ground-truth input — normalizes them separately, concatenates and projects them down to the model's hidden dimension, then runs them through a standard decoder layer. Its output feeds the next step recursively while a shared LM head produces draft logits at each step.

Because the draft tokens are generated cheaply and the main model checks all of them in a single parallel forward pass, accepted tokens are effectively free: verification costs one pass regardless of how many candidates it checks (Red Hat on FastMTP heads).

The same inference technique that frontier serving stacks use to cut latency on H100s is what makes a 2013 console with 1,152 stream processors, 8 GB of unified GDDR5 at ~176 GB/s, and no GPU compute API viable as a fully offline 9B LLM appliance at usable chat speed. The MTP head ships inside the model — it is not a separate drafter you have to train or bolt on — so a bandwidth-starved device gets a lossless speedup for free, provided the acceptance rate is high enough to pay for the extra draft pass.

Caveats

Everything about Garlic's performance is self-reported. As of writing there is no independent benchmark of the 31–39 tok/s figure, no Hacker News thread, and no tech-press coverage; the Reddit threads are largely the author's own announcements. Treat the numbers as vendor claims.

The bigger caveat is that MTP speculation is not guaranteed to speed things up. A vLLM issue documents both significant speedups and significant slowdowns depending on model size, hardware, batching, and prefix caching — one reporter saw Qwen3.5-0.8B generation throughput drop from 342 to 128 tok/s despite 96.6% per-position acceptance, while a counter-test on an L4 found MTP consistently faster (+48% at k=1 up to +139% at k=3) (vLLM issue #39680). The PS4 result is therefore also a data point on where speculative decoding pays off: on a memory-bandwidth-bound device with a single request, where the draft pass is cheap relative to the full-model pass it saves.

A few other limits: the PS4 Pro and other firmware versions are untested; the author's "first public project to run an LLM on a PS4 GPU" claim is open to counterexamples; and Qwen3.5-9B is natively a 262K-context vision-language model, whereas Garlic runs it as a 16K-context text model — so don't conflate the model's capability with what the app exposes. The 147-page write-up, Nine Billion Parameters on a PS4, is pay-what-you-want from $19.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.