Five agentic coding tasks on a 16 GB GPU: a quantized 27B matched the cloud model's 5/5 — at 185k tokens and 94 minutes vs 43k and 236 seconds
A practitioner benchmark of five agentic coding tasks on a 16 GB RX 9060 XT: the quantized 27B matched the cloud API's 5/5 — at 4.3x the tokens and 24x the wall time, with the pass/fail margins thinner than the scoreboard suggests
A practitioner benchmark on r/LocalLLaMA ran nine quantized local models plus one cloud API baseline through five agentic coding tasks on a single 16 GB RX 9060 XT. Two configurations scored 5/5: the cloud model and a Qwen3.8-27B quantized to IQ4_XS with an MTP draft head running on a KV-streaming llama.cpp fork — the local run took 185k output tokens and 94 minutes against 43k tokens and 236 seconds for the cloud. The headline isn't that local code quality caught up; it's where the costs actually landed.
What shipped
Each run starts in an empty directory with a single spec file, must produce solution.py and test_solution.py, run its own tests, and stop. Grading happens with a checker the model never sees: fixed cases from the spec, randomized cases against a brute-force oracle, and a named performance gate. A cell is capped at 1750 seconds; whatever is on disk when the clock runs out gets graded (post).
The tasks: quoting rules for one CSV record (c1), an arithmetic evaluator with pinned exception behavior (c2), a POSIX-ish shell tokenizer (c3), max-profit interval scheduling with two tie-breaks over 200k intervals in under 3 seconds (c4), and shortest-prefix pattern matching with 20 stars against a 4,000-character text in under 10 seconds (c5).
| model | c1 | c2 | c3 | c4 | c5 | score | out tokens |
|---|---|---|---|---|---|---|---|
| deepseek-v4.1-flash (cloud API) | P | P | P | P | P | 5/5 | 43k |
| Qwen3.8-27B-ASCII-Condensed UD-IQ4_XS + MTP (kvstream fork) | P | P | P | P | P | 5/5 | 185k |
| Qwen3.8-27B UD-IQ3_XXS + MTP Q4_0 sidecar | P | P | P | F | P | 4/5 | 125k |
| Swift-1.5-Qwen3.8-27B GSQ-RCO-IQ3_XXS, embedded MTP | P | P | P | P | F | 4/5 | 113k |
| Ornith-1.5-35B-A3B Q4_K_M | P | P | P | P | F | 4/5 | 57k |
| Qwen3.8-Flash-Next Coder GSQ-RCO-IQ1_M | P | P | P | P | F | 4/5 | 89k |
| Qwen3.6-35B-A3B UD-Q4_K_M (unpruned) | P | P | P | P | F | 4/5 | 126k |
| Nemotron-3.5-Bolt-74B-A9B IQ2_XXS | P | F | F | F | F | 1/5 | 51k |
| Mellum2.1-12B-A2.5B-Thinking Q6_K | F | P | F | F | F | 1/5 | 167k |
| LFM2.5-24B-A2B Q6_K | F | F | F | F | F | 0/5 | 187k |
c1 through c3 mostly sort models into "writes code and iterates" and "does not." Everyone who reached c3 wrote a single-pass scanner of 76 to 97 lines and passed, and nobody was tempted into shlex, which would have failed the comment rule. Nemotron-3.5-Bolt is the case study in not iterating: three of its cells made one tool call and then reported files that do not exist, and the c3 report still contained the agent template's placeholder "What was done."
c4 and c5 carry the signal. On c4, every model that produced a file used the same shape — sort the intervals, then a DP with bisect for the predecessor, which the Flash-Next Coder replaced with coordinate compression — and the losses were the tie-break rule, the cost of carrying interval lists through the DP, and duplicate-heavy input. LFM2.5's c4 is correct but slow: 6.98 s against the 3.0 s gate, because it copies the chosen interval list inside the DP. Qwen3.6-35B measured 3.03 s against the same gate, and re-grading the identical file gives 2.97 to 3.23 s — a pass decided by noise. The IQ1_M Coder's c4 row is a snapshot pass: the host OOM-killed the engine and the desktop session 684 seconds in, with the correct file already on disk.
On c5 the split is clean. All passes came from automata, not pattern strings: both Qwen3.8-27B variants built a Thompson NFA with epsilon closure (0.03 s), while the cloud run and the Flash-Next Coder used big-integer bitsets (the local one cleared its gate at 0.5 ms) (post). The failures came from parse-then-match routines that missed one rule each, or from handing the matching to Python's re, which backtracks on star piles. Mellum's regex-based c5 file needed a memory cap during grading because uncapped it reached 7.3 GB.
The two 5/5 runs are worth contrasting. The cloud model finished in 236 s and 43k tokens, writing a test file, running it, and stopping. The kvstream fork took 28 turns, hit the cell cap on c5, and its two capped cells were self-inflicted: it reported "all tests pass" (the grader agreed) and then hung a 9,000-case fuzz whose oracle translated patterns to re. Qwen3.8-27B's cheaper IQ3_XXS sibling scored 4/5 but burned 46k tokens of ls, pwd, and re-reading the spec on c4 with no code in its reasoning — no file at all.
Why it matters
Algorithmic quality holds up under brutal quantization, on this evidence. A 27B at ~4 bits per weight matched the frontier cloud model on all five tasks, and even an IQ1_M-quantized coder with experts cached in host RAM cleared four of five — its one miss being the disputed c5 spec call. The 74B Nemotron at IQ2_XXS collapsed — but that looks like agent behavior (one tool call, then report a file that doesn't exist) rather than a code-ability ceiling.
The local cost shows up in tokens and wall-clock, not correctness. 185k tokens and 94 minutes for the fork's 5/5, versus 43k and 236 s for the cloud. Ornith-1.5 was the cheapest local (57k tokens, ~16 minutes) and its only miss was one randomized c5 case where a group and a star interact. The cheapest 5/5 is the fork, and it earns the score through 28 turns and two capped cells. If you are pricing a coding agent on tokens or latency, the local option is not free — it is a different trade, not a discount.
The margins around the performance gate are thinner than the pass/fail column suggests. 3.03 vs 3.0 s. A correct file on disk at the moment the host OOMs. A bisect DP at 0.25 s versus coordinate compression when both clear the gate. On this class of task, gate-adjacent implementation choices (copying a list inside the DP, choosing the predecessor lookup) decide scores more than model "intelligence" does.
Fitting a 27B at full context on a 16 GB card is the enabling trick. The post names only "kvstream fork" with no link; the matching public project is llama.cpp-adaptive-kv-streaming, which keeps the full KV history in host RAM, streams pages into VRAM through a shared ring buffer, and explicitly targets this exact model file — Qwen3.8-27B UD-IQ4_XS with embedded MTP — on 16 GB. The chart below is from that repo, not from the author's runs; treat it as the mechanism behind the fork's row, not the run's measurement.
Sampler configuration is part of the result. The author ran Swift-1.5 at temperature 1.0 and calls it "an oversight." A commenter on the thread points out that the Qwen3.8-27B run used Qwen3.6's non-thinking preset (0.7/0.80/20 with presence penalty 1.5) while thinking was enabled, and relays Qwen's own thinking-mode recommendation of 1.0/0.95/20 with presence 0. Read the 4/5-to-5/5 band as "with that sampler," not "the model." (comment thread)
Caveats
- Everything here is author-reported, single-run, and not reproducible yet. The harness runs through the author's private "pi" subagent extension, which they say they are still "thinking about how to exactly share everything needed." No independent run exists; treat the table as one careful experiment, not a benchmark suite.
- The Flash-Next row was added by editing the post after it went up, and its only miss — reading
{inside a group as a literal where the spec wants an error — is one the author says "could've been interpreted as the correct solution as well." The c5 spec is ambiguous at that point. - Sampling settings were not controlled across models, including the thinking-preset mismatch above.
- The kvstream fork's identity is inferred. The post names the fork but links nothing; the linked repo is the best public match for the exact model file and GPU class.
- Quant bit-width figures (IQ1_M 1.75, IQ2_XXS 2.06, IQ3_XXS 3.06, IQ4_XS 4.25 bpw from llama.cpp's quantize table) come from a community reference, not from measurements in this run.