Korean RAG on a 3090: a local 27B tied Opus 5.5 — but two-thirds of misses came before generation, and a one-line tokenizer swap moved BM25 hit@5 from 56.2% to 80.3%

A developer scored his local Korean RAG pipeline on Allganize's 300-question Korean benchmark: with the same five retrieved pages in front of every model, a Qwen3.8 27B on a single RTX 3090 was statistically indistinguishable from Claude Opus 5.5. More usefully, roughly two-thirds of the top models' wrong answers were already lost before generation, in PDF parsing and retrieval. The full writeup and repro bundle are on his blog, with the r/LocalLLaMA thread summarizing it.

What shipped

The eval uses allganize/RAG-Evaluation-Dataset-KO (MIT): 300 questions over Korean public PDFs in finance, public sector, medical, legal, and commerce (60 each), each with a gold answer, a gold page, and an evidence type (paragraph, text, table, image). The dataset also ships the O/X-labelled 2024 answers of the commercial systems Allganize benchmarked. The author scored 299 questions (one has no gold page), plus 20 unanswerable questions he wrote. Because only 21 of 64 original source links were direct PDFs, he used the page-level mirror with PDFs embedded (64 docs, 1,450 pages), verified against the originals.

Method: every model received the same retrieved pages and the same instruction ("answer only from the excerpts; if it's not there, say you can't find it"). Retrieval was page-level BM25 (both tokenizers) plus bge-m3 dense, fused with RRF, top 5 pages fed to generation. Parsing was pymupdf4llm with tables as markdown; no OCR. Local models ran through Ollama at temperature 0; Claude models ran via API with adaptive thinking. Answers were judged by two LLMs — a qwen3.6 27B (main) and Opus 5.5 (second) — with the main judge first validated against the dataset's 2024 labels (88.5% agreement, Cohen's κ 0.76).

The headline tie: qwen3.8 27B with thinking on scored 77.3% vs Opus 5.5's 76.6% under the main judge (McNemar p=0.82, 10 vs 8 discordant). Under the second judge it was 80.6% vs 80.3% (p=1.0). The rest of the field: Sonnet 5.5 73.9%, Haiku 5.5 72.2%, gemma4 31B 70.2%, gpt-oss 20B 69.6%, gemma4 12B 66.6%, qwen3.5 9B 65.6%, gemma4 26B MoE 63.5%, EXAONE 3.5 7.8B 63.2%.

Turning thinking off was the single biggest generator-side lever: qwen3.8 dropped to 71.2%, clearly behind Opus under both judges (p=0.009 / ≤0.001). Thinking on was worth about six points overall and the most on table questions (68% → 80%), for 2.3× the time per question on the 3090 (4.1 s → 9.7 s).

Error accounting. Opus 5.5's 70 misses split into 22 where the gold page has no text layer (a parsing failure), 23 where retrieval ranked the gold page out of the top 5, and 25 where the model had the right page and still answered wrong. Qwen3.8 thinking's 68 misses: 22/22/24.

"At the top, parsing and retrieval each cost about as much as generation," he concludes. And retrieval has an obvious, cheap fix: tokenization. Korean is agglutinative — 은행법 (Banking Act), 은행법에 (in the Banking Act), 은행법상 (under the Banking Act) are three different whitespace tokens. Swapping BM25's whitespace tokenizer for a Korean morphological analyzer (Kiwi, via kiwipiepy, keeping noun/verb/adjective stems plus numbers and Latin tokens) moved hit@5 from 56.2% to 80.3% — a one-function change:

from kiwipiepy import Kiwi
kiwi = Kiwi()
KEEP = ('NN', 'VV', 'VA', 'SL', 'SN', 'XR', 'SH', 'MM')
def tok_kiwi(s):
    return [t.form.lower() for t in kiwi.tokenize(s) if t.tag.startswith(KEEP)]

Full hit@5 table: whitespace BM25 56.2%, kiwi BM25 80.3%, bge-m3 dense 77.3%, hybrid (kiwi + bge-m3, RRF) 81.3%. The parser choice barely mattered (PyMuPDF 81.3%, pymupdf4llm 81.3%, pypdf 77.9%), and the same lesson applies to Japanese, Turkish, Finnish, and other suffix-heavy languages.

The OCR gap dominates the 2024 comparison. 178 of 1,450 pages (~12%) yield fewer than 50 characters — scans and image-only pages — and four documents have no text at all (a 32-page KPMG report, an 11-page hospital cancer guide, a 9-page court case, a 6-page journal article). 23 of the 299 questions have gold pages in that set. Re-judged with the author's own judge, Allganize's 2024 Alli + Claude 3.5 Sonnet system (80.6%) beat his 2026 pipeline + Opus 5.5 (76.6%). The entire gap is those 23 questions: Alli got 19 right, Opus got 1. On the other 276 questions the pairing was 25 vs 31, p=0.50; on the 242 non-image questions both scored exactly 83.9%. "Two years of model progress didn't make up for missing OCR," he writes. Tables went the other way (Opus 80% vs Alli 64%, p=0.057). The 2024 LangChain quickstart + GPT-4-turbo (56.2%) lost clearly to a qwen3.5 9B on a single GTX 1080 Ti running this pipeline (65.6%, p=0.003).

A settings mistake that looks like a model failure. gpt-oss-20b cannot turn reasoning off — the OpenAI model card offers reasoning levels low/medium/high, no off. Under the author's standard 1,024-token output cap, 48 questions thought to the cap and returned empty and 28 more were cut mid-sentence — 57.5% overall. A rerun at an 8,192 cap scored 69.6%, with 2 questions still empty and counted wrong. "Ranking with that first number would have blamed the model for a settings mistake." His advice: always count empty and truncated answers after a run — errors accumulate silently (an older Ollama build also failed all 320 gemma4 12B questions with unknown model architecture: 'gemma4-assistant').

A follow-up on Oct 22 ran KRAFTON's new Korean-focused Raon-1 mini — a 30B-A4B MoE preview (28.8B total / 4.3B active per the model card, released Oct 21) — through the same set: 74.2% with thinking vs 72.6% without on the main judge. At temperature 0 with thinking on, it exhausted the 8,192-token cap on 16 of 320 questions with 14 empty answers.

Why it matters

For this workload — Korean PDF QA with gold-page evidence — model choice past a mid-size local model is rounding error. The 27B-vs-Opus tie is individually fragile (more below), but the error decomposition is not: for the top models, about two-thirds of misses happen before the generator ever sees the evidence, and retrieval gave up one full category of miss for a single function call. Teams should be spending the OCR and tokenization budget before the model budget.

The practical takeaway: tokenize retrieval with a morphological analyzer for agglutinative languages, run OCR (or a vision step) on pages that yield near-zero text, and count empty answers — truncation and harness failures masquerade as model quality. Hold the retrieved context constant when comparing models so you can attribute errors to parser, retriever, or generator.

Caveats

• All accuracy numbers are the author's own measurements from single runs — local models at temperature 0, Claude at default sampling — not independently replicated (a separate reviewer did re-derive the numbers from the raw files before publication, and the corrections are folded in). He is unusually explicit about the noise: the same qwen3.8 27B scored 71.2% on the 3090 and 72.2% on two 1080 Tis, with 11 verdicts flipping — by his own rule, 1–2 point gaps mean nothing.
• Both judges are LLMs. The main judge is the same model family as the headline model (mitigated by the second, different-family judge), and the Opus judge was measurably kinder to Claude answers (+4 to +6 points vs −2 to +5 for local models).
• The 2024 vendor scores were re-judged with the author's judge and came out 3–10 points below the published numbers; that is a judge-vs-judge comparison — Allganize scored its 2024 systems with its own auto-evaluator ensemble, and the author re-scored the same answers with his LLM judges — so part of the gap is evaluation methodology, not system quality.
• The 20 unanswerable questions were too easy: every model, 7.8B to Opus, refused all 20 — and his own "EXAONE invented 8 answers" scare turned out to be 8 refusals his detection regex missed, a good reminder that rule-based judging needs eyeballing.
• Raon-1 mini is a preview release and KRAFTON says specs may change; the benchmark scores there are the author's measurements, not KRAFTON's.

Sources

• Author's writeup: "local RAG, 27B ties Opus on Korean documents" (bric.pe.kr)
• Korean version of the same post (same numbers)
• r/LocalLLaMA thread
• Author's Raon-1 mini follow-up
• Repro bundle: scripts and per-question verdicts (ko-rag-eval.zip)
• Allganize RAG-Evaluation-Dataset-KO (Hugging Face)
• datalama page-level mirror with PDFs (Hugging Face)
• KRAFTON Raon-1-mini-preview model card
• KRAFTON Raon-1 mini release coverage (Yonhap)
• OpenAI gpt-oss-20b model card
• Kiwi Korean morphological analyzer (GitHub)
• Allganize's 2024 RAG leaderboard announcement