Chrome's Modern Web Guidance lifts coding-agent pass rates from 55% to 92% — and the 507MB→7MB retrieval engine is the real story
An open-source agent skill from Chrome, Edge, Firefox, and MDN — and the retrieval engine that went from a 507 MB native stack to a 7-line embedding loop.

Google Chrome shipped Modern Web Guidance 1.0 on November 10: an open-source "agent skill" plus CLI that injects curated, expert-written web-platform knowledge into coding agents. Chrome's own evals show agents going from roughly 55% to 92% pass on ~200 web tasks. The project is now stewarded jointly by Chrome, Microsoft Edge, Mozilla Firefox, and MDN. The retrieval engine behind it is the more interesting story: a 507 MB native RAG stack rebuilt three times down to a 7.1 MB pure-JS package whose query embedding is a 7-line for loop.
What shipped
Modern Web Guidance is an npm package (modern-web-guidance, v1.0.0, Apache 2.0) that installs as a skill into Claude Code, Codex CLI, Antigravity CLI, Gemini CLI, GitHub Copilot, Cursor, Grok, and others. It tells the coding agent how to call a local CLI — search "<query>", retrieve "<guide-id>", and legacy-scan <path> — to pull the right guidance for a frontend task (get-started docs, announcement).
The content is 203 use-case guides covering 250 web-platform features — CSS & Layout (140), HTML & DOM (36), JavaScript & APIs (74) — written by browser engineers, web standards experts, and accessibility specialists, with ~70 guides getting a human accessibility review (release notes, announcement). The point is to stop agents from defaulting to ~2019-era <div>-and-JS patterns when the platform now has native primitives: CSS.highlights for text-range highlighting, <dialog> + showModal() for backdrops and focus management, field-sizing for auto-growing textareas.
The headline numbers are the eval. The team runs Claude Code, Codex CLI, and Antigravity CLI against ~200 tasks with ~1,700 Playwright/AST assertions, once without the skill and once with it. Aggregate pass rate goes from ~55% unguided to ~92% guided; the README's latest eval table shows 54%→82% (Codex), 57%→88% (Claude Code), and 58%→94% (Antigravity) (HN post, repo README). The graders open the finished app in a real browser and check DOM, styles, and source; accessibility checks cover keyboard support, focus and inert, prefers-reduced-motion, ARIA, and form labels — explicitly "not a full WCAG audit."
Why it matters
The retrieval engine is where this gets interesting. Paul Irish's technical deep dive describes three generations over six months (deep dive):
- Gen 1: a 507 MB unpacked
node_modules— LanceDB + Apache Arrow + Transformers.js ononnxruntime-native, a C++/node-gypbuild that "routinely fails in cloud sandboxes." - Gen 2: 23 MB pure JS —
all-MiniLM-L6-v2on TensorFlow.js, with the CPU backend pruned from 400+ modules to the 26 kernels MiniLM uses. - Gen 3: 7.1 MB — guide vectors pre-baked at release time in the cloud with
gemini-embedding-001, shipped as binary 256-dim Int8 vectors, plus a 29,165-word query lookup table tuned offline against 33,000 real web queries. Runtime query embedding is a 7-lineforloop.
The scoreboard: npm tarball 507 MB → 7.1 MB; cold search latency ~35 ms → 185 ms; warm query embed ~34 ms → 30 µs; peak RSS ~400+ MB → 145 MB. Search quality got better, not worse: Hit@K went from 80.6% to 95.4%, versus 96.9% for the cloud embedding API itself (deep dive).

The honest failure narrative is the part worth stealing. The Gen 2 MiniLM model was trained around 2018–2020, so its WordPiece tokenizer "literally does not know post-2020 web APIs exist" — fetchpriority gets shredded into fetch·##pr·##ior·##ity. A strict 0.38 cosine-similarity cutoff then rejected valid matches ("confirm destructive action" scored 0.349 against the dialog guide), spiking empty [] responses 8x, and the fallback dumped ~200 guide descriptions (~1,500 wasted tokens). The fix was hybrid BM25 keyword + semantic search, plus returning 10 compact {id, description} results in the same ~530-token budget as 5 verbose ones. The skill description itself is ~205 tokens at rest; retrieve pulls a full guide at ~800 tokens (deep dive).

Two design decisions are worth noting for anyone building agent tooling. First, the team deliberately avoided MCP: "MCP tools either waste tokens or have unreliable trigger mechanisms (due to dynamic tool discovery)," so the skill description is minmaxed for activation instead (deep dive). Second, the legacy-scan subcommand is a deterministic, zero-LLM static scan — AST/DOM/CSS parsers over HTML, CSS, JS/TS, and package.json — that flags old patterns for modernization. Candidate rules were tested across ~45,000 open-source repos split in half; to ship by default a rule must hit at least 80% static matcher precision and 90% post-verification precision on unseen code, giving a finding roughly 92% probability of not being a false positive (modernization workflow). It only covers the ~20% of guides where legacy code can be spotted deterministically, and there's deliberately no --fix flag.
Caveats
Every headline number here is vendor-reported by the Google Chrome team — the pass rates, the RAG size/latency figures, the legacy-scan precision, the guide counts. None have been independently benchmarked. The methodology is at least inspectable: the Playwright graders, base apps, and prompts live in the open-source modern-web-guidance-src repo, and the eval dashboard publishes per-agent/per-guide jobs (eval dashboard).
A few things to weigh:
- The "55% → 92%" is an aggregate. Per-agent spread is real: 54%→82% for Codex versus 58%→94% for Antigravity in the README's latest eval table. The eval also excludes trials that hit infra errors or time out, and the task prompts never mention Modern Web Guidance, so the agent must self-trigger.
- Chrome-centricity is the main criticism. The HN thread flagged ~40 features the guides recommend that are stable in Chrome but not in Firefox or Safari (e.g.
@function,calc-size(),interpolate-size, the Prompt API,fetchLater()). The maintainers respond that guides document full browser-support tables and cross-browser fallbacks, and that the default target is Baseline Widely Available (interoperable ≥30 months) (HN thread). - Internal number discrepancies. The legacy-scan training set is "30k GitHub repos" in the HN post but "approximately 45,000 open-source repositories" in the docs; package size is "~7MB" (7.1 MB
.tgz) in the deep dive but "drops from 23.5 MB to 8.8 MB" in the release notes. Not reconciled publicly. - Telemetry is on by default (opt-out via
DISABLE_TELEMETRY=1). The deep dive says it logs only the agent's 2–5 wordsearchargument, never user prompts or repository code. - The negative-result claim is unenumerated. The team says "some of the things we thought AI didn't know turned out to be wrong," but the only public specifics are about their own retrieval stack, not about model knowledge.