Co-installed skills silently substitute in coding agents — one in five runs, and pass/fail benchmarks can't see it
A new arXiv study of Claude Code finds similar skills swap in without lowering task completion; the conflict is decided at the first skill read, where a pre-tool hook can restore the lost behavior.
When two similar skills are co-installed in a coding agent, the model silently substitutes one for the other in one of every five runs — without lowering task completion, so the pass/fail benchmarks most skill evals use never notice. That is the central finding of a new arXiv study, "One Skill Too Many: How Co-Installed Skills Conflict in Coding Agents" (arXiv:2610.11647), which the authors call the first empirical study of skill conflicts. It comes with a concrete fix: the conflict is decided at the first read of a skill, almost always before any file changes, so a pre-tool hook at that read can restore the lost behavior.
What shipped
The study, from researchers at UNSW, Griffith, NTU, and Wollongong, reconstructs the skills that Claude Code would load in 20,947 public repositories, mines 822,109 candidate similar-skill pairs, has an LLM judge a stratified sample of 3,754, and runs 312 confirmed pairs on three models — 6,368 runs, 169,294 tool calls, and 542 agent-hours (abstract).
An agent skill is a directory whose SKILL.md file's name and description tell the model when to use it, followed by instructions and optional scripts. Skills arrive from independent sources: teams commit project skills to a repository, developers keep personal skills under their home directory, and plugins add skills to every project. They are now shared at scale — prior work has collected 138,133 SKILL.md files from 20,556 repositories and 42,447 skills from two marketplaces (intro). When two skills do the same job, the model picks between them from names and descriptions alone, and the installed skill can lose the behaviors the user wanted.
The paper reports five findings:
- Conflict-prone skills are common. Nearly one in four installed skills — about 10,100 when near-duplicates are counted once — is already co-installed with a skill that does the same job. In the 489 projects that copy a whole skill collection, 37% of judged skills have a same-job skill in the same project, versus 19% elsewhere. The authors call these estimates conservative: public repositories rarely show the personal skills and plugins developers add.
- Most conflict-prone pairs involve normative skills — roughly, skills that encode rules and conventions — followed by capability skills.
- Substitution happens without hurting task completion. A similar skill takes one in five runs away from the installed skill, and runs that open the similar skill first lose more than a third of the exclusive core functions — the behaviors only the installed skill fulfills.
- Install location decides which skill runs. Where a skill is installed (project, personal, or plugin scope) determines the outcome; listing order barely matters. And the model almost never tells you: the final reply names the skill used in only 0.9% of substituted runs.
- The conflict is decided at the first skill read, almost always before the agent changes any file — and a pre-tool hook at that read restores fidelity on exclusive core functions to the level of runs that open the installed skill first.
The authors define a skill's core functions as requirements a competent model would not meet without the skill, quoted from the skill's text and checkable in the output; the first three that apply to a task are fixed before any run, and the share fulfilled is the run's fidelity. A core function is exclusive when the similar skill does not ask for it. In the experimental pairs, the similar skill does not ask for 73% of the installed skill's core functions that apply to the task — so substitution is not a harmless swap. The paper distinguishes substitution (the similar skill runs instead) from interference (the installed skill runs but the result changes).
One project co-installs anthropics-skills--docx and openai-skills--doc, both of which handle Word documents; another co-installs qa-patrol with its variant qa-patrol-team. In a public issue report, a developer asked for the self-improvement skill of a project, but a personal fork of the same name ran instead and targeted an unrelated repository — noticed only after reading the output. In one of the study's own runs, a similar skill led the agent to commit work the installed skill forbids, without saying which skill it used, and the task was still completed.
Why it matters
The headline result is that task completion is the wrong yardstick for skills. Teams use skills to encode conventions and safety rules — a pull-request format, a release checklist, a ban on touching git. A conflict can make the agent break those rules while the task still passes, which is exactly the case pass/fail benchmarks are blind to. The paper notes that SWE-Skills-Bench finds 39 of 49 skills yield zero pass-rate improvement, and prior work (Song and Wei, 2026) finds that expanding a skill library to 202 skills lowers pass rate by 21 percentage points on average — but those numbers only capture the cases where the wrong skill breaks the task. The silent-substitution case is invisible to them.
The fix is cheap because of when the conflict is decided: at the first read, almost always before any file is touched. A pre-tool hook there restores fidelity on exclusive core functions to the level of runs that open the installed skill first. The authors' recommendations for builders: benchmarks should score exclusive core functions rather than pass/fail, and platforms should guard the first read and show which skill ran.
For anyone shipping a coding agent with a skill system, the practical takeaways: treat co-installed same-job skills as a real failure mode, surface which skill actually ran (the model will not volunteer it — 0.9% of the time), and evaluate on the behaviors you installed the skill for, not just whether the task passes.
Caveats
This is a single-platform, unreplicated preprint (v1). Everything is measured on Claude Code, so the substitution rates and the effectiveness of the pre-tool hook may not transfer to other agent harnesses or models. The prevalence estimates rest on public repositories, which undercount personal skills and plugins — the authors themselves call the numbers conservative. The similar-skill pairs were judged by an LLM, and the fidelity metric is the authors' own construction, derived from skill text and checked against run output. All figures are the authors' measurements, not independent replications.