Small coding agents don't walk the code graph: tool gains peak at 4B, half is submission formatting from 8B up, and 32B agents stop calling the tools

A UT Austin preprint labels all 500 SWE-bench Verified tasks by the code-graph distance between the code an issue mentions and the code the fix touches, then ablates tool sets across ten small open-weight models. Embedding-search and code-graph tools mostly help by resolving names and expanding exactly one hop; the gain shrinks to 4–5 points at 32B.

What shipped

"How Do Small Coding Agents Use Code Graphs and Embeddings?" (arXiv:2610.33958, cs.SE) is a single-author preprint by Saurav Bellam, submitted 25 October 2026, 29 pages with 5 figures. It makes three moves: a deterministic difficulty label for every SWE-bench Verified task, a controlled tool ablation, and linear probes of whether small models represent the repository graph internally.

The difficulty axis. The paper defines "hop depth" as the shortest code-graph path from code the issue mentions to code the gold patch changes, computed with regular expressions and Python's ast module at each task's base commit — no LLM in the loop. Of the 500 Verified tasks, 186 sit at hop 0, 128 at hop 1, 78 at hop 2, and 76 at hop 3+ (with 31 no-mention and 1 unreachable). The label is robust to alternative call resolution (κ = 0.96–0.98) but sensitive to mention extraction: a strict extractor that counts only back-ticked code and traceback frames leaves 311 of 500 tasks without a mention. The abstract attributes the measure to a companion paper not released alongside this preprint.

The ablation. A localization-only agent runs in four arms that differ only in tool set: G (grep, read_file, list_dir), E (G + embedding search over function and class texts), T (G + a name lookup and one- or two-hop neighborhood query over a static code graph), and ET (all). The four arms run on a depth-balanced 160-task split and on all 500 tasks, across ten models — six Qwen3 sizes from 0.6B to 32B with five decodings each, an FP8 control, and three Gemma 4 sizes — for 79k episodes total.

The headline numbers:

• Structured tools roughly double grep-only accuracy at small sizes. Grep-only function Acc@10 is 12.8% for Qwen3-8B and 9.6% for Gemma 4 E4B; any structured tool about doubles it. The tool gain peaks at 17–21 points at 1.7–4B and falls to 4–5 points at 32B, an odds ratio of 0.77–0.82 per doubling of parameters.
• From 8B up, about half of that gain is the form of the submission, not discovery. With graph tools, 85% of the agent's top-5 function strings are exact entity ids versus 47% with grep only. A canonical rescoring that repairs unambiguous location-string formats closes roughly half the measured tool advantage — so part of what looks like "the tool found the code" is the tool making the answer easy to grade.
• The fall from 14B to 32B is a fall in tool uptake, not capability. The two sizes match on grep-only accuracy, but find_entity is called in 96% of episodes at 14B and 45% at 32B. The bigger model simply stops calling the tool.
• Checkpoint identity matters as much as size. The same 4B size gains +17.6 points from embedding search under one checkpoint and nothing under another.
• Depth beats every tool set. Each extra hop multiplies the odds of correct function-level localization by 0.50–0.67 at every size (0.47 for Qwen3-4B and 8B, 0.54 for Gemma 4 E4B, 0.55 for Qwen3-32B, all p ≤ 1e-3). No tool set flattens that gradient.
• Agents don't walk the graph. Traces show the graph tools' gain is name lookup at hop 0 and one neighborhood expansion at hop 1; agents chain expansions in only 12–14% of episodes.
• Grep-only small agents loop. Qwen3-8B with grep alone averages 10 grep calls per episode, 40% of episodes hit the step cap, and 41% never submit; a 50-step, 40k-token budget does not rescue it.
• Localization is only useful if it fixes. A fixed Qwen3-8B repair prompt resolves 6/13/12/13 of 160 tasks from the G/E/T/ET localizations (26 with oracle locations), and resolution is almost entirely conditional on correct localization (24–27% versus 1–2%).

The probes. The paper asks whether the repository graph is in the weights, not just the tools. Under a within-anchor, degree-stratified protocol (designed to defeat a node-level shortcut the authors found in a naive global probe), direct call edges are linearly decodable from Qwen3-4B and 8B activations at AUC 0.74/0.68 and 0.73/0.66 on held-out folds, and remain decodable inside real agent conversations at 0.61–0.66. But a one-line lexical name-match baseline hits 0.99, two-hop edges fall to 0.57 once the lexical cue is removed, and inheritance is not decodable beyond one-sided information. The synthesis: small agents carry a one-hop, largely lexical picture of their repository — in weights and in use.

What's released. The depth-labeling package is on arXiv now: hop_depth_labels.csv (one row per Verified instance, computed deterministically at each base commit), the graph-builder and labeller scripts, the difficulty-analysis and robustness scripts, and the odds-ratio refit. The arXiv comments promise that "full code, raw results and analysis will be released in the near future"; the abstract claims "We release the agent, probes, logs and traces," but the arXiv file listing does not yet contain them.

Why it matters

The premise behind embedding-search and code-graph tools is that repository structure is something the model lacks and the tool supplies. This paper's answer for small models is narrower than the premise: the tools' benefit is exact-name resolution plus one hop — which is also the extent of the structure the models appear to carry internally.

A cheap name index may capture most of the value. If agents use graph tools for name lookup at hop 0 and one expansion at hop 1, and chain expansions in only 12–14% of episodes, then a full graph-traversal stack is mostly dead weight for a small-model localization agent. An exact-name resolver plus a one-hop neighborhood query is the part that earns the 17–21-point gain at 1.7–4B.

Don't expect tools to flatten the depth gradient. Deeper tasks fail at every size with every tool set. If your eval mixes hop depths, a tool that helps at hop 0–1 will look better than it is; stratify by depth or you will misread where the gain comes from.

Rescore submissions before crediting tools. From 8B up, about half the measured tool gain is the form of the submission (exact entity ids) rather than discovery. An evaluation that counts formatting as localization will systematically over-credit structured tools relative to grep.

At 32B the problem flips to uptake. The model stops calling find_entity (45% of episodes versus 96% at 14B) at equal grep-only accuracy. If you ship graph tools with a larger model, the binding constraint is whether the model invokes them, not whether they exist.

The probe result is a warning about "the graph is in the model" claims. Direct call edges decode at AUC 0.74, but a lexical baseline hits 0.99 and two-hop edges collapse to 0.57. Whatever relational structure small models encode is shallow and lexical — a useful prior when deciding whether to rely on in-weights structure versus an explicit index.

Caveats

• This is a single-author preprint, days old, with no peer review and no independent replication; every number above is author-reported.
• The 32B result should not be read as an asymptote: 14B is the stronger checkpoint in this study, and the 32B fall is a tool-uptake effect at equal grep-only accuracy.
• The comparison with prior graph-agent work like LocAgent is not controlled — different models and agent designs.
• The probes test only linear readouts; nonlinear structure would not show up.
• Hop depth is a static shortest path, not the path an agent actually takes, and name-based matching makes hops optimistic; the README warns the labels must not be shown to a system under evaluation.
• Release status is partial: the depth labels and scripts are public now, but the agent, probes, logs, and traces are promised, not yet published.

Sources

• arXiv abstract: https://arxiv.org/abs/2610.33958v1
• Full paper (HTML): https://arxiv.org/html/2610.33958v1
• Ancillary hop-depth package README: https://arxiv.org/src/2610.33958v1/anc/swebench-verified-hop-depth/README.md
• SWE-bench (Verified = 500 human-filtered instances, SWE-bench × OpenAI): https://www.swebench.com/
• LocAgent (prior graph-agent work cited by the paper): https://arxiv.org/abs/2503.09089
• alphaXiv overview of the paper: https://www.alphaxiv.org/abs/2610.33958