Richland Technology
Richland Technology
← Back to Blog
open-sourceagentsevaluationresearchbenchmarkmcpfault-injection

MCP faults mined from 57,240 real issue threads cut agent scores 60% — and gpt-oss-120b tolerates them no better than the 20B

A NeurIPS 2026 fault-injection study finds the agent framework you pick moves ~0.8/10 more than scaling the model; MCP-Injector and all 1,350 experiment cells ship open-source.

Richland AI Desk·26 October 2026·6 min read
MCP faults mined from 57,240 real issue threads cut agent scores 60% — and gpt-oss-120b tolerates them no better than the 20B

Faults mined from the real MCP ecosystem show that scaling up the model does not make tool-calling agents more robust. Under a calibrated fault profile, OpenAI's gpt-oss-120b completed fewer tasks than gpt-oss-20b (91.1% vs 95.6%), while the agent framework an engineer picks moved task-completion scores by ~0.8 points out of 10.

What shipped

The paper, "Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance" (arXiv 2610.32555, accepted as a NeurIPS 2026 poster), ships MCP-Injector: a drop-in, protocol-aware fault-injection middleware that interposes on the MCP tool-call path between the agent and its tool server. It requires no changes to the agent runtime, MCP client, or servers, and toggles on and off with a single configuration flag (paper, §3).

It runs as a proxy, looking like an MCP server to the agent and an MCP client to the real server, passing each invocation through four stages — Observe, Decide, Act, Record. Request-side interception synthesizes timing faults and explicit errors without touching the server; response-side interception mutates the server's real output before it reaches the agent (Figure 1 caption). Six fault operators cover the modes the authors observed in the wild, plus an OP0 passthrough: minor mistake (random character replacement), incorrect output (word-level substitution), invalid output (type coercion, e.g. str→int), explicit error (structured MCP error message), delayed response (artificial sleep), and silent drop (indefinite hang) (Table 1).

Faults carry a persistence schedule — permanent, transient, or intermittent — drawn from a seeded pseudo-random stream, so injections are deterministic given the seed and the call history, and every fired fault is logged (§3.2).

The calibration is what separates this from a chaos-engineering demo. The authors mined the public MCP ecosystem — 1,774 open-source repositories, 103,166 issue/pull-request threads, filtered to 57,240 English-language threads in a September 2025 snapshot — classified each by failure mode × persistence with GPT-4o at temperature 0, manually validated 117 threads (agreement κ ≥ 0.80), and synthesized a seedable fault profile (Figure 2 caption; mining README). The shipped profile weights (fault_mode_1.json): explicit error is 71.2% of incidents (incorrect output 13.7%, silent drop 8.2%, invalid output 4.1%, delayed response 1.5%, minor mistake 1.4%), and 91.3% of injected faults are permanent — the "transient blip, retry will fix it" story describes under 1% of the ecosystem.

The evaluation is a 1,350-cell full factorial: two backbones (gpt-oss-20b and gpt-oss-120b, served locally via LM Studio on a single Apple M3 Ultra with 96 GB RAM and a 16K context window) × three agent stacks (a custom multi-round plan-and-execute harness, the OpenAI Agents SDK, and LangChain) × three conditions (proxy off; proxy on with null injector; proxy on with the calibrated profile) × 25 single-server MCP-Bench tasks × three seeds (README; analysis notebook). A gpt-4o-mini judge scored each run 0–10; the primary analysis is intent-to-treat, so non-completed cells score zero (README).

Why it matters

Under the calibrated faults, task-completion scores collapse by 60%, from 8.08 to 3.23 out of 10, pooled across stacks and models (abstract; rq3_score_degradation_ITT.csv). Per metric: planning effectiveness −66.8% (7.51 → 2.49), parameter accuracy −63.6%, grounding −61.7% (same file).

The headline negative result is about scale. On the resilience-ratio model — the share of fault-free score retained under faults — the 20B and 120B are statistically indistinguishable (p = 0.96) (abstract; rq5_scale_stack_interaction.csv). Completion rate under faults is 91.1% for the 120B versus 95.6% for the 20B — a 4.4-point gap whose 95% CI [−0.2, 9.3] straddles zero, which the authors summarize as "if anything, lower" (abstract). Whatever the 20B→120B scale-up buys on capability benchmarks, it does not buy operational resilience to MCP faults.

Framework choice moves more. Under faults, LangChain scored 3.76, the custom plan-and-execute harness 2.98, and the OpenAI Agents SDK 2.96 — baseline scores were nearly identical (~8.0–8.2), so the gap is fault absorption, not capability (rq4_c_interaction_means.csv). A linear mixed-effects model attributes both gaps to steeper fault-condition degradation (mcpbench×faults −0.934, p = 0.048; openai×faults −0.842, p = 0.074, the latter suggestive rather than significant at the 0.05 level) (rq4_lme_results.csv). LangChain's advantage is not smarter retries: 23.5% of its post-failure actions are retries versus 33.0% tool switches, while the custom harness switches tools after 68.2% of faults and the OpenAI SDK simply stops (fallback, no further calls) after 49.0% (rq4_d_postfailure_pct.csv).

Post-failure responses per stack and backbone under the calibrated fault profile: the custom plan-and-execute harness switches tools after most faults, while the OpenAI Agents SDK stops after nearly half of them.

Naive retry is also expensive. The worst logged cascade: one task (scientific_computing_000, gpt-oss-120b, seed 3) burned 12 consecutive retries across 22 tool calls and 47,761 tokens (rq6_retry_loop_table.csv). Against a profile where 91% of faults are permanent, an unbounded retry loop is the worst possible policy.

Resilience heatmap under calibrated faults — the fraction of fault-free score each stack and backbone retains — showing the 120B backbone holds no advantage over the 20B.

The build changes are concrete. First, ship a fault-injection pass in your agent eval: MCP-Injector is drop-in and the whole 1,350-cell dataset, profiles, and analysis pipeline are public (GitLab repo, Apache-2.0 code / CC-BY-4.0 data, archived at Zenodo DOI 10.5281/zenodo.23573765). Second, treat retries as a bounded, framework-level decision with a switch-or-fallback escape hatch — model scale will not save you. Third, when comparing vendors or model upgrades, add a fault-condition score; capability benchmarks alone "mask operational brittleness," as the authors put it (abstract).

Caveats

  • The fault profile is a September 2025 snapshot of the open-source ecosystem, and the authors label it "a lower-bound stress test ... not a certification instrument" (README). Enterprise or proprietary MCP deployments may fail differently, and the specific weights are shipped config, not ecosystem ground truth.
  • The threat model is non-adversarial — misconfiguration, schema drift, malformed outputs — and deliberately excludes security attacks and model-internal faults; only the tool-call boundary is exercised, not MCP prompts, resources, or streaming (§2).
  • The experiment used two gpt-oss backbones on one machine (LM Studio, M3 Ultra, 16K context). The abstract's completion-rate gap (91.1% vs 95.6%) is not statistically significant, a point the authors themselves flag.
  • Scores come from an LLM judge (gpt-4o-mini) with intent-to-treat scoring; completed-only results are reported as a secondary analysis (README). End-to-end runs are not bit-reproducible because local serving and the judge are stochastic (§3.2).
  • This is a preprint-era, author-reported study; it is accepted at NeurIPS 2026 but the numbers above are as shipped in the artifact.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.