llmopen-sourceagentsbenchmarkmemory

Prompt rules as agent memory: IBM's ALTK-Evolve claims 67.5% Pass^4 on τ²-bench retail with a stock GPT-4.1 agent — and the leaderboard entry is still unmerged

ALTK-Evolve distills an agent's own trajectories into behavioral guidelines appended to the system prompt — no weight updates, and the fine-tuned 30B it beats is verifiable. The "#1" itself is IBM's snapshot, not the live board.

Richland AI Desk·20 October 2026·8 min read

Prompt rules as agent memory: IBM's ALTK-Evolve claims 67.5% Pass^4 on τ²-bench retail with a stock GPT-4.1 agent — and the leaderboard entry is still unmerged

IBM Research reports that a stock GPT-4.1 ReAct agent on the τ²-bench retail domain reaches 67.5% Pass^4 — the strict all-four-attempts metric — after distilling behavioral guidelines from its own training trajectories into its system prompt. That put it above every retail entry in IBM's October 12, 2026 leaderboard snapshot, including RAFT-30B-A3B at 61.4%, a 30B model fine-tuned on synthetic retail data — with no weight updates, no human annotation, and no change to the benchmark scaffold (Hugging Face blog).

What shipped

ALTK-Evolve is the customer-service branch of IBM's open-source agent-memory toolkit. The loop is simple: run the agent on training tasks, have the agent's own LLM read the trajectories and distill successes and failures into short behavioral rules ("Authenticate the user before taking any order-related action", "Take one action per step: never combine a tool call with a user-facing message in the same turn…"), consolidate duplicates, and append the survivors as a markdown list to the system prompt. The same model extracts and acts; nothing is trained (HF blog).

The evaluation is on τ²-bench, Sierra's dual-control customer-service simulator, in its retail and airline domains. A task pairs the agent with an LLM-simulated customer, a set of tools, and a written policy document; success is a strict binary reward on the final database state, and each task runs 4 trials. The only change IBM makes to the stock ReAct scaffold is the guideline block appended after the domain policy — agent is gpt-4.1, customer simulator is gpt-5.5 (HF blog).

Each guideline carries a support count — the number of distinct training tasks it was extracted from — a dial for tuning the dose of memory. They compare a baseline (no guidelines), sup3 (rules with support ≥ 3: 31 in retail, 14 in airline), and lossless (every rule that survives consolidation: 292 in retail, 101 in airline). Two metrics: Mean Accuracy across the 4 trials (capability) and Pass^4, the share of tasks solved on all four attempts (reliability) (HF blog).

Held-out results

On the held-out test split, guidelines lift both metrics in both domains, and the right dose differs by domain:

Held-out τ²-bench results for the GPT-4.1 agent: the lossless guideline set lifts retail mean accuracy from 78.1% to 82.5% and Pass^4 from 57.5% to 67.5%; in airline, the compact sup3 set gives the best Pass^4 at 45%.

Retail (40 test tasks): baseline 78.1% accuracy / 57.5% Pass^4 → sup3 78.8% / 60.0% → lossless 82.5% / 67.5% (+4.4 and +10.0 points). Airline (20 test tasks): baseline 57.5% / 35.0% → sup3 60.0% / 45.0% → lossless 65.0% / 40.0%. The domains disagree on dosage: retail needs the full lossless set for the real jump, while in airline the 14-rule sup3 core is the more reliable configuration, beating lossless on Pass^4 (45% vs 40%) even though lossless wins on accuracy. Support count is a dial to tune per domain, not a constant (HF blog).

The improvement is not model-gap patching: rerunning airline with gpt-5.5 as both agent and extractor, on a higher floor, still gains +15 points Pass^4 (55.0% → 70.0%) and +10 points accuracy (72.5% → 82.5%) on the same held-out tasks. IBM notes none of the gpt-5.5-derived rules reached support ≥ 3 (the compact core is sup2, 103 rules; lossless is 146), and that the lossless run reproduced exactly across reruns while the baseline wobbled (HF blog).

Better answers cost tokens, not turns

The price of memory lands on the input side, not in longer conversations. IBM reports total LLM cost per conversation (agent + simulator) rising in retail from $0.084 to $0.139 with the 292-rule lossless set (+65%) and in airline from $0.081 to $0.107 (+32%). On matched runs solved under both baseline and guidelines, conversations get shorter, not longer — retail lossless trims 10.59 → 10.16 assistant turns, airline 9.15 → 8.82, and no variant lengthens them. Fewer turns means lower end-to-end latency, and since the guideline block is byte-identical on every call, most of the input cost is reclaimable with prompt caching (HF blog).

Total LLM cost per conversation on the full base split: retail rises from $0.084 to $0.139 with the 292-guideline lossless set (+65%), airline from $0.081 to $0.107 (+32%).

The leaderboard claim — and what is actually merged

The headline rests on IBM's own snapshot. Like all τ²-bench submissions, IBM's entries are scored on the full base split (train ∪ test) at Pass^4, which is why base-split and held-out numbers differ (HF blog). Against its October 12, 2026 snapshot, IBM reports retail #1 at 67.5%, 6.1 points ahead of RAFT-30B-A3B at 61.4% — a Qwen3-30B-A3B-Thinking model fine-tuned on synthetic retail-domain data by Northeastern University's HAI Lab, and the comparison number is independently verifiable from the benchmark repo. On airline it reports the gpt-5.5 agent first at 82.0%, ahead of frontier entries it lists in the 66–72% range, with the gpt-4.1 sup3 variant eighth at 44% (Unite.AI, dataist.ai).

τ²-bench leaderboard comparison from the IBM post: the ALTK-Evolve retail run at 67.5% Pass^4, and airline runs at 82.0% and 44.0%.

What is not merged: both submissions are still open pull requests on the τ²-bench repo — retail, #542, opened September 14, reporting Pass^1 81.14% through Pass^4 67.54% across 114 base-split tasks at $0.1084 per conversation, and airline, #551, opened September 21, reporting Pass^4 82.00% across 50 tasks. Both declare modified_prompts: true and state they expect to be listed as Unverified; neither appears in the current leaderboard manifest, and the live taubench.com board does not list them. The retail PR also discloses that the guidelines were learned from a train split that the base evaluation split includes — expected under the benchmark's rules, but it means the base-split "#1" measures partially in-distribution memory.

Why it matters

For anyone building agents, the interesting part is what the memory costs at inference time: no fine-tuning, no RL, no vector-store retrieval in this evaluation — an offline distillation pass amortized once per policy or task set, then a static, inspectable, editable block riding the system prompt. That is a materially different failure mode from weight updates: when the policy changes, you diff the guidelines instead of re-running training, and the agent's behavior is auditable from the prompt alone. The gpt-5.5 rerun is the important control — it shows the ceiling isn't "patching a weak model's gaps," and that reliability gains survive a stronger base.

Two engineering takeaways. First, measure reliability, not just average accuracy: IBM's Pass^4 framing is the right instinct, because a support agent that resolves a refund 3 of 4 times fails a quarter of real customers even at 78% mean accuracy — and the dosage knob (support threshold) is where the tuning value lives. Second, leaderboard claims need a status check before you plan around them: IBM's own numbers are precise and its comparison target is real, but "tops the leaderboard" currently means "top of our own snapshot, not the public board," with the entries flagged Unverified and unmerged — the opposite of toothless, but not yet independently scored.

Caveats

All performance numbers are vendor-reported. IBM's held-out results are its own runs on the stock scaffold; no independent replication surfaced. The one claim verified outside IBM is the comparison target: RAFT-30B-A3B's 61.4% comes from the benchmark repo's submission file.
The leaderboard entries are not live. Both PRs remain open, self-flag Unverified (modified_prompts: true), and are absent from the current manifest — the "#1" is against IBM's own October 12 snapshot. And don't compare these 1.x numbers to taubench.com's headline scores: the current board tab is 2.0, which the repo says is not comparable to 1.x (tau2-bench).
Small held-out sets. 40 retail and 20 airline test tasks; a ±10-point Pass^4 swing is a handful of flipped tasks, and the airline verdict rests on 20.
Scope. Two of τ²-bench's domains (retail and airline), on the v1.0.1 English text leaderboard — no telecom, and nothing tested transfers to the 2.0 multilingual or voice races, which the repo says are not comparable to 1.x. The extractor cost (offline LLM calls over training trajectories) is not reported anywhere, and it is the one cost that does not amortize to per-conversation.
Two different cost frames. The blog's per-conversation totals (agent + gpt-5.5 simulator) are $0.139 (retail) / $0.107 (airline) for the GPT-4.1 lossless variants; the PRs list $0.1084 (retail, gpt-4.1) and $0.1690 (airline, gpt-5.5). The airline pair is simply different models, but the retail pair looks like the same configuration at a per-conversation cost that differs by ~28% between the blog and the PR ($0.139 vs $0.1084) — neither source explains it, so treat both numbers as reported and don't blend them.