AutoCompact: coding agents that learn when to compact their own context — +9.2 points on SWE-bench Verified
Training compaction timing, summary content, and post-compaction behavior as policy decisions — gains hold from $0.10 to $4.00 per task.
AutoCompact, a paper from Singapore Management University, NTU, and Harvard, trains a coding agent to decide when to compact its own context as part of its policy. It reports +9.2 percentage points on SWE-bench Verified (39.6% vs 30.4%) and +5.0 on SWE-PolyBench Verified (24.5% vs 19.5%), with gains holding at every inference budget from $0.10 to $4.00 per task (arXiv, alphaXiv).
What shipped
Long-horizon coding agents accumulate stale exploration — dead hypotheses, ruled-out paths, verbose tool dumps — and the industry default compacts only when the context window fills. AutoCompact ties compaction to task progress instead: compact when a stage is resolved (bug localized, edit planned), not when a threshold trips mid-stage while evidence is still needed (project page).
The mechanism is a compact() action added to a terminal-REPL agent scaffold. When the model invokes it, earlier interaction history is replaced with a model-written # Auto Context Summary that preserves the working state — conclusions reached, files changed, what to verify next — while the original task and recent turns stay in context (project page). The policy learns three decisions: when to compact, what to preserve, and how to continue from the summary.
The training pipeline is the novel part. Prompting alone does not induce proactive compaction — the base model rarely invokes compact() before the limit even when compaction rules are written into the prompt (arXiv HTML). The authors collect on-policy trajectories with an online judge: GPT-5.5-Codex reviews each proposed action against the current history and, when flawed, replaces it before the environment executes it, so every subsequent step runs from the corrected decision (project page, PREreview).
379 SWE-rebench tasks yielded 1,052 judge-corrected trajectories for SFT, split 24% trigger corrections, 53% summary repairs, 23% continuation fixes (project page). GRPO on SWE-Gym follows, with binary task-success as the only reward — no KL penalty, no entropy bonus, no compaction-specific reward shaping (PREreview). The judge is removed at inference.
On the same Qwen3-Coder-30B-A3B-Instruct base and scaffold, AutoCompact reaches 39.6% on SWE-bench Verified (base: 30.4%) and 24.5% on SWE-PolyBench Verified (base: 19.5%), averaged over three runs (arXiv, project page, alphaXiv). The trained policy invokes compact() on 44.3% of tasks after SFT and 58.5% after RL (PREreview, aicoder), and summary quality improves: key state omitted drops from 3.1% to 0.2%, missing next-action from 8.2% to 2.2% (PREreview).
Pass rate is reported against per-task inference budgets ($0.10–$4.00, priced with Alibaba Cloud Model Studio rates, cached tokens at 20% of input) (PREreview):
- 256K context, no forced compaction. No trajectory reaches the threshold, so length-triggered compaction never fires — yet AutoCompact-SFT beats the full-history baseline at every budget, ending 32.2% vs 30.4% at $4.00 (project page, alphaXiv). The paper's "long context is not enough" claim rests on this.
- 16K context with shared length-triggered fallback. AutoCompact ends at 38.0% vs 28.8% for the base (alphaXiv).
- Same checkpoint, compaction executed vs skipped. Holding the learned parameters fixed, executing
compact()is worth 19.9 percentage points at $0.10/task and 1.9 points at $4.00; a "summary ignored" variant scores 39.6% vs 37.7% (alphaXiv, PREreview). This is the paper's cleanest evidence that the gains come from the act of compacting, not just from training.
Why it matters
The takeaway for anyone building long-horizon coding agents: context management is a policy decision, not a capacity problem. The 256K result is the strongest version of this — proactive compaction helps even when the window never fills, because stale exploration degrades reasoning long before overflow (project page).
The ablation matters for cost-constrained deployments: the trained policy's value concentrates at low budgets, exactly where token spend is the binding constraint. And the paper's specific contribution — supervising what the agent does after compaction — is what distinguishes it from prior work: rubric-based proactive compaction provides no training signal, and offline-insertion fine-tuning (SWE-Compressor) leaves the original post-compaction actions untouched. AutoCompact-SFT beats SWE-Compressor by 1.2 points on SWE-bench Verified and 1.6 on SWE-PolyBench Verified with the same base model, scaffold, and comparable training data (arXiv HTML, PREreview, PREreview).
Caveats
- No code or weights yet. The GitHub repo is a placeholder for an upcoming release of the 1,052 SFT examples — this is a method paper, not a drop-in (GitHub, aicoder).
- Self-reported, thin statistics. All results are the authors' own, averaged over three runs with no variance reported — relevant when several baseline gaps are 1–2 points (PREreview). The summary-quality metrics (3.1%→0.2%, 8.2%→2.2%) likewise rest on keyword screening plus random manual spot checks, with the screening keywords and sample size unspecified (PREreview).
- The judge is load-bearing but unvalidated. No agreement rates, no human audit, no correction rate. SFT distills GPT-5.5-Codex's compaction preferences to the extent the judge is idiosyncratic (PREreview).
- RL credit assignment is unresolved. The binary reward is shared across coding and compaction, and there is no RL-without-
compact()control — the 7.4-point RL-over-SFT gain could be partly general coding improvement, since RL on SWE-Gym is independently known to improve coding (PREreview). - The main table confounds trigger type with context regime. Length-triggered baselines run at a 16K threshold while everything else runs at 256K; the fair 16K-vs-16K comparison lives only in the budget subplot (PREreview).
- 256K is the less realistic regime. Production harnesses (Codex, Claude Code) compact at the window limit — the 16K numbers are closer to deployment reality, and the paper leads with 256K (PREreview).
- Compaction's own cost is unmeasured. No output-token cost per
compact()call, no net token delta per solved task, and the judge-inference training bill is unreported (PREreview). - Generalization untested. All training and evaluation is SWE-bench-style bug fixing; no feature development, data analysis, or tool-heavy research tasks. RL also trains on 32K-token sequences while evaluation runs at 256K (PREreview).