A minimal coding agent matched four purpose-built MLE harnesses at equal budget and backbone
A controlled ablation from EPFL and Apple finds the coding-agent environment is the only intervention that moves the needle on MLE-bench and NatureBench — search trees, orchestrators and memory hierarchies add nothing statistically significant.
Researchers at EPFL and Apple report that, under an equal time budget and the same frontier LLM backbone, four open-source machine-learning-engineering (MLE) harnesses provide no advantage over a single session of a minimal coding agent — on MLE-bench and on NatureBench. In a controlled ablation (arXiv:2609.40303), they argue the backbone, not the harness machinery, is the primary driver of performance.
What the paper found
Submitted 30 September 2026 by Kirill Brilliantov, Alejandro Hernández-Cano and Emmanuel Abbé (EPFL and Apple; one author contributed while interning at Apple), the paper asks a question the MLE-agent literature has been poorly placed to answer: how much of the elaborate machinery — search trees, populations of programs, memory hierarchies, teams of specialized agents — earns its keep once backbone, hardware and time budget are held fixed?
The baseline is Malena, "the machine learning engineer agent baseline": a single, well-prompted session of the OpenCode v1.15.6 coding agent with a minimal toolset — read, write and bash primitives for direct access to the execution environment, and little else. The paper compares it with four released open-source MLE harnesses — MLEvolve, AiScientist, Arbor and ScienceFlow — run as released on a fixed 30-task subset of MLE-bench (which the authors describe as "harder and broader than Lite") and on NatureBench: 90 tasks distilled from Nature-family publications, scored against each source paper's reported state of the art. Backbones, hardware (1×A100 80GB, 12 CPU cores, 144GB RAM, models served locally via vLLM) and the 24-hour budget were matched.
The headline result: with the GLM 5.2 backbone, the authors report Malena's self-selected any-medal rate at 62.5%, versus 47.1% for the strongest external harness (AiScientist). Malena matched or outperformed all four harnesses at every frontier backbone tested. The one exception was the weaker Gemma 4 31B, where MLEvolve led on mean percentile, though medal-rate confidence intervals still overlapped.
The ablation ladder is where the paper earns its conclusion. The authors re-implemented the intervention families common to MLE harnesses — execution environment, search, autonomy, multi-agent orchestration — in a single codebase so that only one thing varies at a time. Giving the model a coding-agent environment was by far the largest effect: on GLM 5.2 and Kimi K3, moving from a chat interface (write a script, get an error report) to a coding agent (inspect files, execute code, debug) improved results markedly, with Gemma 4 also benefiting. Beyond that, no further intervention yielded a statistically significant gain.
A trace analysis suggests why: the agent carries out the search these harnesses hard-code on its own, balancing rare techniques against performance-oriented refinement while reusing the artifacts it has already produced. The authors frame the result as an instance of the bitter lesson: "coding-agent post-training substitutes for the scaffolding that harnesses hand-design."
Why it matters
For anyone building MLE agents, this is a budget-allocation argument. A single 24-hour evaluation of three seeds over MLE-bench's 75 tasks consumes thousands of GPU hours — the authors note this is why faithful re-runs of existing baselines are rare and why published harness gains are usually confounded with backbone and hardware changes. By their account, this is one of the few controlled comparisons that holds those fixed, and its answer is blunt: the machinery layers become redundant in the coding-agent setting.
The practical reading: if you are building an MLE agent on a frontier model, spend your effort on the backbone and on giving it direct access to the environment — not on hand-crafted search topology, multi-agent coordination or bespoke memory management. The paper's own caveat is that weaker backbones still benefit from hand-crafted workflow priors; as agentic capability grows, a plain coding-agent loop closes the gap.
One methodological detail matters for reading the numbers: the paper reports self-selected performance — the agent picks its best submission by validation score, not by peeking at hidden test scores. Self-selection is the honest default, and it is the metric on which the harnesses lose.
Caveats
- These are the authors' own reported results on a preprint; we are not aware of any independent replication.
- The comparison covers current MLE-bench and NatureBench settings — 30 of MLE-bench's 75 tasks — not all of agentic ML engineering.
- The authors themselves note limits in statistical power and describe their multi-worker coordination study as narrow.
- Benchmark outcomes are not evidence that Malena uses fewer attempts, less wall-clock time or less total compute, and the Chat-versus-Oneshot comparison does not isolate coding-agent post-training from the tool-enabled environment as a whole.
- The Gemma 4 31B result shows the conclusion does not extend to weaker backbones.
- No code release for Malena appears alongside the paper in the sources we checked.