Richland Technology
Richland Technology
← Back to Blog
llmagentsevaluationresearchbenchmark

DAYJOB: best agent passes 24.7% of long-horizon professional tasks — median config passes 0.6%

Surge AI's open benchmark of 130 healthcare and finance tasks finds agents fail on judgment, not capability

Richland AI Desk·2 October 2026·5 min read
DAYJOB: best agent passes 24.7% of long-horizon professional tasks — median config passes 0.6%

Surge AI released DAYJOB, a benchmark of 130 long-horizon professional tasks in healthcare and finance. Across 30 model configurations from 13 developers, the strongest — Claude Opus 5.5 — passed 24.7% of healthcare and 23.9% of finance attempts; the median configuration passed 0.6% and 2.5%. The paper, posted to arXiv on October 1, also releases the evaluation harness, 100 of the 130 tasks, and a live leaderboard.

What shipped

DAYJOB rests on a specific theory of professional work: a brief request that assumes shared context, a workspace of files the request implicitly refers to, and a deliverable a competent colleague could grade. Working professionals built the 50 healthcare and 80 finance tasks, estimated to take a human 13.6 hours (healthcare) and 16.6 hours (finance) on average. Each runs in a containerized Harbor environment with network access limited to the model provider's API; the agent gets only general tools: a terminal, a file editor, and a task tracker.

The workspaces are the point. Input files are mostly PDFs (80% in healthcare, 70% in finance), plus spreadsheets and Word documents, and relevant and irrelevant material sit side by side — in one urgent-care task, seven of nine PDFs are administrative forms. Sources are stale or conflicting by design: a 2019 site evaluation sits inside a 2026 expansion proposal, and a bank memo's 3.25% borrowing rate is superseded by a CFO chat screenshot at 5.25%.

Grading is rubric-based. Each task has an expert rubric of binary criteria — median 47.5 per healthcare task, 57.5 per finance task — and an attempt passes only if it meets every criterion. Prohibitions ("Does not...") make up 16% of healthcare criteria and 8% of finance criteria, and all 50 healthcare rubrics and 43 of the 50 public finance rubrics contain at least one. After the agent finishes, an agentic judge (Claude Opus 4.8 running OpenCode) works in the same container: it opens the delivered files, recomputes values where needed, and returns a verdict with evidence for every criterion.

Two examples show what "professional work" means here. In the Caldervane Emerging Markets fund task, the agent gets 13 files and a request to check whether anything should stop the first client reports going out that day. A Bloomberg screen quotes the Nerovexa price in South African cents; the typed price table says ZAR; the price was entered as rand — so a position worth about $260,000 appears to be worth $26 million. The 20 criteria require finding and quantifying the error, withholding the reports, and recognizing the manual keying as a control weakness. In the urgent-care task, the chart records right-sided neck pain after moving heavy furniture and slightly slurred speech, and the rubric requires ruling out arterial dissection before any discharge — even though the request frames the case as "textbook Bell's Palsy."

Results table from Surge AI's DAYJOB launch post

Across 30 configurations from 13 developers (five attempts per task), Claude Opus 5.5 leads both domains at 24.7% and 23.9%. GPT-6 Astra is next at 11.6% and 21.5%. The median configuration passes 0.6% of healthcare and 2.5% of finance attempts, and 19 of the 30 configurations score below 5% in both domains. The arXiv release includes per-attempt cost and token data, which is rare: Opus 5.5 spent about $7.10 and 9.85M tokens per healthcare attempt, and $11.47 and 17.75M tokens per finance attempt.

Surge releases all 50 healthcare tasks, 50 of the 80 finance tasks, the harness (Apache 2.0), and the leaderboard; 30 finance tasks are withheld. An earlier version was accepted to the AABA4ET workshop at NeurIPS 2026.

Why it matters

The headline number is less interesting than the failure mode. In case studies, agents accept premises the record contradicts and carry wrong inputs through otherwise consistent analyses. On the Bell's Palsy task, Surge reports that GPT-5.6 recognized in its reasoning that facial weakness plus neck pain could have a vascular cause, then decided "I shouldn't question the diagnosis unless necessary." A Grok run on the same task recommended holding discharge for emergency evaluation.

On the Caldervane task, only 2 of 66 runs across 22 configurations caught the cents-versus-rand error — both Claude Opus 5 runs. Surge argues these are judgment failures, not capability failures: the models can find the evidence; they decline to act on it.

For engineers, three things are actionable. First, the harness is open and reproducible — Harbor containers, five attempts per task, a pinned judge — so teams can run their own models against the same rubrics rather than trusting a vendor leaderboard. Second, the cost data is a concrete datapoint for budgeting agentic workloads: near-frontier performance on a single long-horizon task (roughly 13–17 hours of professional time) runs about $7–11 in tokens. Third, the spread between the frontier and the median (24.7% vs 0.6% on healthcare) quantifies how much of the field is not close to professional-grade work — useful calibration for anyone scoping what agents can take over.

Caveats

Everything above is vendor-reported. Surge AI built the tasks, wrote the rubrics, ran the models, and graded the results; independent coverage notes the results are not an independent measurement. The judge is a single model — Claude Opus 4.8 — with no published human-agreement study for DAYJOB specifically. Pass rates average five attempts per task, with errored trials excluded. The human-time estimates are Surge's own.

The paper says all healthcare tasks are released, but the public Hugging Face repos currently hold three sample tasks each, with full access gated behind contacting the team. The paper's critique of GDPval (prompts averaging 337.2 words, at most one reference file in 75% of tasks) is Surge's interpretation, not an established finding.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.