GLM-5.3 tops CivBench's controlled Civilization V run at 1634 Elo — and a 27B open model holds its own at a fifth of the cost
813 games on three fixed starts, Bradley-Terry Elo from win-probability estimators, and a live argument over whether the models are really playing.
The Vox Deorum team, led by John Chen at the University of Arizona, published a controlled run of CivBench, its Civilization V benchmark for long-horizon agent decision-making: 19 models, 30 setups, and 813 games across three fixed map starts. GLM-5.3, deciding every five turns, leads at 1634 Elo; Opus-5.5 is second at 1594; and Qwen-3.8-27B, a 27B open-weight model, rates 1518 — above the built-in AI's 1500 baseline and within 80 Elo of Opus-5.5 at roughly a fifth of the API cost.
What shipped
CivBench tests whether an LLM can plan across the hundreds of turns of a Civilization V game, where a decision's payoff may not show up for 50–100 turns. The new controlled version rotates every strategist through the same three fixed 8-player maps. Each game contains two copies of the tested LLM plus six Vox Populi AI players (mod version 5.2.7); completed setups run 24 games (48 player-seats); games run on Prince difficulty with no handicap, per the team. Two decision conditions are compared: deciding every turn or every five turns (plus whenever an important event comes up). Every game is replayable in the replay viewer.
The LLM does not move units or queue city production. Each turn it receives a structured Markdown report of the game state and sets high-level strategy — victory target, tech research, social policy, diplomacy — steering the built-in AI's tactical execution through 34 "flavor" settings (0–100) and 26 diplomatic persona values (0–10), per the COLM 2026 paper.
Ratings are Bradley-Terry Elo computed from "model-adjusted strength" — a pre-trained win-probability estimator applied in-sample to the games — rather than raw win/loss, with the built-in AI anchored at 1500:
| # | Player | Deciding | Elo |
|---|---|---|---|
| 1 | GLM-5.3 | every 5 turns | 1634 |
| 2 | Opus-5.5 | every 5 turns | 1594 |
| 3 | Kimi-K2.7 | every turn | 1575 |
| 4 | GLM-5.2 | every turn | 1537 |
| 5 | Kimi-K2.7 | every 5 turns | 1518 |
| 6 | Qwen-3.8-27B | every turn | 1518 |
| 7 | GPT-6-Luna | every 5 turns | 1510 |
| 8 | Kimi-K2.6 | every turn | 1504 |
| 9 | Built-in AI | — | 1500 |
| 10 | DeepSeek-V4.1-Flash | every 5 turns | 1496 |
Other headline numbers from the report: GLM-5.3 beats the built-in AI 66.7% of the time head-to-head across 288 player comparisons; Opus-5.5 won 27% of its games (an even chance in an 8-player game is 12.5%); and the team's best predictor picks the eventual winner 85% of the time well before the game ends.
Per LLM player per game, from the usage table: Opus-5.5 (per-5) $32.22, GLM-5.3 (per-5) $20.55, Qwen-3.8-27B (every turn) $6.80, Qwen-3.8-27B (per-5) $2.72, GPT-6-Luna (per-5) $0.61. Fitting Elo against cost per game, GLM-5.3-Per-5 is the most cost-efficient setup (+99 Elo vs the fitted curve) and Qwen-3.6-27B-Per-5 the least (−116) — though the fit is loose (R² = 0.40).
Everything is open: the Vox Deorum bridge (v0.13.6 beta) that connects an LLM to the game, the civ-bench harness (MIT), the replay viewer, and the full report with CSVs. The methodology paper, "CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V", was accepted at COLM 2026. Sonnet-5.5, GPT-6-Astra, GPT-6.1-Sol, and Qwen-3.8-Flash-Next are mid-run.
Why it matters
The transferable idea is progress-based evaluation. Terminal win/loss is too sparse a signal in a 411-turn, 8-player game, so the COLM paper trains state-value estimators — the probability a player eventually wins given the current state — and validates them. The paper's AttentionMLP estimator reaches ROC-AUC 0.865 per the team's docs; the controlled run's own prediction page reports 0.8545. The Elo table is built from pairwise comparisons of that adjusted strength, which is why a model can rate high without winning many games outright. For anyone evaluating long-horizon agents, this is a concrete recipe: train a win-probability model, adjust for game state, then rank — and publish the replays so results are checkable.
The deciding-frequency result is subtler than the headline. The two top-rated setups both decided every five turns, and per-5 cuts API cost by roughly 60% (Qwen-3.8-27B: $6.80 → $2.72). But Kimi-K2.7 rated higher deciding every turn (1575) than per-5 (1518), so the optimal frequency interacts with the model — treat it as a hyperparameter, not a rule.
The cost-efficiency framing is something most agent benchmarks omit: Elo per dollar, with a fitted curve and outliers named. A 27B open model at $6.80/game rating above the built-in AI and within 80 Elo of a $32/game frontier model is a useful data point for anyone choosing between hosted frontier APIs and self-hosted open weights.
Caveats
Everything above is self-reported by the team; there is no independent replication, and the Elo ratings use in-sample predictions from the strength estimator.
The biggest open question is whether the task is "playing Civilization V" at all. The top comment on the announcement thread argues the LLMs never choose city production or move units — they tune parameters that steer the built-in AI, closer to writing an AI mod than playing the game. The author's reply: tech, policy, and diplomacy commands are sent directly, and full low-level control would be prohibitively expensive. Treat "playing" as shorthand for "setting high-level strategy."
Also unverified: sampling settings (temperature, etc.) are not published in the report; and the companion paper on nuclear authorization — which reports that no prompt intervention reliably eliminated escalated nuclear authorization in replayed high-tension games — is listed on arXiv as accepted to the Findings of EMNLP 2026.