AuXSpec makes program specs executable so test cases become the RL reward — a 9B model jumps from 0.7% to 82.7% SpecGen pass@4
ETH's Lean framework pairs declarative postconditions with proved-equivalent executable checkers, so test suites become a cheap, trustworthy RL signal for spec generation.

AuXSpec turns a program spec into an executable checker, so test cases become the RL reward — and a 9B model jumps from 0.7% to 82.7% SpecGen pass@4.
ETH Zurich and HKUST researchers released AuXSpec (https://arxiv.org/abs/2611.21421v1), a Lean 4 framework that pairs each declarative postcondition with a proved-equivalent executable checker, so test cases evaluate a specification by direct execution. That agreement becomes the RL reward for training spec-generation models. Trained on it, the 9.4B-parameter open-weight model hits 82.7% SpecGen pass@4 on the held-out AuXSpec-Test benchmark (up from 0.7%), and the 27.4B model beats all seven frontier models tested on SpecGen and CodeGen there, per the authors' Table 1. Weights and a 3,037-problem dataset are released under Apache-2.0.
What shipped
AuXSpec is a framework, a training recipe, two trained models, and a dataset.
The core idea is to make a specification runnable. In Lean, a declarative postcondition is a Prop: a function (input : Input) → (result : Output) → Prop stating what a correct output is. Because it can quantify over unbounded domains (e.g. ∀ j : Nat), it is provable but not executable. AuXSpec pairs it with an executable counterpart, a Bool-valued checker that decides the same property. Where the spec is built from decidable connectives and bounded quantifiers, Lean's type-class resolution synthesizes the decider automatically via decide. Where decidability can't be synthesized — an unbounded ∀ or ∃ — the authors hand-write a checker that searches a finite range justified by the input constraints, such as scanning a removal index over List.range (input.length + 1).
The two are connected by an equivalence theorem, check input result ↔ spec input result, proved in a separate step after the definitions are accepted. Crucially, it carries a validity hypothesis, so the two postconditions agree on valid inputs only. That theorem lets any number of test cases be decided by execution without proving each one.

Test cases are #guard commands that fail compilation when false, layered so the reward is progressive: the file compiles, guards reproduce the given test cases, and an LLM judge passes quality checks. The RL algorithm extends DAPO with batch-level advantage normalization, a KL penalty toward the SFT model, and a loss mask on repetitive rollouts.
The released numbers, all self-reported by the authors:
- SpecGen pass@4 on AuXSpec-Test: Qwen3.5-9B base 0.7% → 82.7%; Qwen3.8-27B base 14.3% → 84.3%. The 27B model outperforms all seven frontier models tested (Claude-Opus-5, Claude-Sonnet-5, GPT-5.6-Sol, GPT-5.6-Luna, Kimi-K3, DeepSeek-V4-Pro, DeepSeek-V4-Flash) on SpecGen and CodeGen there.
- Verina (out-of-domain, original prompts): both models reach 90.5% SpecGen pass@4, from 7.0% and 21.9%.
- vs. prior open-source SOTA VeriSpecGen-30B-A3B on Verina: pass@1 47.6% vs. 73.7% (9B) and 84.5% (27B).
The models are eth-algopt/AuXSpec-9B-RL (9.4B params, base Qwen3.5-9B) and eth-algopt/AuXSpec-27B-RL (27.4B params, base Qwen3.8-27B), both Apache-2.0, on HuggingFace. The dataset is 3,037 LeetCode problems — 2,737 train plus a 300-problem AuXSpec-Test split (72 easy / 155 medium / 73 hard), each with nine test cases (three valid, three invalid, three wrong-output) with rationale for the invalid and wrong cases.
Why it matters
The interesting claim isn't the frontier-beating number — it's the mechanism. A 9B model gains roughly 58 points of SpecGen pass@4 from RL on top of SFT, using a reward that is nothing but "does a proved-equivalent executable checker pass the test cases." That is a cheap, automatic, and — because of the formal discipline around it — trustworthy signal.
The inversion is the point. Prior autoformalization approaches get feedback from explicit proofs or from separately implemented code with formal correspondence proofs, both expensive. AuXSpec instead makes the spec itself executable, so the test suite becomes the reward. The equivalence theorem keeps that honest: it's proved in a separate step, on valid inputs only, and bounds the checker's finite search. The reward is only as good as the theorem.
For anyone building verification tooling, this is a directly runnable recipe: released weights, released data, and a mechanism that doesn't require a theorem prover to grade every candidate spec.
Caveats
This is a 62-page preprint with no peer review or independent reproduction found. All benchmark scores, the frontier-model comparisons, and the VeriSpecGen comparison are the authors' own.
The cross-paper comparison is apples-to-oranges: AuXSpec's 90.5% is pass@4 (4 samples), while VeriSpecGen's 63.9% was measured at pass@10 (10 samples), and those VeriSpecGen figures are AuXSpec's own re-evaluation, not that paper's headline result (its headline is 86.6% on Verina SpecGen with Claude Opus 4.5). Note also that "pass@1" here is the empirical mean over 4 samples, not the standard unbiased pass@k estimator.
"Outperforms all frontier models" is scoped to SpecGen and CodeGen on AuXSpec-Test. On the two repair tasks, Claude-Opus-5 remains best per the authors' own Table 1.
The equivalence theorem only relates the two postconditions to each other; whether the postcondition faithfully captures the natural-language problem rests on test cases, LLM judges, and — for the test split — a hand check. Correctness theorems cover only the 300 test problems, and 2 of those 300 equivalence theorems are left as sorry (Float-to-Rat cases). The paper says the SFT set was expanded via expert iteration on Codeforces, while the model cards say LeetCode and TACO — unresolved.
Not measured: whether RL-trained specs are actually usable downstream (e.g. letting Lean verify real implementations at scale beyond the 300 reviewed problems), performance on problem types beyond LeetCode/Codeforces-style tasks, and the inference cost of the 4-sample protocol.
Sources
- AuXSpec paper (arXiv): https://arxiv.org/abs/2611.21421v1
- AuXSpec project website: https://auxspec.org
- AuXSpec models and data (HuggingFace): https://huggingface.co/collections/eth-algopt/auxspec
- AuXSpec-9B-RL model card: https://huggingface.co/eth-algopt/AuXSpec-9B-RL/raw/main/README.md
- AuXSpec-27B-RL model card: https://huggingface.co/eth-algopt/AuXSpec-27B-RL/raw/main/README.md
- AuXSpec dataset card: https://huggingface.co/datasets/eth-algopt/AuXSpec/raw/main/README.md
- VeriSpecGen prior work (arXiv): https://arxiv.org/abs/2604.10392v1