KaliBench: open-weight models cap at 42% on exact Kali commands — an 8B trained on its rewards reaches 685B-MoE-level scores

KaliBench, a new NeurIPS 2026 benchmark, translates natural-language security requests into exact, executable Kali Linux CLI commands. It reports that no open-weight model exceeds 42% exact-command accuracy without tool hints, and that an 8B model trained on runtime-free verifiable rewards reaches performance comparable to a 685B-parameter MoE.

What shipped

KaliBench (arXiv:2610.02206, accepted at the NeurIPS 2026 Evaluations and Datasets Track, from Khalifa University's RISys Lab) is a dataset, evaluation harness, and training framework for natural-language-to-CLI translation on Kali Linux. It targets a gap between two existing eval styles: knowledge quizzes (does the model know what nmap does?) and end-to-end agentic tasks (can a whole system complete a mission?). Neither isolates the skill of turning an analyst's intent into a command a real tool will accept and run.

The dataset is 8,504 verified query–command pairs spanning 1,642 tools across 23 capability dimensions and five security phases — 3,504 training and 5,000 held-out test samples after deduplication, plus 2,809 tool-documentation entries (abstract, repo reference). Construction is "manuscript-grounded": tool names, flag–value pairs, and documented aliases are extracted from official Kali Linux documentation to guide candidate generation, then filtered through a multi-stage verification pipeline combining LLM-based validation, sandboxed terminal execution in a network-disabled Docker container, and human-in-the-loop refinement (data-construction docs).

Evaluation runs in three modes that vary how much tool information the model gets (README):

• unrestricted — the query only; the model must select the tool itself
• restricted — the query plus a candidate set of 20 allowed tools
• hinted — the query, the target tool, and its usage documentation

Scoring is deterministic rather than judge-based: commands are canonicalized (flag reordering and documented aliases normalize to the same form, while positional arguments must stay in order) and compared against reference labels. Total Score is the mean of tool accuracy, optional-argument F1, and positional-argument F1 (alphaXiv summary).

The authors release the dataset and models on Hugging Face, the code on GitHub, and three trained variants — RedSage-K-SFT, RedSage-K-GRPO, and RedSage-K-SFT-GRPO — all starting from the 8B RedSage-Qwen3-8B-Ins cybersecurity-tuned base, with reproducible SFT and GRPO/RLVR training pipelines.

Why it matters

The headline result is a ceiling: across 24 configurations of general-purpose and security-focused open-weight models, none exceeds 42% exact-command accuracy in the unrestricted setting (abstract). The strongest score belongs to GLM-5.2 (753B) at 41.3% (project page, leaderboard data).

The interesting part is where the authors find models fail. Their diagnosis: the bottleneck is not tool selection but argument construction. Restricting the candidate tool set lifts average tool accuracy from 72.0% to 95.2%, but exact correctness only moves from 22.3% to 28.3%; giving the tool plus its documentation raises exact correctness to 73.1% and optional-argument F1 to 87.8% (alphaXiv summary). In the authors' reading, models largely know which tool to call but cannot type the flags, bindings, and ordering that make a CLI command run. Where a wrong flag can mean scanning the wrong subnet, that distinction is the whole problem.

The second result is the mechanism that makes the problem cheap to train against. Because ground-truth commands were verified at construction time, KaliBench's rewards are "runtime-free": the reward is computed by canonicalized scoring against reference labels, with no command execution in the training loop (training docs). The GRPO reward weights are exact output format 2.0, approximate format 1.0, tool score 1.0, optional-argument F1 1.5, positional-argument F1 1.5, exact match 2.0. Skipping execution makes RL loops faster and, for security tooling, safer — you never run a model-generated exploit command to find out if it was right.

That combination produces the paper's central claim: supervised fine-tuning plus GRPO on KaliBench rewards lifts the 8B RedSage-K model to a mean Total Score of 79.2 across the three modes, versus 80.2 for DeepSeek-V3.2 (685B total / 37B active) — and the 8B model beats the giant in the hardest mode, unrestricted, 70.3 vs 67.2 (leaderboard data). That is a 7.5-point gain over the untrained base's 71.7 (paper).

For anyone building natural-language-to-tool systems, the practical read is direct. First, don't ship free-form autonomous command generation for security tooling yet — the 42% ceiling argues for a human confirmation step or parameterized templates. Second, if you have a narrow, high-value command surface, the 8B result is a playbook: build a small verified dataset for your own tools, canonicalize equivalent commands, and run RLVR against it — the training loop needs no execution.

Caveats

• All scores are the authors' own, reported in the paper and its leaderboard; nothing here has been independently reproduced.
• The evaluation covers open-weight models only. No closed frontier model (Claude, GPT-class) scores are reported, so there is no public read on how the ceiling compares to what a hosted model would score.
• The 8B-matches-685B result is specific to this benchmark's distribution. A model trained on KaliBench rewards gets good at KaliBench-style commands; it says nothing about planning, judgment, or generalization to tools and phrasings outside the training set (Ken Ashe's write-up).
• The gain is concentrated in the hard modes: in hinted mode the trained 8B is flat (−0.19 points vs the base) and still 5.6 points behind DeepSeek-V3.2 (alphaXiv summary).
• Exact-command accuracy is strict by design. As Ken Ashe notes in his write-up, a functionally correct but non-canonical command can be marked wrong, so 42% is a tight lower bound on a demanding metric, not "models fail 58% of real tasks."
• The "no execution" property applies to the training loop; KaliBench's own data construction used a frontier API model (Qwen3-Max) for candidate generation and validation (data-construction docs).
• Per-stage verification counts (how many candidates were generated and filtered at each pipeline stage) are not published in the released documentation.

Sources

• arXiv abstract — KaliBench
• arXiv full text (HTML)
• Project page
• GitHub repository
• Training docs (reward weights)
• Data-construction docs
• Dataset reference
• Leaderboard data (authors' Table 1)
• Hugging Face dataset
• RedSage-Qwen3-8B-Ins model card
• alphaXiv summary
• Ken Ashe — KaliBench write-up