Verdict Without the Rule: LLM compliance verdicts change only 5–8% of the time when the supplied rule is deleted, swapped, or negated
A Lexsi Labs audit across five models and 20 domains finds high benchmark accuracy coexists with near-zero rule sensitivity — and neither prompting nor activation steering fixes it.
A preprint from Lexsi Labs finds that LLM compliance verdicts barely change when the supplied regulatory rule is deleted, swapped, or negated. Across five models and 20 regulatory and platform-policy domains, verdicts changed only 5–8% of the time on average, and the dedicated guard model was the least rule-sensitive and least accurate of the five.
What shipped
The paper (arXiv:2610.12313) introduces two instruments for auditing rule dependence and runs both across five models and 20 domains from the OmniCompliance-100K dataset: 200 cases per domain per model, 64,485 total generations.
OCS (Output Compliance Sensitivity) is a reference-free, black-box behavioral metric. For each (case, rule) sample it applies one of three perturbations to the rule field while holding the document fixed: delete the rule entirely, swap it for the canonical rule of a different domain, or negate its mandatory modals ("must"/"shall" → "must not"/"shall not"). OCS is 1 if the verdict flips, 0 otherwise. A high OCS means the system reacts to what the rule says; a low OCS means it largely does not.
ICS-delta is the representational counterpart: it measures whether the compliance-predictive direction in the model's residual stream shifts under the same perturbations, independent of what the model outputs. This catches the case where a model suppresses the rule's influence right before answering while still representing it internally.
The models are Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Llama-Guard-3-8B. The headline numbers, as reported by the authors:
- Pooled OCS is 0.053 under deletion, 0.070 under negation, 0.082 under swap. Per-model aggregate OCS ranges from 0.013 (Llama-Guard-3-8B) to 0.094 (Qwen2.5-7B), mean 0.069.
- Baseline accuracy on the same cases is 90.3–91.9% for the four general-purpose models — and 51.1% for Llama-Guard-3-8B, barely above chance.
- ICS-delta is 12.8–22.9% of mean baseline |ICS| across the four general-purpose models: the compliance-predictive representation barely moves when the rule is perturbed.
So four models that score 90%+ on a compliance benchmark change their verdict only ~5–10% of the time when the rule they were given is deleted, swapped for another domain's rule, or negated. The guard model is the worst of both worlds: least rule-sensitive and least accurate.
Why it matters
The paper's own rule-necessity control keeps this from being a cheap shot at benchmark accuracy. On the subset of cases where deleting the rule changes a previously correct prediction (94–187 cases for the four general-purpose models), OCS-swap jumps to 0.73–1.00 for every model. The low aggregate reflects easy cases more than blanket neglect: on cases where the rule genuinely determines the outcome, models track it. The problem is that a compliance system cannot tell you which regime it is in, and neither can its accuracy score.
Two intervention attempts make the practical stakes concrete. Rule-grounded prompting (the model must first quote the most relevant rule clause, then give its verdict) produced no significant improvement in any model — and on Qwen2.5-7B's LOW-OCS cases it significantly backfired (Δ = −0.028, p < 0.0001). Activation steering along the ICS direction, at several strengths, never significantly beat baseline on either model tested. You cannot prompt or steer your way out of this gap.
The linguistic analysis adds a buildable signal: of five textual features tested, only mean dependency-tree depth survives multiple-comparison correction (Spearman ρ = 0.570 vs. domain-level OCS). More syntactically complex rules are more rule-sensitive — a cheap, label-free way to prioritize which rules need auditing.
For anyone shipping a compliance system, the takeaway is that accuracy alone does not establish that a verdict is grounded in the supplied rule. A verdict that survives deleting the rule entirely is not more defensible for having scored well on a benchmark. The paper positions OCS as a reference-free, black-box audit that needs no labels, and its generalization check on Llama-3.3-70B found OCS-agg of 0.255 on LegalBench and 0.318 on ContractNLI versus 0.081 on OmniCompliance-100K — rule sensitivity rises out of distribution, though LegalBench accuracy was near chance, so task difficulty may explain part of the increase.
Caveats
- This is a preprint submitted 8 Oct 2026, not peer-reviewed, with no independent replication. Every number above is the authors' own report, and no code or data release is mentioned in the paper.
- The standard prompt truncates rule text at 300 characters, so low OCS can reflect unavailable rule information rather than ignored rule information. The authors flag this themselves.
- OCS measures whether a verdict changes, not whether a changed verdict would be legally correct under the perturbed rule.
- The guard-model result is a task-format adaptation, not a general claim about guard models: the authors bypassed Llama-Guard-3-8B's fixed 14-category taxonomy and substituted a single custom "O1: {rule text}" category. They explicitly frame the 51.1% as reflecting that mismatch, and the model's extreme response bias (it answers "compliant" almost regardless of condition) shows up in its near-zero OCS.
- The LOW-OCS/HIGH-OCS partitions are defined on Llama-3.3-70B and transferred to the other models; the generalization check uses one model.