Silent cue loss: ingestion drops the colour and highlight that carry compliance meaning — agent extraction F1 falls 0.690 → 0.150
An Amazon preprint localizes the failure to exactly the two cues no mainstream converter preserves (bold and italic are noise), shows a code-only parser recovers the gap, and finds that explicitly typing the cue adds a further +0.085.
Three Amazon researchers quantify what formatting-blind document ingestion costs an agent: on a 76-document task built from real enterprise policy files, micro F1 for recovering the requirements an author singled out falls from 0.690 with formatting-aware parsing to 0.150 with Docling and 0.075 with plain-text extraction. The failure makes agents assert more, not less — Docling's false-positive rate on requirements the author never flagged is four times that of the paper's method.
What shipped
The paper, What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion (arXiv:2610.30026v1, posted 21 Oct 2026), measures and ablates the conversion layer that turns .docx and PDF into tokens. The authors — Yikai Zhao, Saurabh Pandey, and Pradeep Kumar Misra, all Amazon-affiliated — build the task from 76 real policy documents (50–300 pages each: procedures, compliance rules, exception clauses, regulatory references) and ask an LLM to recover the requirements the author flagged visually, e.g. a red requirement or a highlighted exception.
The ground truth deserves attention: labels come from the file's own OOXML run properties, not human judgement, and are scored by a deterministic matcher with no LLM judge. The authors cross-check labels against rendered page images through an independent modality (κ = 0.72) and bound what wording alone can achieve with a supervised text-only classifier (AUC 0.664) — measuring, not asserting, that the task is cue-dependent.
The headline numbers replicate across six answering models from two vendors: five Claude models spanning the Opus, Sonnet, and Haiku tiers, plus OpenAI's GPT-5.6. "The agent does not abstain or return less, it asserts more," the authors write.
A span-level leave-one-out ablation over six cue classes localizes the loss to exactly two:
- highlight: +0.302
- colour: +0.262
Both are Holm-adjusted p < 1e-4; bold, italic, monospace, and strikethrough each sit within ±0.003 of zero. "The same asymmetry appears in PDF," they report. The result inverts intuition: the cues Markdown preserves (bold, italic) have no measurable effect; the two cues every converter drops carry the meaning.
It is also common, not adversarial: 32.8% of a 6,460-document corpus carries a deliberate colour or highlight; 10.3% of documents apply one to an obligation.
Why it matters
The mechanism is structural, not a converter bug anyone can patch in isolation. Docling, Unstructured, and pymupdf4llm target LLM-ready Markdown, and Markdown has no syntax for colour or highlight. The parsing benchmarks the toolchain is measured against — OmniDocBench, ParseBench — score text accuracy, reading order, tables, and formulas, not character-level presentation. Even TexTAR, an ICDAR 2025 oral on recognising textual attributes in document images, covers only bold, italic, underline, and strikeout — not the two cues this ablation finds load-bearing.
The paper's claim is stronger than "formatting matters": to its knowledge, no prior system treats character-level formatting as a first-class extraction cue, and no benchmark scores whether a cue's meaning reached the model. The benchmark it does evaluate on, Microsoft's Office Comprehension Benchmark (OCB), scores tables, charts, and formulas in native Office files — and even there, the authors note, formatting-dependent items are a minority and none turns on what a surviving cue means.
Three properties make this the worst class of failure for a deployed system. It is silent — "nothing in the pipeline signals that information was dropped, so no monitor fires." It inverts the usual failure mode — instead of abstaining, the agent hallucinates extra obligations at four times the false-positive rate, so an eval that treats more answers as more recall will miss it. And it is common — roughly a third of real corpora.
The fix is cheap and measurable. Their formatting-aware parser reads the .docx with python-docx — entirely in code, zero LLM calls — emits FormattingSpan records carrying 12 formatting attributes, and attaches SemanticRole hints through eleven high-confidence formatting-to-semantics mappings: red text or red highlight to critical_warning, any other visible highlight to warning, strikethrough to deprecated. It then serializes a hybrid representation: Markdown for what models saw in pretraining, custom tags for what Markdown cannot express ([COLOR:#FF0000], [HIGHLIGHT:YELLOW]). Restoring 0.690 micro F1 requires preserving two cues, not a new model.
Preserving a cue is necessary but, per the authors' pre-registered test, not always sufficient: typing the cue — telling the model what it means, not just that it exists — adds +0.085 beyond preserving it alone (Holm-adjusted p = 0.049). The authors call the margin modest, and the typed layer showed no benefit on the two models that attempt the task only sparsely. A second negative result: progressive disclosure of the document did not reduce token cost.
For anyone running compliance, policy, or document-grounded agents, the bottom line is that ingestion, not reasoning, is where this reliability is won or lost. The fix is cheap and measurable, which means it is auditable this week: score your own converter against the file's OOXML properties with a deterministic matcher (no LLM judge needed) and check whether colour and highlight survive round-trip.
The paper positions itself between two prior findings: adversarial work at USENIX Security showed moderation systems overlook visual cues (over 86% human recognition at under 1% detection across 13 deployed systems), while PASTA reports a 22% average gain from emphasis for LLAMA-7B and Dillitzer et al. report an attention–performance divergence — emphasis redirects attention but does not inherently improve task accuracy. This result says the question does not generalise across cues: it is specifically about the two no converter preserves.
Caveats
- This is a single, unreplicated preprint v1, not peer-reviewed, and every number is author-reported. The F1 gap, ablation deltas, κ, AUC, and corpus statistics all come from the paper itself.
- No code or data link is public on the arXiv page as of writing, so the deterministic matcher and OOXML labels cannot yet be audited independently, despite the paper stating the evaluation instrument is released alongside.
- The cue-to-semantics mapping is author-designed and acknowledged as fragile: two rules were corrected mid-study after corpus measurement contradicted the initial design.
- The +0.085 typing gain is modest, sits at p = 0.049, and did not hold for two of the six models; the per-model breakdown and PDF converter details sit in sections of the paper that were not independently retrievable for this post. Treat the abstract-level figures as the verified floor.
- The authors' Amazon affiliation is self-reported.