ColPali-style vector indices invert into readable pages: 47% word recall, 98.4% rank-first — and shuffling doesn't stop it
A new Aalto preprint shows the stored index of a visual document retriever is effectively the document; token pooling is the only tested defense that holds.
A new Aalto University preprint shows that the stored index of a ColPali-style visual document retriever can be inverted into a readable page image with zero side information. On the 19,252-page ViDoRe v3 benchmark, pages inverted from raw Tomoro-ColQwen3-8B indices recover 47% of the source words and rank their own source page first 98.4% of the time. The attack also transfers to a second retriever.
What shipped
"Inverting Multi-Vector Visual Document Indices" (arXiv:2610.09920, submitted 7 Oct 2026, 30 pages, under review), by Zhuchenyang Liu, Yao Zhang and Yu Xiao at Aalto University, attacks the storage format used by late-interaction visual document retrievers — the ColPali family and its successors. These models encode a page as roughly a thousand 320-dimensional vectors, one per 32×32-pixel patch, stored in raster order (paper, §1). The vectors live in vector databases and dedicated engines, "often run by a third party." Because no human can read a page from its vectors, the index is easily treated as less sensitive than the page itself.
The authors argue that assumption is wrong for three reasons. First, the index preserves layout: one vector per patch in row-major order fixes every vector's position on the page. Second, each vector comes from a vision-language model that sees its patch in whole-page context, so it carries both local patch information and global page information. Third, these VLMs are pre-trained to read documents — on tens of millions of OCR samples and millions of layout-parsed PDFs — so their vectors plausibly encode enough to reconstruct the page (paper, §1).
The attack frames inversion as conditional document image generation. A conditional flow-matching inverter is trained on about 683k public pages paired with their indices, then evaluated on the 19,252 pages of ViDoRe v3, which are disjoint from training (paper, §1). The attacker works under zero side information: it is not told which encoder produced the store, the page shape, or — for shuffled indices — the vector order. The pipeline infers all three from the index alone: step 0 identifies the encoder using public encoders, step 1 trains models for that encoder on public pages, and step 2 recovers page shape and order per storage configuration (paper, Figure 2).
The paper measures two kinds of leakage. Content leakage is word recall and sensitive-token recall — the fraction of the source page's words, and of its numbers, capitalised words and acronyms, that OCR reads from the inverted page. Identity leakage is re-identification: the retriever ranks all indexed pages against the inverted page and the authors check whether the source page ranks first (paper, §1). From raw indices, inverted pages recover 47% of words and 45% of sensitive tokens, and rank their source page first 98.4% of the time. The paper's teaser shows a mixed-layout page from the ViDoRe v3 industrial subset recovered with tables, illustrations and text legible at 0.620 word recall (paper, Figure 1).
The authors test two cheap protections. Hierarchical token pooling merges vectors at factors 3 and 9; shuffling stores the raw vectors in a random per-page order. Both cut word recall to about 8% (paper, abstract).
Shuffling is attractive: late-interaction scoring does not depend on vector order, so retrieval is unchanged. But it is not a real defense. Each vector was computed with whole-page context, so it still encodes where its patch was; a trained order-restoration model lifts re-identification on shuffled indices from 3.8% to 93.5%. Inverting a pooled index, by contrast, "remains open" (paper, abstract).
The attack generalizes. Applied unchanged to a second retriever, ColQwen3.5-4.5B, inverted pages still rank their source page first 70.2% of the time, though word recall stays below a nearest-neighbour baseline (paper, abstract). The conclusion: multi-vector visual document retrievers "are therefore vulnerable to inversion through their stored index, which should be protected like the documents it encodes."
Why it matters
The headline: the index is the document. The attacked models are real, released artifacts. Tomoro-ColQwen3-8B, the primary target, is an Apache-2.0 ColPali-style retriever built by merging Qwen3-VL-8B-Instruct with Qwen3-Embedding-8B, with a 320-dim projection head and up to 1,280 visual tokens per page. It claims SOTA-or-competitive results on ViDoRe V1–V3 (TomoroAI model card). ColQwen3.5-4.5B, the generalization target, is a ColBERT-style late-interaction model built on Qwen3.5-4B (colpali model table; vLLM docs). If you run any of these behind a managed vector database, the vendor — or anyone who breaches the store — holds the index, and this paper shows that index can be turned back into readable pages.
Concrete implications for people building RAG systems:
- Treat the index as sensitive data. Access control, encryption at rest, and audit logging for the vector store should match the source documents, not the assumption that an unreadable index is low-sensitivity. The paper's threat model is the common deployment: a third party runs the store.
- Do not adopt shuffling as a cheap fix. It costs nothing in retrieval quality but is defeated by a trained position model (3.8% → 93.5% re-identification). Each vector's whole-page context leaks its position.
- Pooling is the only tested defense that currently holds, at the price of a small retrieval-quality cost noted in prior work on pooling and merging (paper, §2). If you must store multi-vector indices with an untrusted party, pooled indices are the safer configuration — for now.
- This is a paradigm property, not a single-model bug. The attack transferred to a second retriever unchanged, so switching ColQwen for Tomoro does not help.
Caveats
This is a single preprint, under review, with no code release found and no independent replication; all numbers are the authors' own. The evaluation runs on ViDoRe v3, a public benchmark of 19,252 pages (vidore), not on sensitive enterprise corpora — though the recovered example is a mixed-layout industrial document, and the sensitive-token metric (numbers, capitalised words, acronyms) targets the kind of content that matters in contracts and filings. The attack also assumes the attacker can identify the encoder and train on public pages, which is a realistic but not universal threat model. Word recall is measured through OCR, so it likely undercounts what a human reader could extract. Finally, the paper's method sections (architecture details, training compute, evaluation protocol) could not be fully verified from the public HTML; the headline figures come from the abstract, introduction, and §3, which are unambiguous about the results.