acceptodds
Under review as a conference paper at ICLR 2027

EVIE: Evidence-Vector-Informed Embeddings for Visual Document Retrieval

Abstract

Accurate and scalable visual document retrieval (VDR) requires both fine-grained page understanding and efficient indexing, yet existing approaches struggle to achieve both. OCR-based text retrieval adds preprocessing latency and can lose visual and structural cues needed to understand complex pages. Single-vector vision-language models bypass OCR, but compressing an entire page into one vector limits the granularity of query–document matching. Multi-vector retrievers with MaxSim provide finer interactions, yet demand large indexes and still leave room for accuracy improvements. We argue that overcoming these limitations requires preserving query-relevant page evidence throughout representation learning and index construction. To this end, we introduce EVIE (Evidence-Vector-Informed Embeddings), a family of native visual document retrievers integrating three key innovations: (1) Evidence-judged data governance, which uses a multimodal judge to identify answer-bearing positives and filter unreliable negatives. (2) Bidirectional teacher–student learning with symmetric listwise distillation and prefix-based Matryoshka representation learning (Prefix-MRL), enabling one student checkpoint to serve six nested embedding dimensions without re-encoding. (3) Hierarchical agglomerative index compression (HAC), which clusters page tokens with spatial regularization and stores semantic centroids for single-stage MaxSim retrieval. Extensive experiments across 138 tasks from ViDoRe V1, V2, V3, and JinaVDR validate EVIE. EVIE-8B achieves 66.75 nDCG@10 on ViDoRe V3, exceeding the strongest compared baseline by 1.43 points, and a four-suite average of 79.51. EVIE-4.5B with HAC retains 59.58 nDCG@10 at only 3.81 GB per million pages, reducing index storage by relative to an uncompressed 1024-token, 256-dimensional BF16 index. Together, these results improve the accuracy–storage trade-off for visual document retrieval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual Document Retrieval, Late Interaction, Vision-Language Models, Matryoshka Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f51pOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is extremely hard to understand. The introduction is hard to follow and it's hard to locate the novelty. I believe large parts of the paper are completely AI-generated (I fed the introduction abstract to Pangram which led to a 100% generated-by-ai score for each). I had a hard time understanding the paper's motivation, contribution and novelty.

Critical strengths

I could not locate those as the paper is not clear to me.

Critical weaknesses

Mentioned above.

Questions for authors

The paper discloses AI-assisted translation and editing. Did this also include generating substantial parts of the text? Can you state the motivation and main novel contribution plainly?

Reviewer_KwoGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new methodology for visual document retrieval, supported by more evidence. The proposed approach is composed of 3 stages. The first stage is an evidence-governed learning stage during which a multimodal judge sorts candidate pages with full, partial, or no evidence, and a teacher/student EVIE scheme is used. The second stage is compact evidence indexing, which exploits spatial clustering to store only semantic centroids, and the last stage is retrieval via the MaxSim strategy. An experimental validation is done on 138 tasks from ViDoRe V1 to V3 and from Jina VDR.

Critical strengths
  • The paper tackles an important challenge. Multi-vector retrieval for visual pages has become the most accurate method for RAG with complex documents, but its storage cost is a significant issue at scale in real-world contexts.
  • The paper is well written with clear motivations.
  • The experimental validation is carefully designed, including ablation studies.
Critical weaknesses
  • The main weakness is the low technical novelty. EVIE does not introduce any conceptual breakthrough: it is a carefully crafted combination of familiar building blocks. The trade-off between accuracy and storage has also already been examined.
  • The procedure for selecting some hyperparameters is not described. For instance, how is the kd temperature chosen?
  • No variance measures are reported. How many runs ?
Questions for authors
  • The idea of evidence-judged supervision is interesting, but why not have more than 3 levels of evidence? In particular, what is the impact of having a large amount of data as partial evidence?
  • How would other hierarchical clustering methods perform? +Storage can also be addressed through quantization. Could this be added to the method?
Reviewer_juhoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EVIE combines evidence-based relabeling, prefix-wise ranking distillation, and spatially guided token clustering. It gives one retriever several storage operating points and performs strongly across four suites. The evidence supervision is convincing, while the compression design leaves a few important comparisons open.

Critical strengths
  1. The 900-candidate human audit makes the evidence relabeling much more credible than relying on a judge alone.
  2. Matched training controls separate several supervision and distillation gains.
  3. The narrow student remains competitive, and HAC improves on semantic-only pooling across three encoders without changing MaxSim.
Critical weaknesses
  1. Equation 15 adds fixed sinusoidal coordinates directly to learned semantic channels. The resulting distances depend on their arbitrary relative basis and include semantic-position cross terms. Compare against concatenated features or an explicit spatial distance penalty to establish that the gain comes from useful spatial structure.
  2. The compact-index comparison omits scalar or product quantization on the same trained EVIE embeddings. At a fixed byte budget, keeping more low-precision tokens may preserve evidence better than averaging them. This comparison matters because HAC still loses 4.81 V3 points after narrowing to 64 dimensions.
  3. Partial-evidence masking applies only to annotated negative slots. Those pages can still enter other candidate pools, while the score-based attenuation does not explicitly enforce the known partial labels. Measure these residual conflicts and test a consistent query-page mask across retrieval and distillation losses.
Questions for authors

Do HAC's gains survive a basis-independent spatial control and a quantized EVIE baseline at equal bytes? How often do known partial-evidence pairs reappear in unmasked candidate pools?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.