acceptodds
Under review as a conference paper at ICLR 2027

Evidence-Aware MaxSim Reranking for Visualized Document Retrieval

Abstract

Late-interaction retrieval models such as ColPali and ColQwen2 have recently shown strong performance on the Visualized Document Retrieval (VDR) task by matching query tokens against page-level patch embeddings via MaxSim. However, we observe that MaxSim-based scoring often exhibits systematic ranking failures, where pages with peaked similarity or insufficient supporting evidence are over-ranked, leading to unreliable retrieval results. In this work, we propose EA-MaxSim, an evidence-aware reranking framework that augments MaxSim with two complementary signals: query-level evidence coverage, which measures how well a page supports the full query intent, and page-level evidence richness, which characterizes the structural sufficiency of supporting evidence within a page by leveraging spatial layout cues. EA-MaxSim operates as a lightweight reranking module, requiring no changes to model training, document encoding, or indexing. Extensive experiments on ViDoRe v2 and ViDoSeek demonstrate that EA-MaxSim consistently improves ranking quality across different backbones and datasets, particularly at top ranks. Further analyses show that evidence coverage provides a robust calibration signal, while evidence richness offers complementary improvements.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual document retrieval, late-interaction retrieval, reranking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aYKLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes EA-MaxSim, a training-free reranking score for visual document retrieval. It augments MaxSim with query coverage and a measure of how many spatially distinct regions on a page provide apparent evidence. The method reuses the existing query–patch similarities and improves top-rank results over MaxSim on ViDoRe v2 with two backbones.

Critical strengths

The method is simple to add to an existing late-interaction retriever: it changes the reranking score without retraining the encoder or changing stored document embeddings. The results on ViDoRe v2 are encouraging across both backbones, and the component studies and reranker comparisons help show where the improvement comes from and what it costs.

Critical weaknesses

The strongest results use parameters selected separately for each document subset. A shared configuration still helps, but gives smaller gains. This leaves open how reliably the method would transfer to a new collection where subset-specific tuning is unavailable. Significance tests for the exact main-result configuration would also help establish their reliability.

I am also unsure that the proposed richness score consistently measures useful evidence. It counts connected regions among the most salient patches on every page, even when those patches provide weak support for the query. A visually busy but irrelevant page could therefore look rich, while a compact table containing the answer might receive a low score. Examples of these cases, or an analysis using annotated evidence regions, would make the interpretation more convincing.

Questions for authors

See weakness above

2 comments

We thank the reviewer for the careful reading and the concrete suggestions.

W1a. Shared configuration. We agree that per-subset calibration gives the strongest results; the method nevertheless remains effective without it. The single shared configuration in Table 7 improves nDCG@1 on every subset, and the macro-averaged gain is significant for both backbones (ColQwen2 54.0 → 57.3, ColPali 53.1 → 56.8; p < 0.01). The gain also does not depend on a precisely tuned point: with one configuration for all four subsets, all 15 combinations with α ∈ {0.2, …, 1.0} and β ∈ {0.05, 0.1, 0.15} improve over MaxSim on both backbones.

W1b. Significance for the exact main-result configuration. Table 11 (Appendix B.6) is computed for the exact main-result configuration on ViDoRe v2; its EA-MaxSim values equal those in Table 1. The improvements are significant at every nDCG and MRR cutoff for both backbones, and at four of the six Recall cutoffs. Table 12 (ViDoSeek) uses the optional query-aware token selection; its EA-MaxSim values equal those in Table 13 (Sel). We additionally report these tests for the ViDoSeek configuration of Table 2, without token selection (paired randomization test over queries):

BackboneMetricMaxSimEA-MaxSimp
ColQwen2nDCG@174.3475.220.117
ColQwen2nDCG@585.5185.870.118
ColQwen2Recall@174.3475.220.112
ColQwen2Recall@594.4894.401.000
ColPalinDCG@167.3469.090.008 ‡
ColPalinDCG@579.9780.780.020 †
ColPaliRecall@167.3469.090.008 ‡
ColPaliRecall@590.2890.810.305

As each ViDoSeek query has a single relevant page, which MaxSim already places in the top 5 for 94.5% (ColQwen2) and 90.3% (ColPali) of queries, the room for gains at larger cutoffs is small. With the query-aware token selection (Appendix B.8), designed for long compositional queries, the nDCG gains increase for both backbones, and the ColQwen2 improvement at nDCG@1 becomes significant (74.3 → 75.8, p = 0.034; Table 12). We will add the table above to Appendix B.6.

W2. Does richness measure useful evidence? We agree that richness alone is not a complete measure of query-specific evidence, and we would like to clarify its intended role.

Role of richness. Coverage and richness are auxiliary signals that calibrate MaxSim; neither is designed to judge relevance or rank pages on its own. EA-MaxSim reranks the MaxSim candidates, and MaxSim and the auxiliary signals are all normalized to [0, 1]. With the recommended weight β = 0.1, richness therefore only affects the order of candidates whose MaxSim and coverage scores are already close. A visually busy page that is unrelated to the query typically has a much lower MaxSim score, and a page that matches only part of the query also receives a lower coverage score, so richness alone cannot lift such a page above a clearly better-matching candidate.

Relative rather than absolute salience. As the reviewer notes, the supporting patches are the top 10% most query-salient patches of each page rather than those above an absolute similarity threshold. An ideal richness measure would consider only the regions that actually provide evidence for the query, but identifying them reliably is difficult: token–patch similarities can be biased, and their scale differs across pages and backbones. We therefore define richness more narrowly, as the number of spatially distinct regions into which a page's most query-relevant patches fall. Because it depends only on how patches rank within a page, this definition is insensitive to the similarity scale: the same setting (p = 10%) is used for both backbones, and EA-MaxSim stays above MaxSim on ViDoRe v2 for every tested p from 5% to 30%.

Compact evidence. A page whose supporting patches fall into several distinct regions receives a higher richness score than one whose evidence is confined to a single compact region. This is intended: our working hypothesis is that evidence appearing in several places is richer. Because the effect of richness is bounded as described above, a compact page that matches the query well, such as a single table containing the answer, still ranks highly through MaxSim and coverage.

Example. Figure 8b (Appendix C) shows a case in which the MaxSim scores of two pages are close. Both pages match the query keywords and receive nearly the same MaxSim score. The supporting patches on the relevant page (right) fall into more distinct regions, spread over several paragraphs that discuss ligand density, and EA-MaxSim ranks it first.

Annotated evidence regions. Neither ViDoRe v2 nor ViDoSeek provides region-level evidence annotations, so an analysis against annotated regions is not possible on these benchmarks. We will add the discussion above to the appendix.

We hope our responses address your concerns and would be grateful if you would consider raising your score.

Thank you for the detailed response. The rebuttal addressed most of my concerns and strengthened my assessment of the paper. I will raise my overall rating accordingly.

Reviewer_GNY4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes EA-MaxSim, a training-free reranking method that augments MaxSim with query coverage and spatial evidence richness. It improves visual document retrieval using existing embeddings, with reported gains across two backbones on ViDoRe v2 and ViDoSeek.

Critical strengths

The paper is well written and easy to follow. The motivation is clear and the solution is well motivated. The problem of getting bad recomendations because of partial but very high scores for some of the query tokens is a well known problem. The proposed solution is training free. The proposed solution makes a lot of sense. The coverage and richness measures are good addons to the ranking score. The benchmark improvements, particularly at rank one on ViDoRe v2, demonstrate practical value. The ablations support the complementary contributions of coverage and richness.

Critical weaknesses

The richness weight β is sensitive to the dataset, making its selection for new domains less straightforward. The shared-configuration experiments partially address this concern, but transfer to unseen domains remains unclear.

The contribution is narrow in scope, offering modest but tangible practical value to a specialized audience.

Questions for authors

How should β be selected for a new domain with little or no labeled validation data? Have you evaluated whether a value tuned on one domain transfers reliably to another?

1 comment

We thank the reviewer for the positive evaluation and the clear questions.

Q1 / W1. Selecting β for a new domain. We agree that the best β differs across datasets (Appendix B.2.2). Our results suggest, however, that a small β is a reliable default when no labeled data are available. We report two analyses on ViDoRe v2, whose four subsets cover different domains (biomedicine, economics, corporate sustainability) and document types (slides vs. long reports).

(i) Joint α×β grid. One configuration shared by all four subsets, macro nDCG@1 (%); α = β = 0 is MaxSim.

ColQwen2 (MaxSim 54.0):

α \ β00.050.10.150.20.3
054.055.055.955.553.852.7
0.255.856.557.255.955.053.8
0.455.955.957.256.255.954.3
0.655.556.257.356.555.554.2
0.855.256.156.956.055.353.7
1.055.155.556.655.154.653.5

ColPali (MaxSim 53.1):

α \ β00.050.10.150.20.3
053.153.653.753.953.651.9
0.254.854.655.755.654.552.8
0.455.155.456.255.755.353.7
0.655.655.456.855.955.453.9
0.855.355.756.555.655.253.3
1.055.655.756.255.855.254.3

For every α ≥ 0.2, β = 0.1 gives the best result on both backbones, and every combination with α ∈ [0.2, 1.0] and β ∈ [0.05, 0.15] improves over MaxSim. Larger values reduce the gain; at β = 0.3, several combinations fall below MaxSim.

(ii) One β for all four domains. We fix β = 0.1 for all subsets, with α and τ chosen per subset. nDCG@1 / @3 / @5 (%):

SettingColQwen2ColPali
MaxSim54.0 / 54.6 / 56.053.1 / 54.7 / 55.7
β chosen per subset (Table 1)58.8 / 56.6 / 58.159.1 / 57.2 / 58.4
β = 0.1 for all subsets58.7 / 56.3 / 57.757.7 / 56.7 / 57.6

With a single β for all four domains, EA-MaxSim remains above MaxSim on every subset for both backbones.

For a new domain without labeled data, we therefore recommend β = 0.1. If a small labeled set is available, β can be tuned around this value. We will add these analyses to Appendix B.2.

W2. Scope. We thank the reviewer for recognizing the practical value of the method, and we agree that EA-MaxSim is a focused contribution. We would only add that it requires no retraining or re-indexing and can be applied on top of existing patch-level late-interaction retrievers, such as the ColPali/ColQwen family. We also hope that our analysis of MaxSim's ranking failures (peaked similarity and insufficient supporting evidence) is of interest to work on late-interaction retrieval more broadly.

We hope our responses address your concerns and would be grateful if you would consider raising your score.

Reviewer_WtGSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a simple and computationally efficient reranking method for Visual Document Retrieval (VDR) based on MaxSim scoring. The proposed approach reranks initial retrieval results by evaluating two aspects of query evidence within a page: token coverage (the percentage of relevant tokens for a document page) and spatial evidence richness (how relevant evidence for query textual tokens is distributed across the page). Evaluated across standard VDR benchmark datasets, the method achieves consistent performance improvements over standard retrieval without reranking while maintaining a low computational overhead.

Critical strengths
  • The paper proposes a simple and computationally efficient reranking method that explicitly models evidence for query tokens using coverage and spatial richness metrics.
  • The paper reports consistent retrieval performance improvements across standard VDR benchmarks and base retrieval frameworks and is competitive (better or comparable) against more expensive VLM-based reranking methods. Furthermore, ablation experiments confirm that both proposed terms (coverage and spatial richness) contribute positively to the overall performance improvement.
Critical weaknesses
  • Incomplete ablation study: the ablation experiments lacks a sensitivity analysis for several of the parameters of the method: the threshold used to filter relevant tokens for the coverage score, the number of initial selected candidates retrieved by MaxSim, the percentage of patches retained for richness, the weights in the combination of MaxSim with coverage and richness, ...
Questions for authors
  • There are some presentation issues in the paper: 1) VDR usually accounts for Visual Document Retrieval and not Visualized Document Retrieval. 2) There is a potential notation missmatch between figure 1 and the text in section 3.1. In the bottom-left the variable N_i is used and it is unclear if it refers to the patch embeddings p_i introduced in Section 3.1. 3) Figure 1 (top-right) visually illustrates a 4-connectivity neighborhood for grouping adjacent patches, whereas the main text in Section 3 explicitly specifies an 8-connectivity scheme. 4) The description of the baseline VLM reranker in section 4.5 lacks some more details
2 comments

We thank the reviewer for the positive assessment and the constructive suggestions.

W1. Parameter sensitivity. We agree that this is important. Sensitivity to three of the listed parameters is reported in Appendix B.2 of the submission, and we have added an analysis of the remaining one:

  • Coverage threshold τ: Appendix B.2.3 (Fig. 4 for ViDoRe v2, Fig. 5 for ViDoSeek), for hard, soft, and percentile-based coverage, both backbones, and nDCG@1/3/5.
  • Number of MaxSim candidates K: Appendix B.2.4 (Fig. 6), K from 20 to 500.
  • Combination weights α and β: Appendix B.2.2 (Fig. 3) reports one-dimensional sweeps; we additionally report a joint α×β grid below.
  • Percentage p of patches retained for richness (new): below.

Support ratio p. Only p is varied; all other parameters are fixed at the main-result configuration. ViDoRe v2, nDCG@1 / @3 / @5 (%):

pColQwen2ColPali
MaxSim54.0 / 54.6 / 56.053.1 / 54.7 / 55.7
5%57.8 / 56.3 / 57.557.8 / 55.8 / 57.2
10% (paper)58.8 / 56.6 / 58.159.1 / 57.2 / 58.4
15%58.3 / 56.9 / 58.058.1 / 56.1 / 57.6
20%57.6 / 56.7 / 57.657.1 / 55.9 / 57.2
30%56.4 / 56.0 / 57.256.8 / 56.0 / 57.1

EA-MaxSim improves over MaxSim at every tested ratio and cutoff. Over p ∈ [5%, 30%], nDCG@1 stays within 2.4 points of the best setting, with 10–15% performing best.

Joint α×β grid. ViDoRe v2, one configuration shared by all four subsets, macro nDCG@1 (%); α = β = 0 is MaxSim.

ColQwen2 (MaxSim 54.0):

α \ β00.050.10.150.20.3
054.055.055.955.553.852.7
0.255.856.557.255.955.053.8
0.455.955.957.256.255.954.3
0.655.556.257.356.555.554.2
0.855.256.156.956.055.353.7
1.055.155.556.655.154.653.5

ColPali (MaxSim 53.1):

α \ β00.050.10.150.20.3
053.153.653.753.953.651.9
0.254.854.655.755.654.552.8
0.455.155.456.255.755.353.7
0.655.655.456.855.955.453.9
0.855.355.756.555.655.253.3
1.055.655.756.255.855.254.3

Performance is stable over a broad range of α: every combination with α ∈ [0.2, 1.0] and β ∈ [0.05, 0.15] improves over MaxSim for both backbones. β should remain small; at β = 0.3, several combinations fall below MaxSim.

We will add both analyses to Appendix B.2.

Presentation.

  1. We will use "Visual Document Retrieval" throughout.
  2. N_i in Fig. 1 denotes the i-th patch embedding of the page, i.e., p_i^d in Section 3.1, and R^{N_{p'}×d} denotes the page's patch-embedding matrix. We will use the notation of Section 3.1 in the figure (p_i^d and R^{|P_d|×d}), and likewise use m_j for patch salience scores in panel (c).
  3. Thank you for catching this. The implementation uses 8-connectivity, as stated in Section 3: two supporting patches are adjacent if their grid positions differ by at most one row and one column, diagonals included (ε = 1 under the L∞ distance). Fig. 1 incorrectly depicts 4-connectivity, and we will correct it.
  4. VLM reranker details (Section 4.5; partly in Appendix B.4). Both VLMs rerank the same MaxSim candidate list pointwise. Qwen2-VL-2B-Instruct is a generic VLM without reranking-specific training; MonoQwen2-VL-v0.1 is a dedicated reranker, LoRA-adapted from Qwen2-VL-2B on ColPali training data with mined hard negatives. Each (page image, query) pair is scored with the prompt "Assert the relevance of the previous image document to the following query, answer True or False. The query is: {query}". The relevance score is P(True), obtained from a two-way softmax over the logits of "True" and "False". Images are limited to 512·28·28 pixels. Inference uses bf16 and SDPA with batch size 8 on one H100-80GB; reported latencies include image decoding, preprocessing, and the forward pass, averaged over 20 queries. We will add the prompt and these details to Appendix B.4 and point to them from Section 4.5.

We hope our responses address your concerns and would be grateful if you would consider raising your score.

Thank you for your detailed response. It has clarified all my previous concerns

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.