- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes EA-MaxSim, a training-free reranking score for visual document retrieval. It augments MaxSim with query coverage and a measure of how many spatially distinct regions on a page provide apparent evidence. The method reuses the existing query–patch similarities and improves top-rank results over MaxSim on ViDoRe v2 with two backbones.
Critical strengths
The method is simple to add to an existing late-interaction retriever: it changes the reranking score without retraining the encoder or changing stored document embeddings. The results on ViDoRe v2 are encouraging across both backbones, and the component studies and reranker comparisons help show where the improvement comes from and what it costs.
Critical weaknesses
The strongest results use parameters selected separately for each document subset. A shared configuration still helps, but gives smaller gains. This leaves open how reliably the method would transfer to a new collection where subset-specific tuning is unavailable. Significance tests for the exact main-result configuration would also help establish their reliability.
I am also unsure that the proposed richness score consistently measures useful evidence. It counts connected regions among the most salient patches on every page, even when those patches provide weak support for the query. A visually busy but irrelevant page could therefore look rich, while a compact table containing the answer might receive a low score. Examples of these cases, or an analysis using annotated evidence regions, would make the interpretation more convincing.
Questions for authors
See weakness above
2 comments
Thank you for the detailed response. The rebuttal addressed most of my concerns and strengthened my assessment of the paper. I will raise my overall rating accordingly.
We thank the reviewer for the careful reading and the concrete suggestions.
W1a. Shared configuration. We agree that per-subset calibration gives the strongest results; the method nevertheless remains effective without it. The single shared configuration in Table 7 improves nDCG@1 on every subset, and the macro-averaged gain is significant for both backbones (ColQwen2 54.0 → 57.3, ColPali 53.1 → 56.8; p < 0.01). The gain also does not depend on a precisely tuned point: with one configuration for all four subsets, all 15 combinations with α ∈ {0.2, …, 1.0} and β ∈ {0.05, 0.1, 0.15} improve over MaxSim on both backbones.
W1b. Significance for the exact main-result configuration. Table 11 (Appendix B.6) is computed for the exact main-result configuration on ViDoRe v2; its EA-MaxSim values equal those in Table 1. The improvements are significant at every nDCG and MRR cutoff for both backbones, and at four of the six Recall cutoffs. Table 12 (ViDoSeek) uses the optional query-aware token selection; its EA-MaxSim values equal those in Table 13 (Sel). We additionally report these tests for the ViDoSeek configuration of Table 2, without token selection (paired randomization test over queries):
As each ViDoSeek query has a single relevant page, which MaxSim already places in the top 5 for 94.5% (ColQwen2) and 90.3% (ColPali) of queries, the room for gains at larger cutoffs is small. With the query-aware token selection (Appendix B.8), designed for long compositional queries, the nDCG gains increase for both backbones, and the ColQwen2 improvement at nDCG@1 becomes significant (74.3 → 75.8, p = 0.034; Table 12). We will add the table above to Appendix B.6.
W2. Does richness measure useful evidence? We agree that richness alone is not a complete measure of query-specific evidence, and we would like to clarify its intended role.
Role of richness. Coverage and richness are auxiliary signals that calibrate MaxSim; neither is designed to judge relevance or rank pages on its own. EA-MaxSim reranks the MaxSim candidates, and MaxSim and the auxiliary signals are all normalized to [0, 1]. With the recommended weight β = 0.1, richness therefore only affects the order of candidates whose MaxSim and coverage scores are already close. A visually busy page that is unrelated to the query typically has a much lower MaxSim score, and a page that matches only part of the query also receives a lower coverage score, so richness alone cannot lift such a page above a clearly better-matching candidate.
Relative rather than absolute salience. As the reviewer notes, the supporting patches are the top 10% most query-salient patches of each page rather than those above an absolute similarity threshold. An ideal richness measure would consider only the regions that actually provide evidence for the query, but identifying them reliably is difficult: token–patch similarities can be biased, and their scale differs across pages and backbones. We therefore define richness more narrowly, as the number of spatially distinct regions into which a page's most query-relevant patches fall. Because it depends only on how patches rank within a page, this definition is insensitive to the similarity scale: the same setting (p = 10%) is used for both backbones, and EA-MaxSim stays above MaxSim on ViDoRe v2 for every tested p from 5% to 30%.
Compact evidence. A page whose supporting patches fall into several distinct regions receives a higher richness score than one whose evidence is confined to a single compact region. This is intended: our working hypothesis is that evidence appearing in several places is richer. Because the effect of richness is bounded as described above, a compact page that matches the query well, such as a single table containing the answer, still ranks highly through MaxSim and coverage.
Example. Figure 8b (Appendix C) shows a case in which the MaxSim scores of two pages are close. Both pages match the query keywords and receive nearly the same MaxSim score. The supporting patches on the relevant page (right) fall into more distinct regions, spread over several paragraphs that discuss ligand density, and EA-MaxSim ranks it first.
Annotated evidence regions. Neither ViDoRe v2 nor ViDoSeek provides region-level evidence annotations, so an analysis against annotated regions is not possible on these benchmarks. We will add the discussion above to the appendix.
We hope our responses address your concerns and would be grateful if you would consider raising your score.