acceptodds
Withdrawn from ICLR 2027

RegionMax-WA: Supervising MaxSim Winners for Visual Document Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual document retrieval preserves the layout and visual context needed to search complex pages, but its usual supervision identifies only which page is relevant, not which evidence should support the match. This gap matters for late-interaction retrievers, whose page score sums a separate patch decision for each query token. We introduce RegionMax-WA, which supervises evidence membership in the same full-page token-wise competition used at retrieval time. OCR text and boxes propose query-dependent candidate evidence; Winner Allocation encourages each content token’s MaxSim winner to fall within it; and the auxiliary loss supplements standard page-ranking training. Across four ViDoRe V2 collections, RegionMax-WA improves macro nDCG@5 (0–100 scale) by 2.21 and 4.21 absolute points over matched page-only PaliGemma-3B and Qwen2.5-VL-3B baselines. It also exceeds a full-complement aggregate-loss control by 0.49 and 0.95 points. On 250 independently annotated query–page pairs, MaxSim winners show greater spatial agreement with human evidence boxes. OCR is used only during training; inference retains the same index format and MaxSim scorer.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual Document Retrieval, Late-Interaction Retrieval, MaxSim, Token-Wise Supervision, OCR-Guided Evidence, Weak Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zU87OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper descrives an approach for learning visual document embeddings that incorporates both full-page similarity and regional evidence similarity during training. To identify evidence regions, the method measures the similarity between query tokens and OCR bounding boxes on the document page. A novel loss function is introduced to enforce that query token embeddings share higher similarity with visual patches inside the evidence region than with patches outside it. Region-level evidence is utilized only during model training; at inference, retrieval is performed at the page level. Experimental evaluations on visual retrieval datasets show performance improvement over the page-level only baseline.

Critical strengths
  • Grounding visual page embeddings with region-based OCR evidence provides a novel way to learn fine-grained visual-semantic representations without introducing extra latency at inference time.
  • The method demonstrates performance improvement over the baseline trained using only full-page visual similarity across most of the selected benchmark datasets.
Critical weaknesses
  • The writing in the Introduction and Related Work / State of the Art sections is difficult to follow and understand without reading the rest of the paper.
  • Experimental setting and results: 1) The evaluation relies exclusively on NDCG@k. Other standard retrieval metrics, such as MRR, Precision@K, and Recall@K are missing and would be helpful to aspects of retrieval behavior. 2) valuation is restricted to four subsets from the ViDoRe benchmark, rather than utilizing the complete benchmark or testing on additional visual document retrieval benchmarks. 3) Although overall performance averaged across the four evaluated ViDoRe subsets improves, the proposed method fails to outperform the baseline in some of the individual subsets. Some discussion explaining these performance difference across subsets is necessary along with evaluation on other datasets to test the generalization of the method.
Questions for authors
  • The paper does not specify the granularity of the OCR bounding boxes used to detect region evidence (words, lines, paragraphs, cells in a table, ...). Box granularity can impact visual overall retrieval performance and should be discussed and evaluated.
  • Some presentation isssues: 1) at the end of Section 3.2, settings A3 and A4 are mentioned prior to being formally defined.2) Table 1 displays the proposed method's results in bold even in cases where competing models achieve higher scores. Common practice uses bold formatting to denote the best score across all compared methods.
Reviewer_TufpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is extremely hard to understand. The introduction is hard to follow and it's hard to locate the novelty. I believe large parts of the paper are completely AI-generated (I fed the introduction to Pangram which led to a 100% generated-by-ai score). I had a hard time understanding the paper's motivation, contribution and novelty.

Critical strengths

I could not locate those.

Critical weaknesses

Mentioned above.

Questions for authors

The paper discloses AI-assisted translation and editing. Did this also include generating substantial parts of the text? Can you state the motivation and main novel contribution plainly?

Reviewer_SCKwOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the granularity of supervision in visual document retrieval. Page-level relevance labels supervise page ranking but do not constrain whether each query token’s highest-scoring patch under MaxSim falls within a region that supports the query. The authors propose RegionMax-WA, which uses OCR text, bounding boxes, and a frozen text encoder to select query-relevant regions from positive pages during training. These regions are mapped to a set of candidate evidence patches on the full-page grid. For each content token, the Winner Allocation auxiliary loss compares the maximum similarity within this evidence set against the maximum similarity over all remaining page patches, encouraging the former to exceed the latter by a specified margin. All content tokens share the same evidence set. At inference, the model retains the original full-page MaxSim scorer and requires no OCR or additional modules.

Critical strengths
  1. The method improves retrieval across all four evaluated collections on both backbones, with macro nDCG@5 gains of 2.21 and 4.21 points over matched page-only baselines. The shared training protocol, three-seed results, and paired bootstrap intervals lend credibility to these gains.
  2. The auxiliary loss directly supervises each content token’s winning patch while preserving the original MaxSim scorer and index format at inference. It requires no OCR or additional inference module, and evaluation against independently annotated evidence boxes provides evidence of improved spatial alignment beyond page-level retrieval accuracy.
Critical weaknesses
  1. The paper needs a token-count-normalized aggregate-loss baseline, compared with the per-token loss under matched margins and temperatures and equally tuned loss weights. The current experiments do not establish whether the gains come from preventing compensation across tokens or from changes in the effective margin or loss scale.

  2. The human-box grounding evaluation needs paired confidence intervals, results across random seeds, and grounding results for the random and shuffled evidence controls. The reported point estimates do not establish whether the grounding gains are robust or adequately distinguish the effects of query-relevant evidence supervision from those of local training more generally.

Questions for authors
  1. Does A4 still perform better when the aggregate baseline averages token scores instead of summing them, as in Eq. (13)? Reporting both retrieval and grounding results may be helpful.
  2. How did the author choose the 250 query–page pairs? More details are encouraged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.