acceptodds
Under review as a conference paper at ICLR 2027

When Retrieval Attribution Misses Visual Evidence in Late-Interaction Models

Abstract

Does higher annotation overlap in a retrieval-score highlight reflect sensitivity to the matched query? We test both properties in late-interaction visual retrievers. At , and , fixed query-bank correction increases mean max-direct RegionHit@4 in all ten original checkpoint–support conditions while retaining retrieval scores. An industrial-source position prior nevertheless has higher full-support means than corrected max-direct in all ten, with seven contrasts surviving its separate Holm10 procedure, conditional on the fitted field. A retrospective same-page analysis on a separate subset of 503 eligible pages compares archived highlights against the annotations of their own and another real question. Corrected max-direct favors matched annotations in all ten means, by 2.31 to 19.13 percentage points; seven contrasts meet the adjusted threshold in the new Holm20 family. Correction increases this advantage in all ten means, with six increments meeting that threshold. Inference on this eligible annotation subset is limited by few, unequal dependency blocks. A supporting v0.2 audit traces the suppression–deletion gap to contribution transfer. The findings show why localization levels and matched-query annotation advantages should be reported separately; answer utility remains unestablished.

Primary area:
interpretability and explainable AI
Keywords:
late-interaction retrieval, visual document retrieval, attribution, evidence localization, hubness, spatial bias

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_r1wmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether evidence highlights from late-interaction visual document retrievers like ColPali actually point to the content relevant to a query. The authors apply an existing hubness correction technique to highlight selection, leaving retrieval scores unchanged. They evaluate it on ViDoRe v3 evidence annotations across four retrievers and three domains. The correction consistently improves highlight overlap with annotated evidence, but a simple query-independent layout prior still does better. On pages annotated for two different questions, corrected highlights favor their own question’s evidence, which suggests they carry real query-specific signal. The authors conclude that localization accuracy and query sensitivity should be reported separately. Whether the highlights help downstream answering remains untested, because the reader study failed its qualification screen.

Critical strengths
  1. I think separating absolute localization from query sensitivity is a valuable point. The matched-minus-crossed contrast S in section 2 is a clean design. Any query-independent selector, including the position prior, scores S = 0 by construction. This makes S a natural complement to overlap metrics, which can be inflated by layout regularities.

  2. Highlight selection is decoupled from retrieval. Correction leaves R_0 and the page ranking unchanged by construction (given in figure 2). The paper contrasts this with deletion, which harms nDCG@5 in section 6. I find this design practically sensible.

  3. The authors built a source trained spatial prior that beats their own intervention and report it prominently in table 2. The one-ring exclusion check in section 3 is also reported transparently, even though raw interior selection beats the all-candidate correction for Smith and v0.2. I think the baselines are quite strong and overall transparency is good.

Critical weaknesses
  1. My primary issue with the paper is that I feel like the contribution is modest, and the main method is not the best one in the paper. The correction is an existing hubness normalization, as section 2 says. That alone seems fine, but the paper’s own results show that other methods work better. In table 1, full CSLS direct beats corrected direct in 6 of 10 conditions. In table 3, corrected LOO and full CSLS direct both show stronger query sensitivity (higher S) than corrected direct, in 8 and 7 of 10 conditions. The paper never explains why corrected direct is the main method. This leaves the paper’s key message as "report localization and query sensitivity separately". That is a sensible suggestion, but it is not a strong enough contribution on its own.

  2. Even after correction, the highlights often do no better than random selection. In table 1, corrected direct is below or close to the uniform baseline on several supports, and for ColPali on physics and energy the gap from uniform is not significant. Simple methods that ignore the query do better. A layout prior beats corrected direct in every condition. Just skipping the outer ring of patches with raw scores beats the correction by 2–3 points for all smith supports and v0.2 Energy. So the main body shows that these highlights lose to simple layout rules. To show the highlights are useful, I would recommend something like showing that query information adds something on top of the layout prior, for example by combining the two.

  3. The two main findings are never connected. The paper has two results: i) the layout prior localizes better than the corrected highlights, and ii) the corrected highlights are sensitive to the query. I think an obvious next question is whether that query signal helps on top of the prior. The paper never tests this. A simple version would combine the prior with the corrected scores, or use the query score to re-rank the prior’s top candidates. The two results also come from different data. The first one was never evaluated on the 503-page subset used for the query-sensitivity test in section 3. Because of this, I find it hard to tell how the two findings relate, or which one should matter more in practice.

  4. There is no evidence that the highlights are helpful. The reader study failed early on. Even with the full page, the model answered only 12 of 20 questions correctly, so the planned comparison between corrected and raw highlights was never run. There is also no human study. So the paper does not show that the highlights help a reader answer questions, or that overlap with annotated boxes means a highlight actually contains the needed evidence. I understand the authors are open about this, but it leaves the practical value of the work unclear.

  5. The paper is hard to read and follow. The writing is very dense. Many terms appear without explanation, such as "Holm20, Smith, v0.2" as well as "original" vs "new" families, which makes the abstract and introduction hard to follow on a first read. The sentences/findings contain caveats, so the main takeaway message gets lost. The main text frequently keeps referring to the appendix instead of showing some numbers, which makes it a hard read. I would encourage the authors to rewrite the paper more simply.

Questions for authors
  1. Why is corrected max direct the primary selector when full CSLS direct and corrected LOO score higher on most conditions, for both localization (table 1) and S (table 3)?

  2. Does any combination of the position prior with a query-dependent score exceed the prior alone?

  3. All fixed-bank conditions use the same 120 industrial questions. Would a same-domain held-out bank behave differently? This would clarify whether the correction captures generic hubness or domain mismatch.

  4. What role does the v0.2 deletion audit play in the paper’s central argument?

Reviewer_7dSEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the question "Does higher annotation overlap in a retrieval-score highlight reflect sensitivity to the matched query" but it is very hard to grasp what this actually means. And in the abstract they give no or at least an inconclusive answer to this question.

Critical strengths
  • A lot of work has been done (including a long appendix).
  • Some conclusion is made that for proper understanding it is better to separately report absolute localization and matched-query annotation.
Critical weaknesses
  • Although I have read the paper a number of times it remains completely unclear what the paper is doing. I'm not an expert on this topic but I'm well experienced in reading ICLR-like papers. The task that the paper is aiming to address is never made clear and might be implicit in citing some references that the reader is supposed to know.
  • There is no clear narrative and order in the paper. Very often terms are referred to that have not been defined yet. E.g. in the introduction it talks about original supports and same eligible page while not statement on a dataset has been made. Even the simple use of the word page is only implicitly indicating that this paper is about documents which is never stated.
  • The paper at several places talks about a v0.2 audit which is never defined.
  • Figure 1 and its caption are also not sufficient to give an understanding of the method. E.g. what is a Nemotron pair? What is the outer ring the caption is talking about?
  • The start of section 2 talks about raw score and selectors which are terms that come completely out of the blue. A good description of the overall idea is needed first.
  • Some other limitations:
  • the dataset used for experimentation is quite small (on page 3 it indicates 300 pages).
  • results are for a very specific setting: K = 10, λ = 1 and M = 4
Questions for authors
  • on page 4 the paper talks about annotation but what does that mean? Is that human annotation prior to experimentation? It then later talks about an annotation rule but no such rule has been defined. So what does that mean
  • In section 3 you make a selection of dataset, why are certain documents excluded.
  • on page 7 you have 81/2768 = 0.029263. What does that refer to?
Reviewer_AAcbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether visual-retrieval highlights reflect the matched question, using query-bank correction, a position prior, and a same-page diagnostic on 503 pages. Correction improves localization across ten conditions, but the prior remains stronger on full-support means. A deletion analysis explains attribution changes through contribution transfer.

Critical strengths
  1. Separating absolute localization from matched-query advantage is useful. The position prior and same-page comparison provide complementary controls for interpreting retrieval highlights (Sections 2–3).
  2. The deletion analysis explains selection changes through contribution transfer, supported by a derivation and empirical audit (Appendix B; Figure 3). The paper also reports negative results and limitations transparently.
Critical weaknesses
  1. Same-page uncertainty remains difficult to assess. The authors acknowledge the few, uneven HR/Physics components and provide direction counts (Table 11). However, the largest-component removal in Table 16 concerns different endpoints. Sensitivity of corrected-direct SS and its corrected-minus-raw increment to dominant components remains unreported, limiting confidence in their magnitudes and uncertainty.
  2. A common-sample comparison would strengthen the synthesis. The paper explicitly distinguishes the samples and weights. Tables 7–9 give matched/crossed levels for query-dependent selectors, but the prior's absolute localization on those eligible pages was not evaluated (p. 6). This leaves the relationship between the two findings less directly demonstrated.
  3. The annotation-based scope limits practical conclusions. The authors explicitly disclaim human semantic validation and answer-utility gains. Appendix E reports reader studies, including a historical correction comparison, but failed capability screens prevent a reliable utility conclusion. The low-overlap eligibility rule also lacks a reported threshold sensitivity check, leaving its influence on the same-page findings unclear.
Questions for authors
  1. How sensitive are the effect sizes and intervals for corrected-direct SS and its corrected-minus-raw increment to removing the largest HR/Physics component? Please distinguish this check from Table 16 and retain the twenty-contrast family.
  2. Can you report the position prior's absolute matched-annotation overlap on the same 503 eligible pages, with the same page weighting as Tables 7–9, alongside the query-dependent selectors?
  3. Could you test alternative Jaccard thresholds and manually audit a small sample of eligible question pairs and selected patches, to assess semantic distinctness and evidence sufficiency?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.