- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies whether evidence highlights from late-interaction visual document retrievers like ColPali actually point to the content relevant to a query. The authors apply an existing hubness correction technique to highlight selection, leaving retrieval scores unchanged. They evaluate it on ViDoRe v3 evidence annotations across four retrievers and three domains. The correction consistently improves highlight overlap with annotated evidence, but a simple query-independent layout prior still does better. On pages annotated for two different questions, corrected highlights favor their own question’s evidence, which suggests they carry real query-specific signal. The authors conclude that localization accuracy and query sensitivity should be reported separately. Whether the highlights help downstream answering remains untested, because the reader study failed its qualification screen.
Critical strengths
-
I think separating absolute localization from query sensitivity is a valuable point. The matched-minus-crossed contrast S in section 2 is a clean design. Any query-independent selector, including the position prior, scores S = 0 by construction. This makes S a natural complement to overlap metrics, which can be inflated by layout regularities.
-
Highlight selection is decoupled from retrieval. Correction leaves R_0 and the page ranking unchanged by construction (given in figure 2). The paper contrasts this with deletion, which harms nDCG@5 in section 6. I find this design practically sensible.
-
The authors built a source trained spatial prior that beats their own intervention and report it prominently in table 2. The one-ring exclusion check in section 3 is also reported transparently, even though raw interior selection beats the all-candidate correction for Smith and v0.2. I think the baselines are quite strong and overall transparency is good.
Critical weaknesses
-
My primary issue with the paper is that I feel like the contribution is modest, and the main method is not the best one in the paper. The correction is an existing hubness normalization, as section 2 says. That alone seems fine, but the paper’s own results show that other methods work better. In table 1, full CSLS direct beats corrected direct in 6 of 10 conditions. In table 3, corrected LOO and full CSLS direct both show stronger query sensitivity (higher S) than corrected direct, in 8 and 7 of 10 conditions. The paper never explains why corrected direct is the main method. This leaves the paper’s key message as "report localization and query sensitivity separately". That is a sensible suggestion, but it is not a strong enough contribution on its own.
-
Even after correction, the highlights often do no better than random selection. In table 1, corrected direct is below or close to the uniform baseline on several supports, and for ColPali on physics and energy the gap from uniform is not significant. Simple methods that ignore the query do better. A layout prior beats corrected direct in every condition. Just skipping the outer ring of patches with raw scores beats the correction by 2–3 points for all smith supports and v0.2 Energy. So the main body shows that these highlights lose to simple layout rules. To show the highlights are useful, I would recommend something like showing that query information adds something on top of the layout prior, for example by combining the two.
-
The two main findings are never connected. The paper has two results: i) the layout prior localizes better than the corrected highlights, and ii) the corrected highlights are sensitive to the query. I think an obvious next question is whether that query signal helps on top of the prior. The paper never tests this. A simple version would combine the prior with the corrected scores, or use the query score to re-rank the prior’s top candidates. The two results also come from different data. The first one was never evaluated on the 503-page subset used for the query-sensitivity test in section 3. Because of this, I find it hard to tell how the two findings relate, or which one should matter more in practice.
-
There is no evidence that the highlights are helpful. The reader study failed early on. Even with the full page, the model answered only 12 of 20 questions correctly, so the planned comparison between corrected and raw highlights was never run. There is also no human study. So the paper does not show that the highlights help a reader answer questions, or that overlap with annotated boxes means a highlight actually contains the needed evidence. I understand the authors are open about this, but it leaves the practical value of the work unclear.
-
The paper is hard to read and follow. The writing is very dense. Many terms appear without explanation, such as "Holm20, Smith, v0.2" as well as "original" vs "new" families, which makes the abstract and introduction hard to follow on a first read. The sentences/findings contain caveats, so the main takeaway message gets lost. The main text frequently keeps referring to the appendix instead of showing some numbers, which makes it a hard read. I would encourage the authors to rewrite the paper more simply.
Questions for authors
-
Why is corrected max direct the primary selector when full CSLS direct and corrected LOO score higher on most conditions, for both localization (table 1) and S (table 3)?
-
Does any combination of the position prior with a query-dependent score exceed the prior alone?
-
All fixed-bank conditions use the same 120 industrial questions. Would a same-domain held-out bank behave differently? This would clarify whether the correction captures generic hubness or domain mismatch.
-
What role does the v0.2 deletion audit play in the paper’s central argument?