acceptodds
Under review as a conference paper at ICLR 2027

VAR: Mining Visual Anchors from Vision Encoders for Entity Retrieval in Knowledge-Based Visual Question Answering

Abstract

In Knowledge-Based Visual Question Answering (KB-VQA), correct answer generation relies on the evidence contained in the retrieved gold document. Therefore, accurately retrieving the gold document is critical. This requires matching the question to entities in the knowledge base. However, standard vision encoders often underemphasize subtle, entity-discriminative cues, entangling them with irrelevant background noise in the final global representation. While recent methods address this via query-side region cropping or knowledge-base-side captioning, they either rely on brittle explicit region selection or incur substantial preprocessing overhead. Moreover, existing verification strategies often rely on binary relevance decisions, discarding the relative strength of visual support among candidates. In this paper, we reveal an overlooked phenomenon: fine-grained, entity-discriminative semantics already persist in high-level token representations, merely underexpressed during global aggregation. We conceptualize these compact, retrieval-critical token subsets as Visual Anchors. Based on this insight, we propose VAR (Visual Anchors for Entity Retrieval), a fine-grained entity retrieval framework. VAR first mines visual anchors and uses them to guide [CLS] updates through the remaining frozen encoder layers, strengthening their contribution to the retrieval representation without pixel-space region extraction or knowledge-base index reconstruction. To further improve gold-document ranking, VAR leverages concise visual-attribute descriptions from top-ranked candidates. Beyond binary verification, VAR uses a VLM calibrator to estimate continuous visual support and trains a lightweight gate to jointly calibrate the relative visual support of a candidate set against their retrieval confidence for Top-1 document correction. Extensive experiments demonstrate that VAR achieves state-of-the-art performance on E-VQA, InfoSeek, and OK-VQA, showing the effectiveness and efficiency of our method. Code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Knowledge-based visual question answer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cqnUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigated entity retrieval in the KB-VQA field and found evidence that the detailed distinctions relevant to the gold entity already existed in the intermediate encoder representations but were not fully expressed in the global representation. This discriminative evidence could be represented by a few groups of high-level token embeddings, which they called Visual Anchors. With Visual Anchors, the authors proposed the VAR framework, where Visual Anchor Mining (VAM) selects the candidate Visual Anchors and uses these anchors to update the [CLS] token through the remaining frozen layers, generating query embeddings to retrieve the Top-K documents, whereas Description-Grounded Evidence Calibration (DEC) calibrates the retrieval ranks with visual-support scores and retrieval confidence. The authors provide experiments that support their observations regarding underutilized information in intermediate encoder representations and demonstrate that the VAR framework improves visual entity retrieval in the KB-VQA field.

Critical strengths
  1. Pointed out that the high-level visual representations themselves contain subtle clues to match the gold documents. This is a valuable observation (with empirical support from the Oracle experiments) and suggests that current methods have not fully exploited the vision encoder's potential.
  2. The proposed VAM updates the [CLS] token by focusing attention on anchors rather than simply cropping the corresponding image regions. As the authors claimed, this could help preserve more contextual information (compared to methods relying on explicit region selection). The VAR framework achieves competitive retrieval and question-answering performance across multiple KB-VQA benchmarks.
Critical weaknesses
  1. The effectiveness of the Anchor Selector is not sufficiently evaluated. For a trained selector to be effective, the retrieval utility of Visual Anchors should be predictable from their features rather than arbitrary. However, the paper does not directly evaluate how accurately the selector learns to identify retrieval-effective anchors. The current Ablation Study mainly analyzes the two components, VAM and DEC, and does not delve into VAM. Yet the improvement of VAM comes from Anchor Selection, multi-Anchor retrieval, and score aggregation. The paper lacks a direct comparison of the trained selector, random selector, and oracle selector, so it is not easy to clearly distinguish the learning benefits of the selector from the benefits of multiple spatial retrievals. DEC further improves Top-1 ranking, but it remains unclear whether the residual errors of VAM are due to the limitations of the selector or because the selected Visual Anchors still lack some discriminative information.
  2. The semantic interpretation of the Visual Anchor is not sufficiently clarified. The authors define Visual Anchors as groups of high-dimensional intermediate-layer token embeddings, but the extent to which these embeddings represent localized versus global information is not demonstrated in detail. The authors use bounding boxes in the original image to represent Visual Anchors. Although the token positions correspond to these boxes, this is somewhat misleading; that is, it makes people think that the details inside the boxes alone account for the high-level features. However, high-dimensional embeddings also include information from surrounding regions and even global image features, which may also be reflected in certain Visual Anchors. Moreover, according to Appendix C.3, the oracle retrieval performance from Blocks 1–28 is very similar. This might suggest that Visual Anchors act as stable spatial attention supports across layers, rather than relying exclusively on discriminative semantic information present in high-level representations
Questions for authors
  1. Could you please explain more clearly the exact semantic meaning of your proposed Visual Anchor? It is visualized as blocks in the original image, but in the paper, it is actually a group of higher-level token embeddings. To what extent do you think these anchors represent globally contextualized information versus local visual information?
  2. Appendix C.3 explains the choice of Block 28 based on the performance–efficiency trade-off. However, have you evaluated whether the trained selector performs similarly when applied at different intervention depths? Since oracle performance for Blocks 1–28 is almost the same, does that mean that Visual Anchors primarily represent stable spatial attention supports rather than the higher-dimensional discriminative information? Could you further clarify the role of semantic information at different encoder depths in anchor-conditioned retrieval?
  3. How well does the trained selector perform compared to the oracle selector? Could you provide a direct comparison under the same evaluation setting?
  4. Could you compare the trained selector against a random selector under the same retrieval budget? This would help distinguish the benefits of learned anchor selection from those of multiple spatially conditioned retrieval attempts.
5 comments
AuthorsOpenReview ↗

Response to Reviewer cqnU [1/4]

Thank you for your feedback. Your questions are insightful and valuable for further research. We address your concerns point by point below.

Responses to Weaknesses

W1: Insufficient evaluation of the anchor selector's effectiveness.

R: Your questions are crucial. We acknowledge that we do not report the accuracy with which the Visual Anchor Selector identifies effective retrieval anchors. We address this in two parts below.

(1) How effective is the Visual Anchor Selector at selecting anchors?

Following your suggestion, we compare the trained selector with oracle, random, and uniform selectors under the same budget, as shown in Figure R1 https://anonymous.4open.science/r/reply-B420/. The oracle follows our data construction procedure: exhaustively evaluating all 64 windows and reporting the retrieval performance of the best window.

The results show that the Visual Anchor Selector outperforms random and uniform selection across all window budgets and recall cutoffs. This indicates that windows differ in retrieval value and that learned selection more consistently identifies effective visual evidence. The advantage is particularly pronounced under small window budgets; even with eight windows, the Selector still leads by approximately 3.5 percentage points in Recall@1.

Random and uniform selection also improve as the number of windows increases, because additional anchors provide more spatially conditioned retrieval attempts, allowing the union candidate pool to capture useful information distributed across different regions. The Selector consistently performs better, indicating that its gains arise not only from more retrieval attempts but also from effectively selecting more informative windows.

The remaining gap between the Selector and Oracle indicates room for improvement in individual anchor selection and further supports the need to aggregate complementary evidence from multiple anchors. After Multi-Anchor Retrieval and Scoring, the final Recall@1/5/10/20 reaches 58.4/78.7/82.9/86.4, approaching the corresponding upper bounds of 60.7/79.3/83.4/87.4 for the eight-anchor candidate pool. This indicates that the subsequent scorer effectively consolidates retrieval information from multiple windows.

(2) What causes VAM's residual errors?

They arise from both selector limitations and the lack of other discriminative information.

  • The Oracle performs post hoc selection among 64 windows using the actual retrieval rank of the gold document, whereas the learned selector must predict each window's downstream retrieval value in advance using only visual features. This inevitably introduces prediction errors relative to the Oracle.
  • When an image contains multiple entities, the selector can only assess visual salience and cannot determine which entity the question refers to. Fine-grained entity identification also requires question text and cross-modal matching against textual descriptions in candidate articles. These signals are absent from VAM's selection stage.

W2: The semantic interpretation of the Visual Anchor is not sufficiently clarified.

R: Thank you for your feedback. We acknowledge that our paper lacks an analysis of the semantics encoded in Visual Anchor token embeddings. We address this in two parts below.

(1) Do Visual Anchors encode only local information, and what do the bounding boxes represent?

We agree that the boxed visualizations in the paper may give the impression that a Visual Anchor is a local image crop or a set of isolated local features. In fact, the boxes indicate only the spatial locations of Visual Anchors. The actual Visual Anchors are groups of token embeddings within the model that have undergone global self-attention in deep layers. Thus, each token contains not only its original local image information but also aggregated global context. The CLS representation obtained from these tokens through CLS replay is the anchor-conditioned query, namely the final retrieval representation.

AuthorsOpenReview ↗

Response to Reviewer cqnU [2/4]

(2) Does similar oracle performance across layers imply that Visual Anchors are merely stable spatial attention supports?

We agree that effective spatial locations can remain stable across layers. However, stable spatial support does not imply that the underlying token content is semantically equivalent across depths, or that the anchors' contribution is purely spatial. Global contextualization is part of our interpretation of Visual Anchors.

In fact, spatial stability is expected because the object's geometric location remains unchanged, while ViT patch indices and residual connections preserve spatial identity. Self-attention nevertheless changes what is encoded at those locations. Similarly, comparable oracle performance indicates that useful windows remain available across layers; it does not establish that their features contain equivalent information or are equally accessible to a learned selector.

The experiments in Q2 distinguish these factors: depth-matched selector performance improves despite nearly unchanged oracle performance, while intervening too late reduces retrieval performance when little capacity remains for subsequent aggregation. These results support the importance of both contextualized token content and its integration into CLS. Thus, Visual Anchors combine stable spatial support with retrieval-relevant semantic information: where the evidence is can remain stable, while what the selected tokens encode and how effectively that information supports retrieval evolve across layers. Please refer to Q2 for the detailed experiments.

Responses to Questions

Q1:The exact semantic meaning of your proposed Visual Anchor; the extent to which Visual Anchors encode global and local information

R: The exact semantic meaning of your proposed Visual Anchor is: a spatially localized group of high-level token embeddings that encode local visual cues enriched with global context.

To examine how much global information these representations retain and what role the selected region plays, we conduct an additional representation analysis using the Top-1 anchor selected by the selector for each of 1,000 InfoSeek examples.

We define the following visual representations:

  • Inner CLS: The CLS representation at layer 28.
  • Anchor Replayed CLS (i.e., the Visual Anchor-conditioned query in our manuscript): The CLS representation obtained by selecting an anchor at layer 28 and replaying CLS through the remaining four layers using tokens within the anchor.
  • Full CLS: The original CLS representation at the final layer.
  • Non Anchor Replayed CLS: The CLS representation obtained by selecting an anchor at layer 28 and replaying CLS through the remaining four layers using tokens outside the anchor.
  • Raw Anchor Pool: The mean of the token embeddings within the anchor selected at layer 28.

We analyze the following:

(1) How much does a local anchor change representation similarity for retrieval?

For each sample ii and a given visual representation uiu_i, we define:

B(ui)=cos⁡(ui,gi)−16∑k=16cos⁡(ui,gπk(i)).B(u_i) = \cos(u_i,g_i) - \frac{1}{6}\sum_{k=1}^{6} \cos\bigl(u_i,g_{\pi_k(i)}\bigr).

Here:

  • gig_i is the gold embedding matched to sample ii;
  • gπk(i)g_{\pi_k(i)}, for k=1,…,6k=1,\ldots,6, are six deterministically mismatched gold embeddings.

Thus, B measures how much the similarity between a sample's visual representation uiu_i and the correct target exceeds its average similarity to incorrect targets. A larger value indicates a stronger ability to distinguish the correct target from incorrect targets.

Table R7. Comparison of different CLS representations for similarity-based retrieval of gold document embeddings.

RepresentationB
Inner CLS0.094
Non-Anchor Replayed CLS0.305
Full CLS0.306
Anchor Replayed CLS0.343

The B values for Full CLS and Non-Anchor Replayed CLS are 0.306 and 0.305, respectively, whereas reading only the selected anchor yields 0.343. Although the increase is only 0.037/0.038, this additional similarity gain translates into a substantial retrieval benefit: R@1 increases from 45.2% to 48.1%, a gain of 2.9 percentage points. We believe this similarity gain reflects discriminative local features.

AuthorsOpenReview ↗

Response to Reviewer cqnU [3/4]

(2) Is the final Anchor Replayed CLS representation structurally closer to global or local representations?

We use Linear CKA [5] to measure the similarity between the geometric structures of different representations and that of Anchor Replayed CLS. Linear CKA is a standard method for comparing neural network representations. For example, it can assess whether features from different layers of the same network are similar, or whether two independently trained networks learn similar representations. A higher CKA value indicates greater similarity between the representational structures.

Table R8. CKA similarity between different visual representations and the final Anchor Replayed CLS representation.

RepresentationCKA
Inner CLS0.765
Raw Anchor Pool0.095
Full CLS0.867
Non-Anchor Replayed CLS0.864

Inner CLS achieves a CKA of 0.765, while Full CLS and Non-Anchor Replayed CLS also show high CKA values. This indicates that the final anchor's representational structure across samples primarily builds on the global semantic structure already established in the first 28 layers.

In contrast, Raw Anchor Pool has a CKA of only 0.095. This indicates that a Visual Anchor is not simply obtained by mean pooling the selected tokens; attention interactions with CLS in the final four layers are required to produce the final retrieval representation.

Thus, our anchor selection and replay mechanism does not fundamentally restructure the overall representation space, but provides the primary discriminative signal for identifying the correct entity.

Q2: The selector's performance across intervention depths and the role of discriminative semantics versus stable spatial attention supports

R: Thank you for your question. We retrain selectors with the same architecture using the tokens and corresponding retrieval supervision at each layer and evaluate direct transfer of the final selector to other layers.

Table R9. Comparison of Visual Anchor spatial selection consistency across different intervention depths.

Depth comparisonRank-1 set Jaccard ↑Spearman correlation of 64-window rankings ↑
After-4 vs After-2891.24%0.901
After-12 vs After-2891.28%0.905
After-20 vs After-2891.41%0.909
After-24 vs After-2892.49%0.919
After-30 vs After-2886.84%0.848
After-31 vs After-2879.48%0.628
  • Rank-1 set Jaccard measures the overlap between the sets of windows that rank the gold document first at two depths. Spearman correlation of 64-window rankings measures the consistency of the retrieval-performance rankings of the 64 candidate windows across those depths. Higher values indicate more stable spatial support. Jaccard ranges from 0 to 1, where 1 indicates complete overlap and 0 indicates no overlap. Spearman correlation ranges from −1 to 1, where 1 indicates perfect agreement, −1 indicates completely reversed rankings, and 0 indicates no monotonic correlation.

Table R9 confirms that the spatial locations of effective evidence stabilize relatively early: window-ranking correlations between After-4 through After-24 and After-28 all exceed 0.90. However, these metrics only indicate consistency in “which locations are more important”; they do not imply that features at the same location contain equivalent information across different depths.

Reviewer_F6ytOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that fine-grained entity cues remain in high-level visual tokens but are underrepresented by global embeddings. It proposes VAR, which mines visual anchors for retrieval and calibrates top candidates using description-grounded visual support. Results on three KB-VQA benchmarks show state-of-the-art retrieval and answer accuracy.

Critical strengths
  • This paper reveals an interesting observation of KB-VQA.
  • The proposed methods are reasonable and effective.
  • This paper is well-written and easy to follow.
Critical weaknesses
  • Concern about the engineering-heavy pipeline design. VAM and DEC are largely independent: one performs visual-anchor retrieval, while the other uses a separately trained VLM for candidate calibration. The selector, scorer, calibrator, and gate also require separate training stages. Thus, the gains may partly result from stacking specialized components rather than a coherent unified design.
  • Concern about the scalability of selector supervision. Its labels require exhaustively replaying 64 candidate token groups and querying the full knowledge base for gold-document ranks using EVA-CLIP-8B. This costly offline supervision is tightly coupled to the retriever, and adapting VAR to another encoder or knowledge base may require regenerating all labels. The paper does not clearly quantify this preprocessing cost.
  • Limited methodological novelty of DEC. DEC largely follows a standard verify-and-rerank pipeline: it extracts candidate descriptions, obtains support–reject logits from a fine-tuned VLM, and combines them with retrieval scores through a three-parameter linear gate. Its relative calibration provides some practical value, but the paper lacks comparisons with generic VLM rerankers or simpler score-fusion baselines, making it unclear whether the gains arise from the proposed calibration design or merely from adding an additional VLM-based verification stage.
Questions for authors

Please kindly refer to the weakness section. I will consider raising score if my concerns are addressed

4 comments
AuthorsOpenReview ↗

Response to Reviewer F6yt [1/2]

We sincerely thank you for carefully reading our paper and providing constructive comments. We address your concerns point by point below.

Responses to Weaknesses

W1: Concern about the engineering aspects of the pipeline.

R: Thank you for your feedback. This is indeed a crucial issue. We acknowledge that the framework appears to contain many components. However, VAM and DEC follow a consistent rationale, and each component is essential, for the following reasons:

  1. The design complexity stems from the complexity of KB-VQA itself. Its final performance depends on multiple stages: (1) retrieving the correct knowledge-base article based on the question; (2) filtering and reranking the retrieved articles; (3) identifying passages containing the answer within the selected articles; (4) filtering and reranking these passages; and (5) generating the correct answer from the provided reference passages. Performance at each upstream stage affects downstream stages. The available data also impose constraints: the knowledge base stores the original article text and global embeddings of the images within each article. In the first stage, retrieving the correct article can only rely on computing similarities between the query-image embedding and the article-image embeddings. Incorporating textual embeddings would require rebuilding the database at substantial cost, which our method specifically aims to avoid.

  2. VAM and DEC follow a consistent rationale. VAM achieves strong retrieval performance using only visual information (coarse-grained and less precise), addressing the question: “Which articles in the knowledge base contain images similar to the query image?” DEC then extracts a small amount of textual information (fine-grained and precise) from the candidate pool as verification evidence, asking: “Judging from its short description, does this article actually correspond to the entity referred to in the question?” Combining this evidence with the preceding visual information further narrows the candidate pool. Together, these modules form a strong retriever whose ultimate goal is to identify the correct entity article for the question and reduce the burden on downstream tasks.

  3. The components within VAM and DEC are essential. In VAM, the selector identifies a few clean, informative anchors, and a scorer is needed to aggregate their information and assign a single score to each candidate article. The VLM calibrator uses visual descriptions in text to verify whether these articles actually match the entity referred to in the question, and a gate is needed to combine the scores. The key insight behind DEC's gate is that, when combining scores, it evaluates the continuous logit differences between challengers and the frontend winner through a lightweight gate, rather than simple hyperparameter weighting. This mechanism is absent from the rerankers of similar methods in other papers. Please also see our response to reviewer kZEx's W2, where we provide additional experiments demonstrating the effect of this relative calibration.

In future work, we will also consider reinforcement learning and other approaches to develop a new framework that unifies these modules and reduces the number of training stages.

W2: Concern about the scalability of selector supervision.

R: We fully understand your concern. In fact, we use only 6K examples to generate selector supervision, accounting for just 0.66% of InfoSeek's full training set and 2.83% of E-VQA's, respectively. Our measurements show that the entire supervision data construction process takes approximately 30 minutes on four A800 GPUs, or approximately 2 hours on a single A800.

Therefore, even if transferring to another encoder or knowledge base requires regenerating labels, this process is performed only once, uses a small sample set, and is readily parallelized. Subsequently, only an 11.76M-parameter selector needs to be trained. Compared with methods requiring database re-encoding (WikiSeeker [2] and OMGM [3]; the InfoSeek knowledge base contains 100K articles and 290K image vectors, while E-VQA contains 2M articles and 5.3M image vectors) or reinforcement learning (Wiki-PRF [1], requiring over 10 hours of training), our approach keeps the overall transfer cost manageable.

AuthorsOpenReview ↗

Response to Reviewer F6yt [2/2]

W3: Concerns about DEC's novelty.

R: Thank you for your question. This is indeed a crucial issue. DEC has two key distinctive features. First, it improves reranking by introducing textual information from only a small number of candidates, rather than directly using generic cross-modal relevance scores. Second, when combining retrieval and reranking scores (Z and S), the relative calibration gate uses the VAM retrieval winner as its reference and fixes the winner's score at zero to learn the logit differences between challengers and the frontend winner, enabling correct keep/switch decisions rather than simple hyperparameter weighting. This mechanism is absent from the rerankers of similar methods in other papers.

Following your suggestion, we add experiments with Q-Former [4] as a generic cross-modal reranker and with simple weighted fusion. The baseline in Table R6 represents the retrieval results after VAM's Multi-Anchor Retrieval and Scoring. All training uses the same 1K training examples (consistent with the manuscript). For “trained gate fusion with Q-Former,” we directly replace Z with the Q-Former score and train the gate using exactly the same procedure.

Table R6. Comparison of a generic cross-modal reranker with DEC's reranking and calibration strategies.

MethodInfoSeek R@1Correction rateDisruption rate
Baseline58.4––
Baseline + Q-Former43.310.525.6
Baseline + fixed-weight fusion with Q-Former (weight = 0.5, selected through a parameter sweep)61.27.14.3
Baseline + trained gate fusion with Q-Former63.26.61.8
Baseline + DEC VLM62.212.68.8
Baseline + fixed-weight fusion with DEC VLM (weight = 0.5, selected through a parameter sweep)62.45.51.5
Baseline + trained gate fusion with DEC VLM66.19.31.6

These results show that the performance gains do not come solely from adding an extra VLM.

First, we acknowledge that our VLM makes a substantial contribution to DEC. Without the gate, the description-grounded VLM achieves reranking results of 62.2/62.4, outperforming the corresponding Q-Former baselines of 43.3/61.2. This indicates that description-grounded evidence provides more effective verification signals.

However, holding the reranker fixed, the relative gate improves over fixed-weight score fusion by 2% and 3.7% for Q-Former and DEC VLM, respectively, while achieving high correction rates and low disruption rates. These results demonstrate the effectiveness of gate calibration. Therefore, DEC's gains arise from combining a description-grounded VLM with relative calibration and cannot be attributed solely to adding a generic VLM reranking stage.

References

[2] Yingjian Zhu et al. WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering. Findings of ACL 2026, pp. 5429–5449, 2026.

[3] Wei Yang et al. OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval. ACL 2025, pp. 24545–24563, 2025.

[4] Yibin Yan and Weidi Xie. EchoSight: Advancing Visual-Language Models with Wiki Knowledge. Findings of EMNLP 2024, pp. 1538–1551, 2024.

Dear Reviewer F6yt,

Thank you again for your careful review and constructive suggestions. As the discussion period draws to a close, we would be grateful to hear whether our responses have helped address your concerns. We would be happy to provide further clarification on any remaining points.

Yours sincerely,

The Authors

Thank you for the detailed response and additional experiments. My concerns regarding W2 and W3 have been adequately addressed. While my concern about W1 remains, I appreciate the authors’ clarification and will raise my rating accordingly.

Reviewer_kZExOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes (VAR), a framework for entity retrieval in knowledge-based visual question answering (VQA) task. This framework contains two parts: Visual Anchor Mining (VAM) and Description-grounded Evidence Calibration (DEC). First, VAM selects spatially coherent groups of high-level visual tokens, then updates the [CLS] representation token through the remaining frozen encoder layers to create multiple complementary retrieval queries. Second, DEC combines visual-support scores from a fine-tuned VLM with retrieval confidence to refine the top-ranked document. Comprehensive experiments on InfoSeek, E-VQA, and OK-VQA demonstrate the efficiency of the proposed method.

Critical strengths
  1. This paper is well-written and very easy to follow. The motivation and method are described clearly.
  2. The method is useful and technically interesting for retrieval based VQA.
  3. The experiments are comprehensive.
Critical weaknesses
  1. According to the experiments, we cannot isolate the contribution of the proposed anchor mechanism. VAM simultaneously introduces learned anchor selection, multiple retrieval queries, a larger candidate pool, and a trained aggregation model. The component ablation therefore establishes the effectiveness of the combined module but does not distinguish the benefits of anchor selection and [CLS] replay from those of additional retrieval and learned reranking. I would suggest to have a budget-matched comparisons with random or uniformly selected windows and simpler feature aggregation.
  2. I am not sure whether I understand it correctly that DEC’s relative calibration seems less distinctive than the author claims. From my understanding, defining Fi=wzZi+wslog⁡SiF_i=w_zZ_i+w_s\log S_i, the proposed gate becomes Hi=Fi−F∗+bH_i=F_i-F_*+b. Its decision is therefore equivalent to weighted score fusion with a challenger-specific bias. So, when b=0b=0, it reduces to ordinary weighted fusion. Table 6 compares visual-support-only ranking against a method that additionally uses retrieval confidence, so it does not isolate the claimed benefit of relative calibration.

I am open to discuss during rebuttal and willing to change my score if authors can provide convincing response to my questions and concerns.

Questions for authors
  1. I have a question about InfoSeek score. The official metric uses the harmonic mean of the Unseen-Q and Unseen-E scores. However, Table 3 reports 48.1 and 46.6 for VAR with Qwen2.5-7B, whose harmonic mean is approximately 47.34, rather than the reported 47.0. This discrepancy is larger than can be explained by conventional rounding and should be reconciled with the evaluation implementation. Can authors explain the difference?
5 comments
AuthorsOpenReview ↗

Response to Reviewer kZEx [1/4]

We sincerely thank you for carefully reading our paper and providing constructive comments. We address your concerns point by point below.

Responses to Weaknesses

W1: Insufficient evidence for the effectiveness of VAM's core components (particularly the Visual Anchor Selector and the CLS replay mechanism).

R: Thank you very much for your feedback. To demonstrate the effectiveness of the Visual Anchor Selector and the CLS replay mechanism, we follow your suggestion and provide additional ablation and comparative experiments on VAM:

(1) Table R1 shows that constructing a new retrieval query using the anchor selected by the Visual Anchor Selector increases R@1 from 45.2 to 48.1. Building on this, aggregating information from multiple anchors through Multi-Anchor Retrieval and Scoring further increases R@1 to 58.4 (the need for multiple anchors and the rationale for the selected number of anchors are explained in Lines 232–234 and 429–453 of the paper). These results demonstrate that each component of VAM plays a critical role.

Table R1. Ablation study of VAM components.

ComponentsInfoSeek R@1InfoSeek R@5InfoSeek R@10InfoSeek R@20E-VQA R@1E-VQA R@5E-VQA R@10E-VQA R@20
Q-Former Retrieval (InfoSeek results reported in Table 12, p. 19)17.146.862.377.428.752.962.868.8
Full CLS Retrieval (original CLS used as the query) (InfoSeek results reported in Table 5, p. 9)45.268.774.879.914.732.842.050.4
Single Anchor Retrieval48.171.877.582.230.654.962.864.8
Multi-Anchor Retrieval and Scoring (InfoSeek results reported in Table 12, p. 19)58.478.782.986.440.158.566.769.0
Multi-Anchor Retrieval and Scoring fused with the Q-Former score (reported in Table 2, p. 7, and Table 5, p. 9)61.280.784.487.542.061.167.869.9

(2) Table R2 shows that our CLS replay method substantially outperforms the first two naive methods, demonstrating its effectiveness. This effectiveness mainly stems from preserving the original inference process, allowing it to focus on individual tokens while maintaining the original CLS distribution.

Table R2. Retrieval performance using different methods to aggregate the tokens within the Top-1 anchor into a query, with the Visual Anchor Selector held fixed.

Aggregation methodInfoSeek R@1InfoSeek R@5
Token Addition and Normalization1.433.29
Token Pruning31.652.3
Token Attention Reconcentration (ours)48.171.8
  • Token Pruning: tokens outside the anchor are removed at the selection layer; the remaining tokens and CLS are updated in the replay layers.
  • Token Attention Reconcentration: no tokens are removed; attention is redistributed in the replay layers, and CLS is updated using the frozen tokens within the anchor.
AuthorsOpenReview ↗

Response to Reviewer kZEx [2/4]

(3) Table R3 shows that the retrieval candidate pool obtained from the Top-8 anchors selected by our Visual Anchor Selector is substantially better than those obtained through random and uniform selection. This demonstrates the effectiveness of the trained Visual Anchor Selector.

Table R3. Candidate pool quality for eight anchor-conditioned queries. Recall@K denotes the proportion of samples for which at least one of the eight queries generated for that sample retrieves the gold document within its Top-K results.

Anchor selection methodInfoSeek Recall@1InfoSeek Recall@5InfoSeek Recall@10InfoSeek Recall@20
Random selection of eight windows57.22 ± 0.03%76.63 ± 0.04%81.63 ± 0.05%85.59 ± 0.05%
Uniform selection of eight windows (four central and four peripheral windows on an 8 × 8 grid)57.16%76.57%81.62%85.61%
Top-8 anchors selected by the trained Visual Anchor Selector60.72%79.13%83.42%87.36%

(4) Table R4 presents VAM's retrieval performance with different numbers of candidate articles, Top-KDK_D (Line 251, Eq. 6). Clearly, varying Top-KDK_D has little effect on either retrieval performance or computation time, making it a robust hyperparameter. These results therefore demonstrate that VAM's core design components play the primary role in retrieval.

Table R4. VAM retrieval results with different values of Top-KDK_D.

Top-KDK_DInfoSeek Recall@1InfoSeek Recall@5InfoSeek Recall@10InfoSeek Recall@20Similarity computation time/queryScorer computation time/query
2058.52%78.72%82.26%86.15%0.0045 ms0.2961 ms
3558.68%78.70%82.54%86.39%0.0078 ms0.5168 ms
5058.43%78.65%82.92%86.45%0.0111 ms0.7371 ms
6558.71%78.59%82.58%86.35%0.0144 ms0.9564 ms
AuthorsOpenReview ↗

Response to kZEx [3/4]

W2: Concerns about the distinctiveness of DEC's relative calibration.

R: Thank you for your question. This is indeed a crucial issue that we do not explain clearly in the paper. DEC has two key distinctive features. First, it improves reranking by introducing textual information from only a small number of candidates. Second, when combining retrieval and reranking scores (Z and S), its relative calibration uses a lightweight gate to measure the continuous logit differences between challengers and the frontend winner, rather than simple hyperparameter weighting. This mechanism is absent from the rerankers of similar methods in other papers. Below, we focus on the relative calibration issue you raise. Keeping the Z and S scores for each sample fixed, we compare using either Z or S alone with jointly using Z and S through different retrained gates:

Table R5. Retrieval results under different DEC gate settings.

Table R5.1. Retrieval results on InfoSeek using S or Z alone.

MethodIDwbInfoSeek R@1
Frontend retrieval ranking using only SA××61.2
Reranking using only visual support Z after frontend retrieval rankingB××63.3

Table R5.2. Retrieval results on InfoSeek jointly using S and Z after frontend retrieval.

IDwbSet the winner's H to zeroSubtract the winner's Z and S from the challengers' Z and SInfoSeek R@1Correction rate ↑ (relative to A)Disruption rate ↓ (relative to A)
C√√/×××67.17.31.4
D√×√×62.73.01.4
E√×√√67.17.31.4
F√√√×62.86.14.5
G√√√√68.58.81.5

Table R5.3. Retrieval results using Weight-Sweep Fusion of S and Z after frontend retrieval.

InfoSeek R@1E-VQA R@1
64.238.1

(1) Is the gate formula equivalent to simple weighting?

In our design, the gate is not equivalent to simple weighting.

The training loss is the listwise softmax cross-entropy loss (Eq. 24, p. 15). Its objective is to assign the highest score to the gold document in each batch (which contains all candidates (the winner and challengers) for a single example). Adding or subtracting the same constant from all candidates in a batch introduces a common shift and therefore leaves the training objective unchanged, as illustrated by settings C and E. The b learned in C is on the order of 10−1610^{-16}, effectively zero, further demonstrating that b has no effect in this setting.

However, in our design G, when the winner's total score H is fixed to zero and b is present, b is added only to the challengers and not to the winner. Here, b not a common shift; it becomes a learnable keep/switch decision threshold. In this case, the winner's Z and S must be subtracted from the challengers' Z and S to express the challengers in a relative coordinate system with the winner as the origin. Otherwise, the decision process becomes inconsistent, as in settings D and F. Our full model G uses:

H∗=0,Hi=wz(Zi−Z∗)+ws(log⁡Si−log⁡S∗)+b,H^{\ast}=0, \quad H_i=w_z(Z_i-Z^{\ast})+w_s(\log S_i-\log S^{\ast})+b,

where the learned value of b is on the order of 10−210^{-2}, substantially larger than the 10−1610^{-16} value in C. This also indicates that our b plays an active role rather than being zero as you mention. The effectiveness of this mechanism can therefore be summarized as: Relative features eliminate meaningless shifts in absolute scores; fixing the winner's score to zero establishes an explicit KEEP reference; and the challenger-specific bias b learns the evidence threshold required to overturn the frontend winner. These three elements form an integrated mechanism. Relative differences alone merely change the coordinate system, and adding b alone merely introduces a common shift. Only the combination of “relative differences + a winner score fixed at zero + a challenger threshold” forms a true calibration mechanism.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks by Authors

Dear Reviewers and Area Chair,

We sincerely thank you for your careful evaluation and constructive feedback. We appreciate the recognition of VAR’s clear motivation (kZEx), valuable observations (F6yt, cqnU), and comprehensive evaluation (kZEx). Our key additions are summarized below:

  • Effectiveness of Visual Anchor Mining (kZEx, cqnU): Budget-matched comparisons and aggregation ablations validate the effectiveness of learned selection and CLS replay.

  • Semantic interpretation of Visual Anchors (cqnU): We clarified that Visual Anchors are groups of high-level tokens that have undergone global self-attention, while the image boxes indicate only their spatial locations. Representation and cross-depth analyses demonstrate that stable spatial locations of effective evidence do not imply that features at different depths are equally accessible to learned selection or equally useful for retrieval.

  • Effectiveness of DEC calibration (kZEx, F6yt): Controlled ablations show that the challenger-specific bias improves R@1 by 1.4 percentage points over learned weighted fusion. Further comparisons confirm the gate’s effectiveness with both a generic reranker and our description-grounded VLM.

  • Framework coherence and manageable supervision cost (F6yt): We explained the complementary roles of VAM and DEC. For VAM supervision construction, generating selector labels for 6K examples takes approximately 30 minutes on four A800 GPUs. The overall cost is manageable.

  • Clarification of metric aggregation (kZEx): The reported InfoSeek score of 47.0 is computed using sample-averaged accuracy; the harmonic mean of the reported split scores is approximately 47.34. Both aggregation rules and corresponding results will be explicitly reported in the revision.

We will incorporate the core findings into the revised manuscript and release our implementation to support verification and reproducibility.

Yours sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.