- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors investigated entity retrieval in the KB-VQA field and found evidence that the detailed distinctions relevant to the gold entity already existed in the intermediate encoder representations but were not fully expressed in the global representation. This discriminative evidence could be represented by a few groups of high-level token embeddings, which they called Visual Anchors. With Visual Anchors, the authors proposed the VAR framework, where Visual Anchor Mining (VAM) selects the candidate Visual Anchors and uses these anchors to update the [CLS] token through the remaining frozen layers, generating query embeddings to retrieve the Top-K documents, whereas Description-Grounded Evidence Calibration (DEC) calibrates the retrieval ranks with visual-support scores and retrieval confidence. The authors provide experiments that support their observations regarding underutilized information in intermediate encoder representations and demonstrate that the VAR framework improves visual entity retrieval in the KB-VQA field.
Critical strengths
- Pointed out that the high-level visual representations themselves contain subtle clues to match the gold documents. This is a valuable observation (with empirical support from the Oracle experiments) and suggests that current methods have not fully exploited the vision encoder's potential.
- The proposed VAM updates the [CLS] token by focusing attention on anchors rather than simply cropping the corresponding image regions. As the authors claimed, this could help preserve more contextual information (compared to methods relying on explicit region selection). The VAR framework achieves competitive retrieval and question-answering performance across multiple KB-VQA benchmarks.
Critical weaknesses
- The effectiveness of the Anchor Selector is not sufficiently evaluated. For a trained selector to be effective, the retrieval utility of Visual Anchors should be predictable from their features rather than arbitrary. However, the paper does not directly evaluate how accurately the selector learns to identify retrieval-effective anchors. The current Ablation Study mainly analyzes the two components, VAM and DEC, and does not delve into VAM. Yet the improvement of VAM comes from Anchor Selection, multi-Anchor retrieval, and score aggregation. The paper lacks a direct comparison of the trained selector, random selector, and oracle selector, so it is not easy to clearly distinguish the learning benefits of the selector from the benefits of multiple spatial retrievals. DEC further improves Top-1 ranking, but it remains unclear whether the residual errors of VAM are due to the limitations of the selector or because the selected Visual Anchors still lack some discriminative information.
- The semantic interpretation of the Visual Anchor is not sufficiently clarified. The authors define Visual Anchors as groups of high-dimensional intermediate-layer token embeddings, but the extent to which these embeddings represent localized versus global information is not demonstrated in detail. The authors use bounding boxes in the original image to represent Visual Anchors. Although the token positions correspond to these boxes, this is somewhat misleading; that is, it makes people think that the details inside the boxes alone account for the high-level features. However, high-dimensional embeddings also include information from surrounding regions and even global image features, which may also be reflected in certain Visual Anchors. Moreover, according to Appendix C.3, the oracle retrieval performance from Blocks 1–28 is very similar. This might suggest that Visual Anchors act as stable spatial attention supports across layers, rather than relying exclusively on discriminative semantic information present in high-level representations
Questions for authors
- Could you please explain more clearly the exact semantic meaning of your proposed Visual Anchor? It is visualized as blocks in the original image, but in the paper, it is actually a group of higher-level token embeddings. To what extent do you think these anchors represent globally contextualized information versus local visual information?
- Appendix C.3 explains the choice of Block 28 based on the performance–efficiency trade-off. However, have you evaluated whether the trained selector performs similarly when applied at different intervention depths? Since oracle performance for Blocks 1–28 is almost the same, does that mean that Visual Anchors primarily represent stable spatial attention supports rather than the higher-dimensional discriminative information? Could you further clarify the role of semantic information at different encoder depths in anchor-conditioned retrieval?
- How well does the trained selector perform compared to the oracle selector? Could you provide a direct comparison under the same evaluation setting?
- Could you compare the trained selector against a random selector under the same retrieval budget? This would help distinguish the benefits of learned anchor selection from those of multiple spatially conditioned retrieval attempts.
5 comments
Response to Reviewer cqnU [2/4]
(2) Does similar oracle performance across layers imply that Visual Anchors are merely stable spatial attention supports?
We agree that effective spatial locations can remain stable across layers. However, stable spatial support does not imply that the underlying token content is semantically equivalent across depths, or that the anchors' contribution is purely spatial. Global contextualization is part of our interpretation of Visual Anchors.
In fact, spatial stability is expected because the object's geometric location remains unchanged, while ViT patch indices and residual connections preserve spatial identity. Self-attention nevertheless changes what is encoded at those locations. Similarly, comparable oracle performance indicates that useful windows remain available across layers; it does not establish that their features contain equivalent information or are equally accessible to a learned selector.
The experiments in Q2 distinguish these factors: depth-matched selector performance improves despite nearly unchanged oracle performance, while intervening too late reduces retrieval performance when little capacity remains for subsequent aggregation. These results support the importance of both contextualized token content and its integration into CLS. Thus, Visual Anchors combine stable spatial support with retrieval-relevant semantic information: where the evidence is can remain stable, while what the selected tokens encode and how effectively that information supports retrieval evolve across layers. Please refer to Q2 for the detailed experiments.
Responses to Questions
Q1:The exact semantic meaning of your proposed Visual Anchor; the extent to which Visual Anchors encode global and local information
R: The exact semantic meaning of your proposed Visual Anchor is: a spatially localized group of high-level token embeddings that encode local visual cues enriched with global context.
To examine how much global information these representations retain and what role the selected region plays, we conduct an additional representation analysis using the Top-1 anchor selected by the selector for each of 1,000 InfoSeek examples.
We define the following visual representations:
- Inner CLS: The CLS representation at layer 28.
- Anchor Replayed CLS (i.e., the Visual Anchor-conditioned query in our manuscript): The CLS representation obtained by selecting an anchor at layer 28 and replaying CLS through the remaining four layers using tokens within the anchor.
- Full CLS: The original CLS representation at the final layer.
- Non Anchor Replayed CLS: The CLS representation obtained by selecting an anchor at layer 28 and replaying CLS through the remaining four layers using tokens outside the anchor.
- Raw Anchor Pool: The mean of the token embeddings within the anchor selected at layer 28.
We analyze the following:
(1) How much does a local anchor change representation similarity for retrieval?
For each sample and a given visual representation , we define:
Here:
- is the gold embedding matched to sample ;
- , for , are six deterministically mismatched gold embeddings.
Thus, B measures how much the similarity between a sample's visual representation and the correct target exceeds its average similarity to incorrect targets. A larger value indicates a stronger ability to distinguish the correct target from incorrect targets.
Table R7. Comparison of different CLS representations for similarity-based retrieval of gold document embeddings.
| Representation | B |
|---|---|
| Inner CLS | 0.094 |
| Non-Anchor Replayed CLS | 0.305 |
| Full CLS | 0.306 |
| Anchor Replayed CLS | 0.343 |
The B values for Full CLS and Non-Anchor Replayed CLS are 0.306 and 0.305, respectively, whereas reading only the selected anchor yields 0.343. Although the increase is only 0.037/0.038, this additional similarity gain translates into a substantial retrieval benefit: R@1 increases from 45.2% to 48.1%, a gain of 2.9 percentage points. We believe this similarity gain reflects discriminative local features.
Response to Reviewer cqnU [3/4]
(2) Is the final Anchor Replayed CLS representation structurally closer to global or local representations?
We use Linear CKA [5] to measure the similarity between the geometric structures of different representations and that of Anchor Replayed CLS. Linear CKA is a standard method for comparing neural network representations. For example, it can assess whether features from different layers of the same network are similar, or whether two independently trained networks learn similar representations. A higher CKA value indicates greater similarity between the representational structures.
Table R8. CKA similarity between different visual representations and the final Anchor Replayed CLS representation.
| Representation | CKA |
|---|---|
| Inner CLS | 0.765 |
| Raw Anchor Pool | 0.095 |
| Full CLS | 0.867 |
| Non-Anchor Replayed CLS | 0.864 |
Inner CLS achieves a CKA of 0.765, while Full CLS and Non-Anchor Replayed CLS also show high CKA values. This indicates that the final anchor's representational structure across samples primarily builds on the global semantic structure already established in the first 28 layers.
In contrast, Raw Anchor Pool has a CKA of only 0.095. This indicates that a Visual Anchor is not simply obtained by mean pooling the selected tokens; attention interactions with CLS in the final four layers are required to produce the final retrieval representation.
Thus, our anchor selection and replay mechanism does not fundamentally restructure the overall representation space, but provides the primary discriminative signal for identifying the correct entity.
Q2: The selector's performance across intervention depths and the role of discriminative semantics versus stable spatial attention supports
R: Thank you for your question. We retrain selectors with the same architecture using the tokens and corresponding retrieval supervision at each layer and evaluate direct transfer of the final selector to other layers.
Table R9. Comparison of Visual Anchor spatial selection consistency across different intervention depths.
| Depth comparison | Rank-1 set Jaccard ↑ | Spearman correlation of 64-window rankings ↑ |
|---|---|---|
| After-4 vs After-28 | 91.24% | 0.901 |
| After-12 vs After-28 | 91.28% | 0.905 |
| After-20 vs After-28 | 91.41% | 0.909 |
| After-24 vs After-28 | 92.49% | 0.919 |
| After-30 vs After-28 | 86.84% | 0.848 |
| After-31 vs After-28 | 79.48% | 0.628 |
- Rank-1 set Jaccard measures the overlap between the sets of windows that rank the gold document first at two depths. Spearman correlation of 64-window rankings measures the consistency of the retrieval-performance rankings of the 64 candidate windows across those depths. Higher values indicate more stable spatial support. Jaccard ranges from 0 to 1, where 1 indicates complete overlap and 0 indicates no overlap. Spearman correlation ranges from −1 to 1, where 1 indicates perfect agreement, −1 indicates completely reversed rankings, and 0 indicates no monotonic correlation.
Table R9 confirms that the spatial locations of effective evidence stabilize relatively early: window-ranking correlations between After-4 through After-24 and After-28 all exceed 0.90. However, these metrics only indicate consistency in “which locations are more important”; they do not imply that features at the same location contain equivalent information across different depths.
Response to Reviewer cqnU [1/4]
Thank you for your feedback. Your questions are insightful and valuable for further research. We address your concerns point by point below.
Responses to Weaknesses
W1: Insufficient evaluation of the anchor selector's effectiveness.
R: Your questions are crucial. We acknowledge that we do not report the accuracy with which the Visual Anchor Selector identifies effective retrieval anchors. We address this in two parts below.
(1) How effective is the Visual Anchor Selector at selecting anchors?
Following your suggestion, we compare the trained selector with oracle, random, and uniform selectors under the same budget, as shown in Figure R1 https://anonymous.4open.science/r/reply-B420/. The oracle follows our data construction procedure: exhaustively evaluating all 64 windows and reporting the retrieval performance of the best window.
The results show that the Visual Anchor Selector outperforms random and uniform selection across all window budgets and recall cutoffs. This indicates that windows differ in retrieval value and that learned selection more consistently identifies effective visual evidence. The advantage is particularly pronounced under small window budgets; even with eight windows, the Selector still leads by approximately 3.5 percentage points in Recall@1.
Random and uniform selection also improve as the number of windows increases, because additional anchors provide more spatially conditioned retrieval attempts, allowing the union candidate pool to capture useful information distributed across different regions. The Selector consistently performs better, indicating that its gains arise not only from more retrieval attempts but also from effectively selecting more informative windows.
The remaining gap between the Selector and Oracle indicates room for improvement in individual anchor selection and further supports the need to aggregate complementary evidence from multiple anchors. After Multi-Anchor Retrieval and Scoring, the final Recall@1/5/10/20 reaches 58.4/78.7/82.9/86.4, approaching the corresponding upper bounds of 60.7/79.3/83.4/87.4 for the eight-anchor candidate pool. This indicates that the subsequent scorer effectively consolidates retrieval information from multiple windows.
(2) What causes VAM's residual errors?
They arise from both selector limitations and the lack of other discriminative information.
W2: The semantic interpretation of the Visual Anchor is not sufficiently clarified.
R: Thank you for your feedback. We acknowledge that our paper lacks an analysis of the semantics encoded in Visual Anchor token embeddings. We address this in two parts below.
(1) Do Visual Anchors encode only local information, and what do the bounding boxes represent?
We agree that the boxed visualizations in the paper may give the impression that a Visual Anchor is a local image crop or a set of isolated local features. In fact, the boxes indicate only the spatial locations of Visual Anchors. The actual Visual Anchors are groups of token embeddings within the model that have undergone global self-attention in deep layers. Thus, each token contains not only its original local image information but also aggregated global context. The CLS representation obtained from these tokens through CLS replay is the anchor-conditioned query, namely the final retrieval representation.