acceptodds
Withdrawn from ICLR 2027

When Less Retrieves More: Selective Token Retention in Visual Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Can discarding visual patch vectors produce a better retrieval representation than aggregating them? We study this question with frozen encoders, full query tokens, and exact MaxSim scoring, developing the study on revisited Oxford. Retaining 39 of 196 database patches by CLS attention improves DINOv2 Medium mAP from 63.91% to 72.30%, with 80.1% fewer stored patch vectors. Holding those attention-selected anchors fixed, averaging all patches into their nearest anchors reduces mAP to 69.13%; normalizing those means reduces it further to 64.57%. Retention also outperforms nine specified hierarchical-pooling adaptations at the same exact count. An additional Paris evaluation improves Medium mAP from 85.92% to 88.23% and Hard mAP by 4.20 percentage points relative to the full representation. The attention-versus-full gain also recurs with DINOv3 on Oxford, but dense CLIP changes by -1.61 points; native DINOv3 CLS remains a strong single-vector alternative. Ordering diagnostics show asymmetric repair and damage, while CLIP demonstrates that improved mean margins against fixed competitive negatives do not suffice to explain improved mAP. The contribution is a controlled selection-aggregation contrast in DINOv2 patch retrieval, not a new compression algorithm or a universal benefit of deletion. The Paris evaluation extends the direction of the attention-versus-full gain, not the Oxford-only same-anchor finding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual retrieval, multi-vector retrieval, token selection, token pruning, representation compression, MaxSim, DINOv2, DINOv3, attention, embedding aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uTiBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is extremely hard to understand. The introduction is hard to follow and it's hard to locate the novelty. I believe large parts of the paper are completely AI-generated (I fed the introduction to Pangram which led to a 100% generated-by-ai score). I had a hard time understanding the paper's motivation, contribution and novelty.

Critical strengths

I could not locate those.

Critical weaknesses

Mentioned above.

Questions for authors

Is this paper fully AI-generated?

Reviewer_7ErFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents experiments in retrieval. The query and database images are represented as a collection of patches, in most experiments produced by DINOv3, but also CLIP. In the experiments, some number of the full 196 patches are retained, based on various attention driven criteria and retrieval performance is measured by mAP.

Critical strengths

None.

Critical weaknesses

The selective token retention is based on ground truth labels. It is thus not at all surprising, that a more compact, supervised representation improves results (over generic, self-supervised).

The results, as stated in the Conclusions, depend on the "operator and the interface", and on the dataset and the query set.

What is the message for the reader? First, reading the Limitations and the Conclusions, that it performance and behavior differs for different situations, which has limited used. The fact that on a given, fairly limited dataset, supervised token selection (retention) mechanism improves performance is of very low interest.

Nothing indicates that the results are general in nature and they hold in some form beyond the two datasets on which experiments were carried out. Methodologically, the work has "a testing on training data" flavor.

Questions for authors

Is the methodology essentially evaluating the method on training data?

Reviewer_PGoSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focus on training-free compression of database patch vectors for multi-vector image retrieval with exact MaxSim. Queries keep all 196 patches. The top 39 database patches selected by CLS attention raises DINOv2 revisited Oxford mAP from 63.91 to 72.30 on Medium and 39.08 to 49.92 on Hard. With the same 39 anchors, storing raw or unit-normalized cluster means gives 69.13 and 64.57; Ward and other controls are lower. The gain recurs for DINOv3 on Oxford and DINOv2 on Paris, but not for dense CLIP, and native DINOv3 CLS matches the 39-vector system. Ranking diagnostics and grouped exact tests over 11 landmarks are reported.

Critical strengths
  1. The same-anchor control fixes anchors, assignment and vector count and changes only the stored representatives, which isolates the retention versus aggregation contrast better than typical pruning versus merging comparisons.
  2. Reporting is complete and inference fits the data: all conditions and seeds, per-landmark and per-query results, negative CLIP and CLS results.
Critical weaknesses
  1. The core contribution is limited. This paper acknowledges that attention-based selection (DELF), fixed-anchor pruning versus pooling, and the fidelity/relevance distinction (Structural Anchor Pruning) are established. The remaining finding, retention beating same-anchor aggregation, is not distinguished from background removal. With 196x196 inputs, ROI-crop queries and full database images, CLS attention likely acts as foreground selection, while aggregation merges background patches into anchors. Top-m-only or attention-weighted aggregation, and foreground mask selection are missing. The paper also does not explain why its ordering is opposite to the cited pooling-favoring results.

  2. The main contrast is Oxford-only and its practical value is unclear. Oxford was also used for development. Paris has no same-anchor control or significance test. Native CLS matches or beats Attention39 for DINOv3 on Oxford and DINOv2 on Paris, and no standard retrieval baseline (i.e., GeM, ASMK, re-ranking) is included.

Questions for authors
  1. Do the attention-versus-full gain and the same-anchor raw-mean gap hold at higher input resolution?
  2. How large is the retention advantage when aggregation uses only top-m attention patches or weighted means?
  3. Does a simple foreground or saliency mask approach the CLS-attention result?
  4. Why does the ordering differ from Jha's work? Is the difference due to document versus natural images?
  5. What are the same-anchor results and grouped tests on Paris?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.