- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how frozen pretrained ViTs distinguish two objects of the same category. Using a rigorously controlled paired-patch test (matching category, foreground, and exact relative offset), the authors demonstrate that Q/K routing separates instances much better than value similarity. They identify an "attention address" mechanism, which is predicted by a "spatial request" (the query's projection onto a fixed quadratic key basis). Editing this request causally redirects attention toward another instance. Finally, they show that clustering patches by their spatial requests can separate repeated objects without any training.
Critical strengths
-
The paper addresses a fundamental and important question in vision about how models group features into distinct objects. It clearly points out that previous methods fail to control for simple cues like object distance and category agreement, making this study's motivation very strong.
-
The controlled patch-pair test designed in this paper is very rigorous. By matching exact distance and direction, it effectively removes confounding factors and successfully isolates the model's true ability to separate instances. The idea of using a spatial request to explain and edit attention is also very clean and logical.
-
The intervention experiments are highly convincing. For example, the model gives different addresses to identical copies of an object, and changing the spatial request directly moves where the model looks, explaining 82 percent of the actual movement. The authors also test their idea on many different models and are honest about where it fails, like in SigLIP.
-
The paper shows a useful application. By grouping patches using these spatial requests without any training, it gets a score of 0.615 on COCO, which is much better than using standard features (0.289) or a trained model called SPOT (0.356).
Critical weaknesses
-
The paper relies on labeled data from the VOC dataset to pick the best attention head and fit the spatial model. Although the authors show this choice is stable, this reliance on labels should be stated more clearly at the beginning of the paper rather than only in the appendix.
-
The analysis of the spatial request is a bit simple. It would be helpful to know if this request changes across different layers and if the model could learn it automatically instead of using a fixed mathematical formula.
-
The paper shows the idea fails in SigLIP and a supervised model, but it does not fully explain why. It would be better to provide a quantitative analysis of when the mechanism holds, such as checking if special tokens or attention sinks cause these failures.
-
The practical use is currently a bit limited. The segmentation masks are not very accurate, and using the full attention pattern actually gives a better score for grouping objects than using the spatial request.
-
The current experiments are mostly focused on human body parts. It is unclear if the mechanism and the request-changing experiment work on non-human objects and different object categories.
Questions for authors
-
Can the authors show the spatial fit results separately for foreground and background patches?
-
Does the mechanism and the successful request-changing experiment extend beyond human body parts to non-human scenes and different categories?
-
Would adding register tokens to SigLIP fix the problem and make the spatial request work again?
-
How sensitive is the grouping score to the clustering radius and the choice of the selected attention head?
3 comments
Response to Reviewer eLe1 (2/2)
Q1. Spatial fit on foreground and background patches
The spatial basis is fitted once, without labels, from the average key at each position, and the request of every patch is read through the same basis. We used annotations here only to report the result separately for object and background patches: how well the requested destination predicts where each patch actually reads (the address prediction of Figure 5).
| Address predicted by the request () | Object patches | Background patches |
|---|---|---|
| DINOv2 | .63 | .64 |
| DINOv2 with registers | .72 | .72 |
| DINOv3 | .79 | .75 |
| MAE | .73 | .75 |
All 1,414 COCO test images: 115,000 object patches and 162,000 background patches.
The same spatial basis predicts the reading of object and background patches equally well, in every backbone. The spatial structure of the keys is a property of the head, not of objects, and this is what lets one fixed basis describe where every patch asks to read.
W2. The request across layers, and without a fixed formula
Yes, the request changes across layers. Early layers already route attention spatially, but their requests do not distinguish instances. From the middle layers on, the requests of patches on different instances separate, and the separation is strongest in the late layers that the paper studies. Figure 19 shows this on two cows, and over all 144 heads the strongest instance heads lie in layers 9 to 12.
On the formula: the quadratic is not imposed on the model. It is a compact description, fitted without labels, of a spatial structure that the keys already have. Replacing it by the average key at each position, with no formula at all, gives nearly the same results (instance separation .80 against .82). We deliberately do not learn the request. The aim of the paper is to describe what the frozen network computes, and a readout that needs no training keeps that description faithful to the network. It also lets future work build on it without annotations or any trained module, for example to keep instances apart in applications where no labels are available.
W4. Practical use, and the full attention pattern
We agree on both points. The full attention pattern is the complete routing of the head, 196 numbers per patch, and it groups slightly better (.649 against .615 FG-ARI). The request is five numbers of the query and recovers most of this separation, with better masks (mIoU .429 against .403). That five spatial numbers are enough is the point of Section 5. Grouping is how we test the mechanism, and the paper states that masks remain incomplete, because background patches also form groups. Completing them with a foreground signal is the next step named in our conclusion. We are pursuing it with promising first results, but we think it deserves its own study, compared with unsupervised object discovery and object-centric methods.
W1. Labels for head selection
We agree this should be visible. The introduction states: "Labels define the test and are used to select the layers, heads, and relations on the VOC development split. Evaluation uses disjoint held-out VOC images. [...] No binding predictor is trained." Labels also choose the one radius of mean shift. The spatial basis is fitted without labels. We will repeat this at the start of Sections 3 and 5.
Q4. Sensitivity to the clustering radius and to the head
Figure 12 shows the choice of radius on the development images. On the test images the result is equally stable: requests stay between .57 and .63 FG-ARI for every radius from .30 to .50, while final features never exceed .29 at any radius. The choice of head also matters little among strong heads: nine of the ten best development heads group repeated objects better than final features.
Dear Reviewer,
As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.
If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.
Thank you again for your time and valuable feedback.
Response to Reviewer eLe1 (1/2)
We thank the reviewer for a careful and encouraging review. Each question pointed to a specific analysis, and we ran all of them. In short: attention sinks on image patches explain the SigLIP result, and registers help as the reviewer expected (W3, Q3). The request swap works in every COCO category, not only for people (W5, Q2). One fixed spatial basis fits object and background patches equally well (Q1). The request separates instances from the middle layers on, and needs no fixed formula (W2). The grouping is stable across radii and heads (Q4), and the use of labels will be stated up front (W1).
W3 and Q3. SigLIP, attention sinks and registers
Thank you for this suggestion. The intuition was right, and following it explained the SigLIP result. Many heads send part of their attention to a few tokens whatever the query asks, often called attention sinks. Darcet et al. (2024) showed that in ViTs such tokens appear on image patches of the background, and that added register tokens take over this role. When a sink is a CLS or register token, it has no position, and the paper already leaves it out of the address (Section 4.1). When it is an image patch, it pulls the address of every patch toward itself, and the request can no longer predict it. Leaving out the two most-attended patches shows this directly:
SigLIP has no CLS token to absorb the sink, and four tenths of the attention of each patch lands on two image patches. Without them, its request predicts the address again. DINOv2 with and without registers is the controlled pair the reviewer asked about: registers move the sink off the image, and the request predicts the address better. We could not retrain SigLIP with registers, but within the DINOv2 family the answer to the question is yes. Across the eight backbones, the attention that lands on image-patch sinks predicts where address prediction fails. We will add this analysis to Appendix E.2.
W5 and Q2. Beyond human body parts
The body-part test uses people because COCO annotates keypoints only for people, which lets us compare different parts of one individual. The request swap itself does not need parts, so we ran it on all 1,414 COCO test images of the paper, which include 63 categories besides people.
Attention on instance B minus attention on instance A at the selected DINOv2 head, as in Table 10.
The swap redirects the head toward the other instance in every category, and the control leaves it unchanged. The same holds for DINOv2 with registers, DINOv3, MAE and CLIP. We will add this table to Appendix E.
References