acceptodds
Under review as a conference paper at ICLR 2027

Objects as Addresses: Instance Binding in Vision Transformer Attention

Abstract

Humans naturally parse visual scenes into coherent individual objects rather than scattered features, a capability cognitive science calls object binding. While recent work shows that pretrained Vision Transformers (ViTs) intrinsically acquire this ability, how native self-attention computations mechanistically keep object instances separate remains unresolved. By studying this across eight pretrained ViTs, we found that instance separation is expressed more strongly in attention routing than in token content: selected query–key relations distinguish individual objects more reliably than value similarity or residual representations. Inside heads, patches belonging to one individual tend to read from a common image region, which we term their attention address. Visually distinct parts of one person share an address, whereas identical duplicated instances receive separate, movable addresses. These shared destinations arise in part because query interactions with the spatial structure of keys direct attention toward consistent regions for one object and elsewhere for another. In DINOv2 and most other backbones we tested, editing this query component at the selected head redirects its attention address toward another individual. Grouping patches by this query component separates same-category instances, reaching .615 FG-ARI versus .289 for the same clustering of final-layer features, which tend to merge them.

Primary area:
interpretability and explainable AI
Keywords:
object binding, mechanistic interpretability, vision transformers, self-attention, object individuation, representation analysis

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_eLe1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how frozen pretrained ViTs distinguish two objects of the same category. Using a rigorously controlled paired-patch test (matching category, foreground, and exact relative offset), the authors demonstrate that Q/K routing separates instances much better than value similarity. They identify an "attention address" mechanism, which is predicted by a "spatial request" (the query's projection onto a fixed quadratic key basis). Editing this request causally redirects attention toward another instance. Finally, they show that clustering patches by their spatial requests can separate repeated objects without any training.

Critical strengths
  • The paper addresses a fundamental and important question in vision about how models group features into distinct objects. It clearly points out that previous methods fail to control for simple cues like object distance and category agreement, making this study's motivation very strong.

  • The controlled patch-pair test designed in this paper is very rigorous. By matching exact distance and direction, it effectively removes confounding factors and successfully isolates the model's true ability to separate instances. The idea of using a spatial request to explain and edit attention is also very clean and logical.

  • The intervention experiments are highly convincing. For example, the model gives different addresses to identical copies of an object, and changing the spatial request directly moves where the model looks, explaining 82 percent of the actual movement. The authors also test their idea on many different models and are honest about where it fails, like in SigLIP.

  • The paper shows a useful application. By grouping patches using these spatial requests without any training, it gets a score of 0.615 on COCO, which is much better than using standard features (0.289) or a trained model called SPOT (0.356).

Critical weaknesses
  • The paper relies on labeled data from the VOC dataset to pick the best attention head and fit the spatial model. Although the authors show this choice is stable, this reliance on labels should be stated more clearly at the beginning of the paper rather than only in the appendix.

  • The analysis of the spatial request is a bit simple. It would be helpful to know if this request changes across different layers and if the model could learn it automatically instead of using a fixed mathematical formula.

  • The paper shows the idea fails in SigLIP and a supervised model, but it does not fully explain why. It would be better to provide a quantitative analysis of when the mechanism holds, such as checking if special tokens or attention sinks cause these failures.

  • The practical use is currently a bit limited. The segmentation masks are not very accurate, and using the full attention pattern actually gives a better score for grouping objects than using the spatial request.

  • The current experiments are mostly focused on human body parts. It is unclear if the mechanism and the request-changing experiment work on non-human objects and different object categories.

Questions for authors
  • Can the authors show the spatial fit results separately for foreground and background patches?

  • Does the mechanism and the successful request-changing experiment extend beyond human body parts to non-human scenes and different categories?

  • Would adding register tokens to SigLIP fix the problem and make the spatial request work again?

  • How sensitive is the grouping score to the clustering radius and the choice of the selected attention head?

3 comments
AuthorsOpenReview ↗

Response to Reviewer eLe1 (1/2)

We thank the reviewer for a careful and encouraging review. Each question pointed to a specific analysis, and we ran all of them. In short: attention sinks on image patches explain the SigLIP result, and registers help as the reviewer expected (W3, Q3). The request swap works in every COCO category, not only for people (W5, Q2). One fixed spatial basis fits object and background patches equally well (Q1). The request separates instances from the middle layers on, and needs no fixed formula (W2). The grouping is stable across radii and heads (Q4), and the use of labels will be stated up front (W1).

W3 and Q3. SigLIP, attention sinks and registers

Thank you for this suggestion. The intuition was right, and following it explained the SigLIP result. Many heads send part of their attention to a few tokens whatever the query asks, often called attention sinks. Darcet et al. (2024) showed that in ViTs such tokens appear on image patches of the background, and that added register tokens take over this role. When a sink is a CLS or register token, it has no position, and the paper already leaves it out of the address (Section 4.1). When it is an image patch, it pulls the address of every patch toward itself, and the request can no longer predict it. Leaving out the two most-attended patches shows this directly:

Selected headAttention on the two most-attended patchesAddress predicted by the request (R2R^2)Same, without those two patches
DINOv2.11.64.75
DINOv2 with registers.05.72.73
SigLIP (no CLS token).41−.19.55

SigLIP has no CLS token to absorb the sink, and four tenths of the attention of each patch lands on two image patches. Without them, its request predicts the address again. DINOv2 with and without registers is the controlled pair the reviewer asked about: registers move the sink off the image, and the request predicts the address better. We could not retrain SigLIP with registers, but within the DINOv2 family the answer to the question is yes. Across the eight backbones, the attention that lands on image-patch sinks predicts where address prediction fails. We will add this analysis to Appendix E.2.

W5 and Q2. Beyond human body parts

The body-part test uses people because COCO annotates keypoints only for people, which lets us compare different parts of one individual. The request swap itself does not need parts, so we ran it on all 1,414 COCO test images of the paper, which include 63 categories besides people.

Images (COCO test set of the paper)NumberBefore swapAfter swapEqual-size control
People653−.60+.27−.58
All other categories761−.40+.16−.40
Cow41−.57+.17−.56
Car35−.51+.18−.50
Wine glass32−.47+.26−.46
Horse26−.53+.32−.52
Zebra26−.44+.14−.44
Microwave22−.52+.32−.51

Attention on instance B minus attention on instance A at the selected DINOv2 head, as in Table 10.

The swap redirects the head toward the other instance in every category, and the control leaves it unchanged. The same holds for DINOv2 with registers, DINOv3, MAE and CLIP. We will add this table to Appendix E.

References

  • Darcet, Oquab, Mairal and Bojanowski. Vision transformers need registers. ICLR, 2024.
AuthorsOpenReview ↗

Response to Reviewer eLe1 (2/2)

Q1. Spatial fit on foreground and background patches

The spatial basis is fitted once, without labels, from the average key at each position, and the request of every patch is read through the same basis. We used annotations here only to report the result separately for object and background patches: how well the requested destination predicts where each patch actually reads (the address prediction of Figure 5).

Address predicted by the request (R2R^2)Object patchesBackground patches
DINOv2.63.64
DINOv2 with registers.72.72
DINOv3.79.75
MAE.73.75

All 1,414 COCO test images: 115,000 object patches and 162,000 background patches.

The same spatial basis predicts the reading of object and background patches equally well, in every backbone. The spatial structure of the keys is a property of the head, not of objects, and this is what lets one fixed basis describe where every patch asks to read.

W2. The request across layers, and without a fixed formula

Yes, the request changes across layers. Early layers already route attention spatially, but their requests do not distinguish instances. From the middle layers on, the requests of patches on different instances separate, and the separation is strongest in the late layers that the paper studies. Figure 19 shows this on two cows, and over all 144 heads the strongest instance heads lie in layers 9 to 12.

On the formula: the quadratic is not imposed on the model. It is a compact description, fitted without labels, of a spatial structure that the keys already have. Replacing it by the average key at each position, with no formula at all, gives nearly the same results (instance separation .80 against .82). We deliberately do not learn the request. The aim of the paper is to describe what the frozen network computes, and a readout that needs no training keeps that description faithful to the network. It also lets future work build on it without annotations or any trained module, for example to keep instances apart in applications where no labels are available.

W4. Practical use, and the full attention pattern

We agree on both points. The full attention pattern is the complete routing of the head, 196 numbers per patch, and it groups slightly better (.649 against .615 FG-ARI). The request is five numbers of the query and recovers most of this separation, with better masks (mIoU .429 against .403). That five spatial numbers are enough is the point of Section 5. Grouping is how we test the mechanism, and the paper states that masks remain incomplete, because background patches also form groups. Completing them with a foreground signal is the next step named in our conclusion. We are pursuing it with promising first results, but we think it deserves its own study, compared with unsupervised object discovery and object-centric methods.

W1. Labels for head selection

We agree this should be visible. The introduction states: "Labels define the test and are used to select the layers, heads, and relations on the VOC development split. Evaluation uses disjoint held-out VOC images. [...] No binding predictor is trained." Labels also choose the one radius of mean shift. The spatial basis is fitted without labels. We will repeat this at the start of Sections 3 and 5.

Q4. Sensitivity to the clustering radius and to the head

Figure 12 shows the choice of radius on the development images. On the test images the result is equally stable: requests stay between .57 and .63 FG-ARI for every radius from .30 to .50, while final features never exceed .29 at any radius. The choice of head also matters little among strong heads: nine of the ten best development heads group repeated objects better than final features.

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

Reviewer_DokUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how pretrained Vision Transformers distinguish individual object instances through self-attention. The authors introduce attention addresses, showing that patches belonging to the same object tend to attend to common spatial regions. They further identify a spatial request component within queries and analyze its influence on attention routing. Experiments across eight pretrained ViT backbones demonstrate instance-sensitive attention patterns, while intervention experiments show that modifying spatial requests can redirect attention. The authors also apply these representations to instance grouping.

Critical strengths
  1. Interesting interpretation of attention. The paper introduces attention addresses and spatial requests to explain how self-attention distinguishes different object instances.

  2. Broad experimental analysis. The authors investigate eight pretrained ViT backbones and multiple datasets, providing evidence that instance-sensitive information exists in attention routing.

  3. Useful intervention experiments. Modifying spatial requests redirects attention toward different objects, providing evidence that the identified components influence attention behavior.

Critical weaknesses
  1. Limited methodological novelty. The proposed approach primarily analyzes existing attention computations using similarity measures and spatial approximations. Its contribution is mainly interpretative rather than a new modeling technique.

  2. Insufficient evidence for object binding. Redirecting attention at a selected head does not necessarily demonstrate that the identified mechanism is responsible for maintaining object identity throughout the network. The causal relationship between spatial requests and actual object binding remains unclear.

  3. Limited comparison with existing methods. The instance grouping experiments compare against relatively few object-centric methods. Stronger comparisons with recent object discovery and segmentation approaches are needed to establish practical advantages.

  4. Limited practical applicability. The proposed grouping method is evaluated using ground-truth foreground patches and does not reliably recover complete object masks. Its effectiveness in realistic unsupervised object discovery remains uncertain.

Questions for authors
  1. How does modifying spatial requests affect object representations and instance assignments in subsequent layers, rather than only the selected attention head?

  2. Can the authors clarify the methodological differences between the proposed spatial request analysis and existing query-key decomposition or positional attention analyses?

  3. How does the proposed grouping method compare with recent object-centric learning and unsupervised instance segmentation approaches under comparable evaluation protocols?

  4. Can the proposed method recover complete object instances without using ground-truth foreground information, and how does performance change under this setting?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DokU (1/3)

We thank the reviewer for finding the interpretation interesting, the analysis broad and the presentation clear. The review helped us sharpen three points, and we address each below. The request matters beyond the edited head: removing it weakens instance separation in the next instance-sensitive head (W2, Q1). The novelty lies in a controlled test, the shared reading destinations it revealed and the spatial request that explains them, which earlier analyses of attention do not address (W1, Q2). On the same backbone, our training-free grouping does better than object-centric models trained for this task (W3, Q3), and no ground-truth foreground is used to form the groups (W4, Q4).

W2 and Q1. Effects of the request beyond the edited head

We answer this shared question in the general response, and summarise it here. The edit of Section 6.1 changes the query of one patch at one head. This is a small change to a residual stream that later heads read again, so they keep most of their preference. The direct test is to remove the request from all patches of the selected head. The next instance-sensitive head, which is never edited, then separates instances much less well, while other directions of the same query, and the other heads of the same layer, have little effect. Changing the request therefore changes instance assignments in the subsequent layers. A full reversal of later layers is not expected from one head, because separating individuals is built by several heads and layers that write into the same residual stream.

On object representations: final features are not the right place to judge the request. A final feature describes what a patch shows. The leg of one zebra looks like the leg of the other, so the two zebras have similar final features, and grouping final features merges them (.289 FG-ARI on DINOv2). Which zebra a patch belongs to is recorded in where it reads, a region of its own zebra, and this is what instance-sensitive heads make explicit (.615 FG-ARI with the request). The request is a small part of what forms the final feature, but it is the part that separates instances. This is why removing it lowers the instance separation of the heads that follow.

AuthorsOpenReview ↗

Response to Reviewer DokU (2/3)

W1 and Q2. Novelty, and differences from earlier analyses of attention

The paper is an interpretability study. Its novelty lies in what we found about how pretrained ViTs separate individuals of one category, and in the tools that made these findings possible:

  • A controlled test. We built, to our knowledge, the first test that compares instance pairs with pairs of another instance of the same category at the same distance and direction. Without this control, category, foreground and proximity explain most of the signal: many heads look instance-sensitive only because they separate objects from background, one category from another, or near patches from far ones, and one cannot tell which heads, if any, separate individuals. With it, we found that a few specific heads, and only those, separate individuals of the same category.
  • Shared reading destinations. We found that the patches of one individual read from a common region, which we named the attention address. An instance-sensitive head marks two patches as one object by giving them a shared destination, not by making them similar. Reading from similar destinations is a stronger same-instance signal than attending to one another.
  • A spatial request. We found that five numbers of the query predict this destination. Changing only these five numbers changes which individual a patch reads from.

Li et al. (2025) showed with a trained probe that binding information is present in ViT representations. We identify which native computations express it and how, without training anything.

The analyses the reviewer mentions answer different questions. Positional analyses of attention (Cordonnier et al., 2020; d'Ascoli et al., 2021) describe heads that read at a fixed offset from each patch, like a convolution. Such heads solve the easy cases by proximity and direction, since nearby patches usually belong to the same object. This shortcut is strong: on same-category pairs that are not matched for distance, proximity alone reaches .85 AUC, and it falls to chance once distance and direction are matched (Section 3.2). A head that reads at a fixed offset cannot link the ear of a zebra to its leg when another zebra stands next to it, which is the real binding problem. Such local heads are typical of early layers, before the network represents whole objects, while in DINOv2 the strongest instance-sensitive heads lie in layers 9 to 12 (Figure 8). Query–key decompositions (Pan et al., 2024) analyse the weight matrices of a head, which are the same for every image. The request is instead what one query asks for in one image: a region of that image, which differs between two instances and is shared by the parts of one instance.

References

  • Cordonnier, Loukas and Jaggi. On the relationship between self-attention and convolutional layers. ICLR, 2020.
  • d'Ascoli, Touvron, Leavitt, Morcos, Biroli and Sagun. ConViT: Improving vision transformers with soft convolutional inductive biases. ICML, 2021.
  • Li, Salehi, Ungar and Kording. Does object binding naturally emerge in large pretrained vision transformers? NeurIPS, 2025.
  • Pan, Philip, Xie and Schwartz. Dissecting query-key interaction in vision transformers. NeurIPS, 2024.
AuthorsOpenReview ↗

Response to Reviewer DokU (3/3)

W3 and Q3. Comparison with object-centric methods

Grouping is how the paper tests that the mechanism carries instance structure. A fixed, training-free grouping of five request numbers separates repeated objects far better than the final features of the same backbone (.615 against .289 FG-ARI on DINOv2), in seven of eight backbones. On the same backbone, it also does better than object-centric models trained on COCO for this purpose, on all three scores:

BackboneMethodTrainedFG-ARImIoUmBO$_i$
DINO ViT-B/16SPOT, autoregressive transformer decoderyes.356.288.350
Final featuresno.194.403.413
Request groupingno.398.350.374
DINOv2 ViT-S/14SPOT, autoregressive transformer decoderyes.489.322.364
Final featuresno.355.372.400
Request groupingno.535.409.431
DINOv2 ViT-B/14DINOSAUR, MLP decoderyes.590.350.382
Final featuresno.289.317.356
Request groupingno.615.429.451

All 1,587 COCO images with repeated objects, scored as in Table 12. Details of the checkpoints and of the head selection are in our reply to Reviewer dia9 (W4).

Figure 14 shows grouped video frames next to final features and SPOT, and Figures 17 and 18 show the shared destinations of each person in crowded scenes.

This matters beyond the paper. Object-centric models such as DINOSAUR (Seitzer et al., 2023) and SPOT (Kakogeorgiou et al., 2024) often merge similar instances into one slot, and SemanticSlots (Sabri et al., 2026) names the ambiguity between identical entities as a limitation. The request is a signal, available with no training, that keeps such instances apart, and these models could build on it. A new object discovery method is beyond the scope of this paper, and we do not present request grouping as one.

W4 and Q4. Ground-truth foreground and complete objects

No ground-truth foreground is used to form the groups. Mean shift groups every patch of the image, background included. Only the FG-ARI score leaves background patches out when scoring, which is its standard definition in object-centric learning. mIoU and mBO$_i$ score the complete groups, background spill included, and on them request grouping is also above the trained slot models on the 1,587 images with repeated objects, and above final features on DINOv2 (Table 12). Ground-truth foreground appears only in the illustrative last column of Figure 14, which we will label more clearly. We will also make the scoring explicit in Section 6.2.

We agree that requests alone do not give complete objects, and the paper states this in its limitations. Instance-sensitive heads focus on objects, so requests separate instances well, but background patches also form groups of their own. Separating objects from background is a different signal, studied in other work, which can be obtained without supervision. Combining the two into complete object discovery is the next step named in our conclusion. We are pursuing this direction with promising first results. A complete object discovery method, with the comparisons it requires, would however be a second contribution beyond the mechanistic scope of this paper.

References

  • Kakogeorgiou, Gidaris, Karantzalos and Komodakis. SPOT: Self-training with patch-order permutation for object-centric learning with autoregressive transformers. CVPR, 2024.
  • Sabri, Bilodeau, Saunier and Bouachir. Semantic slots for video object-centric learning. BMVC, 2026.
  • Seitzer et al. Bridging the gap to real-world object-centric learning. ICLR, 2023.

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

Reviewer_dia9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how pretrained Vision Transformers (ViTs) distinguish different object instances through self-attention. Experiments across eight ViTs show that attention routing is more effective at separating instances than value or residual representations. The authors introduce the concept of attention addresses, where patches belonging to the same object tend to attend to common regions. They further identify a low-dimensional spatial request derived from query–key interactions and show that modifying it can redirect attention in selected heads. Finally, clustering spatial requests achieves better instance separation than clustering conventional ViT features.

Critical strengths
  1. The paper investigates an interesting and fundamental question about how pretrained Vision Transformers distinguish individual objects, providing insights into the internal mechanisms of visual representation learning.
  2. The distinction between attention routing and token content offers an interesting perspective on object binding, going beyond conventional analyses of feature similarity.
  3. The analysis covers eight pretrained ViT backbones with different training objectives, providing reasonably broad evidence for the observed differences between attention routing and feature representations.
Critical weaknesses
  1. The conclusion that instance information is better represented in attention routing than in token content is not fully supported. The comparison mainly uses cosine similarity for residual and value representations, which does not rule out the possibility that instance information is equally accessible through more expressive readouts.
  2. The intervention experiments do not establish that the identified mechanism is responsible for object binding in the overall network. Redirecting attention in a selected head is an interesting result, but the later heads largely retain their original instance preferences. The causal connection to object identity preservation therefore remains unclear.
  3. The intervention does not completely isolate the proposed spatial request mechanism. Modifying the query also changes its interactions with image-specific key components outside the fitted spatial basis. Consequently, the observed attention shifts cannot be attributed exclusively to the proposed spatial formulation.
  4. The comparison with SPOT is not entirely fair. The headline instance-grouping result compares spatial requests extracted from DINOv2 with SPOT using a DINO backbone. When the same DINO backbone is considered, the performance advantage becomes considerably smaller, weakening the claimed superiority over existing object-centric approaches.
Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dia9 (1/2)

We thank the reviewer for a precise and fair review, which helped us state several results more accurately. In short: removing the request of the selected head weakens instance separation in the next instance-sensitive head, so the mechanism matters beyond the edited head (W2). The fitted spatial request alone produces most of the redirection of the swap (W3). A trained probe confirms that instance information is present in the residual stream, while the native routing of the selected head expresses it more directly (W1). On the same backbone, request grouping does better than trained slot models (W4).

W2. Does the intervention show a role in the network, given that later heads keep their preference?

This is the central question, and we answer it in the general response. In short, the edit of Section 6.1 changes the query of one patch at one head. It adds a small change to a residual stream that later heads read again, so they keep most of their preference. Whether later heads depend on the request is tested by removing it from all patches of L10h2. The next instance-sensitive head, L11h6, then loses more than half of its instance separation above chance. Removing other directions of the same query, or the request of any other head of layer 10, has little effect. The observation that later heads keep their preference came from the one-patch edit, which tests where the selected head reads, not whether the network uses it. Tested directly, the request has a large effect on the next instance-sensitive head. We do not claim that one head carries object membership through the whole network. We will replace the last sentence of Section 6.1 by a precise statement of what the edit shows.

W3. The swap does not isolate the spatial request

The swap changes only the five request coefficients of the query. As the reviewer notes, the new query is then compared with the actual keys of the image, which carry more than the average spatial pattern that our basis describes (Eq. 9, Appendix D.5). To isolate the spatial request, we recomputed the attention of the head with only the fitted spatial part of the score change, leaving out the rest of the keys.

Part of the score change appliedAddress moves (patches)
None0
Fitted spatial request only3.3
Full swap (Section 6.1)3.7

The 215 scenes of Section 6.1.

The fitted spatial request alone moves the address 3.3 patches, against 3.7 for the full swap, in the direction predicted by the request in 214 of the 215 scenes. The spatial request therefore causes most of the redirection by itself. The full swap goes a little further because the same request is compared with the real keys of the object at the requested place, rather than with their average pattern. We will add this analysis to Appendix D.5.

AuthorsOpenReview ↗

Response to Reviewer dia9 (2/2)

W1. Routing against content, and cosine similarity

We agree that cosine similarity does not bound what a representation contains. The paper does not claim that residual or value vectors lack instance information. Li et al. (2025) already showed, with a probe trained on object labels, that binding information is present in ViT representations. Our question is different: which native computation of the network makes it explicit, and how. A trained probe can tell that information is present. It cannot tell whether the network itself reads it, or how. This is why the paper studies the comparison that the network already performs, the query–key relation of each head, learned without labels.

The distinction is visible in the numbers. A bilinear probe of the kind used by Li et al., trained on 840,000 pairs built with our controlled test on COCO training images, does recover much of the instance information in the residual stream that cosine similarity misses (.79 against .56 AUC). The native attention of the selected head, with no training at all, still separates the same pairs better (.82). The information is present in the residual, but the network expresses it most directly in the routing of selected heads, in a form that a frozen model exposes with no training. We will make this distinction between presence and expression explicit in Section 3.3, and add the probe to Appendix A.

W4. The comparison with SPOT

We agree that the comparison should be made on one backbone, and we thank the reviewer for pointing this out. We now compare request grouping with a slot model trained on the same backbone, for three backbones:

BackboneMethodTrainedFG-ARImIoUmBO$_i$
DINO ViT-B/16SPOT, autoregressive transformer decoderyes.356.288.350
Final featuresno.194.403.413
Request groupingno.398.350.374
DINOv2 ViT-S/14SPOT, autoregressive transformer decoderyes.489.322.364
Final featuresno.355.372.400
Request groupingno.535.409.431
DINOv2 ViT-B/14DINOSAUR, MLP decoderyes.590.350.382
Final featuresno.289.317.356
Request groupingno.615.429.451

All 1,587 COCO images with repeated objects, scored as in Table 12. Released COCO checkpoints with seven slots (SPOT on DINOv2 ViT-S: SmoothSA release, three seeds). For ViT-S, the head and radius were selected with our controlled test on COCO training images.

On every backbone, five numbers read from one frozen head, with no training, do better than the slot model trained on COCO for this task on all three scores. They also separate instances far better than the final features of the same backbone (FG-ARI). The clearest case is SPOT with its transformer decoder on DINOv2 ViT-S: in 8.7% of the images, most patches of two different instances fall into one slot, against 1.6% for request grouping. We will report these matched comparisons in Section 6.2 and Table 12, in place of the comparison across backbones.

References

  • Li, Salehi, Ungar and Kording. Does object binding naturally emerge in large pretrained vision transformers? NeurIPS, 2025.

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

Reviewer_1zhWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigate how vision transformers seperate instances within the same categoary. Using a controlled patch-pair test that compares same-instance pairs against same-category cross-instance pairs with distance and direction matched, this paper shows across eight backbones that query/key relations separate individuals more reliably than value or residual similarity, and that this ranking transfers from VOC to COCO. They term the resulting pattern an attention address: the mean destination of a patch's attention row, shared by visually dissimilar parts of one person, distinct for identical copies, and following the object when it moves. A low-dimensional spatial request read from the query predicts this address, can be edited to redirect a head toward the other person, and clusters repeated instances notably better than final-layer features.

Critical strengths
  1. This paper focus on a interesting problem, how neural networks seperate different objects, though they belong to the same class.
  2. The finding about instance separation living in routing, not in the content is interesting. The authors do experiments on various pretrained model to find this result.
  3. The experiments is abundant and solid, making convenice results.
Critical weaknesses
  1. The mechanistic claim only be conducted on a single head in a single layer. More comperhensive analysis is also important.
  2. Visual recognition now mostly relys on vision-language-models. Experiments on VLM will improve this manuscript.
  3. The paper may just move the binding problem one step earlier instead of solving it. The request tells us where a patch will read, but not why this query asks for zebra A rather than zebra B.
Questions for authors

Does editing the request change anything outside the edited head? You report that later instance-sensitive heads still prefer person A after the swap. Does the mechanism therefore have no functional consequence for the model?

3 comments
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the reviewers for careful and constructive reviews. For the AC, we first list the contributions, with what the reviewers saw as strengths and concerns, and where each concern is answered. We then answer the question shared by Reviewers dia9, DokU and 1zhW.

Contributions

  1. A controlled test of individuation. Patch pairs from one instance are compared with pairs from another instance of the same category, at the same distance and direction. This isolates the few heads that separate individuals from the many that only separate object from background, one category from another, or near from far. Reviewers: a strength for eLe1 (rigorous). No concern raised.
  2. Routing separates individuals. In eight pretrained ViTs, the query–key relations of specific heads separate individuals more directly than value or residual similarity. Reviewers: a strength for dia9, DokU and 1zhW. Concern: whether a stronger readout of content would match it (dia9, answered in W1).
  3. Shared reading destinations. Patches of one individual read from a common region, its attention address, even when they look different. Identical copies receive different addresses. Reviewers: a strength for DokU and eLe1. No concern raised.
  4. A spatial request. Five coefficients of the query predict this destination, and editing them redirects the head to another individual. Reviewers: a strength for DokU and eLe1. Main concern: its effect beyond the edited head (dia9, DokU, 1zhW), answered below.
  5. A training-free use. Grouping patches by these five numbers separates repeated objects that final features merge, and does better than slot models trained on the same backbone. Reviewers: a useful application for eLe1. Concerns: a fair SPOT comparison (dia9), broader comparisons and complete masks (DokU, eLe1), answered in their replies.

The paper does not claim to solve object binding as a whole. It explains one central part of it: how instance-sensitive heads keep individuals of one category apart.

Shared question: does the request matter beyond the edited head?

Why later heads keep most of their preference. The edit of Section 6.1 changes five numbers in the query of one patch at one head (L10h2), and shows that the request controls where this head reads. Heads communicate through the residual stream: each head reads it and adds its output back into it (Elhage et al., 2021). A one-patch edit changes one of the twelve head outputs added to that patch, and leaves every other patch unchanged. Later heads, which read the stream again, therefore keep most of their preference. A small downstream effect is what a one-head, one-patch edit should produce. Computations in transformers are commonly shared by several heads that compose through the residual stream, as for induction heads (Olsson et al., 2022), so no single head is expected to carry the whole result.

This agrees with the main finding of the paper. Instance information can be present in the residual stream without being visible to simple similarity. The query–key comparison of selected heads is the computation that makes it explicit. Our reply to dia9 (W1) shows this with a trained probe.

Later heads depend on the request. The direct test removes the request for all patches: every patch of an image receives the image-mean request at L10h2. This changes only the five request directions of its 64-dimensional query, and nothing else in the network. We then measure later heads with the controlled test.

Query of L10h2, changed for all patchesL10h2L11h6L12h11
None.82.68.75
Request removed (five directions).63.59.73
Five random directions removed.82.68.75
Five high-variance directions outside the request removed.82.66.75

Same-instance AUC (0.5 is chance), 1,414 COCO test images. Only L10h2 is changed.

The next instance-sensitive head, L11h6, loses about half of its separation above chance, although it is never edited. Other directions of the same query have little effect, and the same removal in any other head of layer 10 lowers L11h6 by at most .02. Further layers depend less on this single head (L12h11), as expected when several heads contribute.

Conclusion. The request controls where an instance-sensitive head reads, and the next instance-sensitive head depends on this routing. We do not claim that one head carries object membership through the whole network.

References

  • Elhage et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
  • Olsson et al. In-context learning and induction heads. Transformer Circuits Thread, 2022.
AuthorsOpenReview ↗

Response to Reviewer 1zhW

We thank the reviewer for the positive assessment and for questions that sharpened the paper. In short: removing the request weakens instance separation in the next instance-sensitive head, so the mechanism has a functional consequence beyond the edited head (Q1). The analysis covers every head of the network and eight backbones (W1). The request is built progressively across layers (W3). The vision encoders used by vision–language models are part of our study (W2).

Q1. Does editing the request change anything outside the edited head?

Yes. We answer this in the general response. The edit of Section 6.1 changes the query of one patch at one head. This is a small change to a residual stream that later heads read again, so they keep most of their preference. Whether later heads depend on the request is tested by removing it from all patches of the selected head. The next instance-sensitive head, which is never edited, then loses more than half of its instance separation above chance, and no other head of the same layer has this effect. The mechanism therefore has a functional consequence beyond the head. As is usual in transformers, the computation is shared by several heads rather than held by one.

W1. One head in one layer

The analysis is not limited to one head. Figure 8 of the paper already shows the instance separation of every head, and the same request analysis applies to each head with its own spatial basis. The heads that separate individuals behave like the selected one: among the fifteen strongest heads of DINOv2, fourteen are redirected by the request swap, and requests separate individuals more strongly with depth. Heads that do not separate individuals, for example heads that separate objects from background, are not expected to, and the controlled test is what tells the two apart. The same organisation holds at the selected head of seven other backbones and of DINOv2 ViT-L (Appendix E.2). We analysed one head in detail because a causal analysis needs a precise target.

W3. Does the request only move the problem one step earlier?

The request is computed from the representation of the patch, so part of the answer lies in earlier layers, and we agree that the paper does not explain all of binding. What it explains is how selected heads keep two individuals apart: their patches ask to read from different places, and the request is that place. This is a concrete step, because it turns the abstract question of which object a patch belongs to into a spatial one that can be measured and edited.

We also know where this comes from in broad terms. The request is built progressively across layers: early layers already read spatially, without separating instances, and the requests of different instances separate from the middle layers on (Figure 19). Changing where a patch reads in those middle layers changes the request it makes later. Which image cues first set the two requests apart is an open question, which we will name in the conclusion.

W2. Vision–language models

CLIP and SigLIP are vision–language models, trained to match images with text, and both are among the eight backbones of the paper. We understand the reviewer to mean generative VLMs, such as LLaVA. Most of them use a CLIP or SigLIP vision encoder, often frozen. LLaVA-1.5 (Liu et al., 2024), for example, uses a frozen CLIP encoder, and CLIP is already part of our study. Our paper studies binding inside the vision encoder. How the language model uses these addresses when it refers to one particular object is an interesting question for future work.

References

  • Liu, Li, Li and Lee. Improved baselines with visual instruction tuning. CVPR, 2024.

The rebuttal successfully addressed most of my concerns. As a result, I decide to keep my original rating as weak accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript uploaded

We thank the reviewers for their thoughtful feedback. We have uploaded a revised manuscript addressing the main concerns raised during the discussion, with additional experiments and clarifications.

All changes are highlighted in blue for ease of review.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.