acceptodds
Under review as a conference paper at ICLR 2027

What-Where Transformer: Concurrent Representation and Localization in a Slot-Centric Visual Backbone

Abstract

Image understanding tasks involve identifying what is present and where it appears. However, classification-oriented visual backbones primarily optimize semantic representations of what, while spatial information about where often remains implicit or entangled with semantic features. We hypothesize that explicit separation of these what and where representations can induce localization ability within backbones. To examine this, we introduce the What-Where Transformer (WWT), a ViT-style attentive backbone that incorporates what-where separation as an architectural inductive bias. Our method introduces two key designs: (1) it treats tokens as representations of what and attention maps as representations of where, and processes them in concurrent feed-forward modules via a multistream, slot-based architecture; (2) it reuses both the final-layer tokens and attention maps for downstream tasks, and directly exposes them to gradients derived from tasks, facilitating explicit learning of localization. Trained on ImageNet with single-label classification and an auxiliary reconstruction objective, WWT discovers multiple objects directly from its raw attention maps without token clustering or other postprocessing, at a modest cost to classification accuracy. Furthermore, WWT achieves superior performance compared to ViT-based methods on zeroshot object discovery and weakly supervised semantic segmentation, and it is transferable to various localization setups with minimal modifications.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
What-where separation, representation learning, zero-shot object discovery, slot attention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vNSiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes What-Where Transformer (WWT), a ViT-style visual backbone that aims to explicitly separate semantic information ("what") and spatial information ("where"). The key idea is to replace the conventional token-token self-attention with a slot-based architecture, where image tokens represent semantic features and attention masks are treated as spatial representations. By maintaining and updating slot-mask pairs throughout the network, WWT aims to learn localization-aware representations directly from the backbone without relying on additional localization modules. The authors evaluate WWT on several tasks, including ImageNet classification, weakly supervised semantic segmentation, object discovery, and object detection. The results show that WWT can generate more interpretable attention masks and achieve better localization-related performance compared with standard ViT/DeiT baselines.

Critical strengths

1. Interesting motivation and architecture design. The paper addresses an important limitation of current ViT-based models: localization information is often not explicitly modeled in the backbone and is usually handled by task-specific heads. The proposed what-where separation provides an interesting perspective for designing more localization-aware visual backbones.

2. The proposed architecture is reasonably novel. Instead of simply applying slot attention or adding an extra localization module after a pretrained ViT, WWT integrates slot-based representations into the backbone itself. The mutual attention mechanism between tokens and slots is a meaningful design choice.

3. The experiments show promising localization ability. The results on weakly supervised semantic segmentation and object discovery demonstrate that the learned attention masks contain useful spatial information. The qualitative visualizations are also interesting and support the intuition behind the method.

Critical weaknesses

W1. The main claim of what-where separation is not fully validated. The paper claims that WWT explicitly separates semantic representations ("what") and localization information ("where"). However, the current evidence mainly comes from visualization and downstream task performance. It is still unclear whether the slots truly capture semantic information while the masks mainly encode spatial information. For example, the translation experiment provides some evidence that slots are less sensitive to spatial changes, but this may not be sufficient to demonstrate a real disentanglement between semantic and spatial representations. More direct analysis would make the main claim more convincing.

W2. The contribution of the architecture and auxiliary reconstruction loss is unclear. The reconstruction objective seems to play an important role in learning localization ability. According to the ablation study, removing the reconstruction loss significantly decreases CorLoc performance. Therefore, it is unclear whether the localization capability mainly comes from the proposed slot-based architecture or from the additional reconstruction supervision. A more detailed analysis separating the effects of different components would help clarify the actual contribution of WWT.

Questions for authors

Q1. Comparisons and practical efficiency. Could the authors provide additional comparisons with recent object-centric methods and localization-oriented ViTs to better contextualize WWT’s performance? In addition, does the FLOPs reduction translate into practical speed or memory benefits when the additional slot and mask processing is taken into account? Could the authors report end-to-end latency, peak memory usage, and throughput under the same hardware and input settings to support the efficiency claim?

If you can satisfactorily address the weaknesses and questions, we would be very happy to increase the score.

4 comments

We thank Reviewer vNSi for the feedback on the representation analysis, the role of reconstruction, and practical efficiency.

W1. Direct evidence for semantic versus spatial specialization

It is still unclear whether the slots truly capture semantic information while the masks mainly encode spatial information.

Fig. 7 and Appendix Fig. 9 examine how slot embeddings and masks respond to object translation. To complement that analysis, we conducted a new experiment in which probes predict object category and position from frozen WWT representations.

In a new experiment to further validate this, we constructed images by cutting objects out of PASCAL VOC2012 using their instance segmentation masks and placing each cutout at all 16 positions on a 4 × 4 grid. We trained separate linear probes to predict the object’s center coordinates from all 64 slots or masks jointly. The slot features were flattened into a 24,576-dimensional vector (64 × 384), and the masks into a 12,544-dimensional vector (64 × 14 × 14). Every category therefore had the same distribution of grid positions. We trained separate category and position probes on slot embeddings and spatial masks, using objects from the VOC2012 training split. We evaluated them on 54,576 composite images generated from 3,411 validation objects.

Category accuracy was 55.2% from slots and 25.2% from masks. For position prediction, the root mean squared error between predicted and ground-truth coordinates was 0.201 from slots and 0.095 from masks, giving masks 22% of the mean squared error obtained from slots. At every grid position, slots yielded higher category accuracy and masks yielded lower mean Euclidean position error. These results support preferential specialization under this protocol, with category more readily decoded from slots and position more accurately decoded from masks.

W2. Architecture versus auxiliary reconstruction

Therefore, it is unclear whether the localization capability mainly comes from the proposed slot-based architecture or from the additional reconstruction supervision.

The strongest localization results come from combining the WWT architecture with RGB reconstruction, rather than from either alone. Table 6(b) shows that removing either reconstruction or logit propagation substantially reduces CorLoc.

Without reconstruction, WWT achieves 19.8% CorLoc, compared with 38.7% for the full model. Conversely, keeping reconstruction but recomputing patch-to-slot logits at each block reduces CorLoc to 23.4%, despite preserving the slot count and bidirectional token–slot interactions. Adding RGB reconstruction to DeiT also falls short, yielding 25.7% CorLoc with patch tokens used as reconstruction units.

These controls show that reconstruction alone does not account for WWT’s localization performance: propagating spatial assignments provides a further 15.3-point gain with reconstruction enabled. We will clarify that the reported localization performance depends on the combination of the architecture and the auxiliary objective.

In this sense, our contribution is to learn recognition and grouping jointly within a classification backbone, with spatial assignments propagated across blocks. This allows the same backbone to classify images and provide localization masks without a separate grouping module.

Q1a. Comparisons with recent object-centric methods and localization-oriented ViTs

Table 3 includes DeiT + LaSt and Obj-DINOv2/v3 for zero-shot single-object discovery. WWT achieves 41.4%/30.2% CorLoc on VOC12/COCO20k, compared with 29.7%/23.1% for DeiT + LaSt and 36.0%/23.4% for Obj-DINOv3. DINOv1 + LOST achieves higher CorLoc at 61.9%/50.7%.

We additionally evaluated ImageNet-supervised GPViT-L1 without localization-specific training, extracting proposals from its internal 16-way group attention maps and ranking them by attention concentration.

ModelImageNet Top-1VOC12 CorLocRecall@1Recall@5Recall@10Recall@15
GPViT-L180.5%17.2%12.5%23.6%25.0%25.0%
WWT-Tiny71.3%41.4%15.8%35.2%38.3%42.4%

GPViT-L1 achieves higher classification accuracy, while WWT-Tiny achieves higher CorLoc and recall at all evaluated proposal budgets under the respective readouts. This comparison supports the usefulness of WWT’s backbone masks for zero-shot object discovery.

For target-dataset object-centric learning, we will add the published SPOT and SmoothSA results alongside our existing comparisons.

Method and mask readoutVOC12 instance mBOCOCO instance mBO
WWT-OCL39.125.2
DINOSAUR-MLP39.527.7
DINOSAUR-Trans44.032.3
SPOT, Slot Attention masks—33.7
SmoothSA, slot attention maps45.233.4

SPOT and SmoothSA report stronger results with additional mechanisms for object-centric learning, including decoder-guided self-training in SPOT and input-conditioned slot initialization in SmoothSA. DINOSAUR-MLP is our primary baseline because it combines DINO features with standard Slot Attention and uses the same decoder type as WWT-OCL. Against this baseline, WWT-OCL achieves nearly identical instance mBO on VOC12 and scores 2.5 points lower on COCO. These results support our proof of concept for in-backbone grouping: WWT’s backbone slots support object-centric reconstruction with an MLP decoder, without an SSL-pretrained encoder or a separate Slot Attention module.

Q1b. End-to-end latency, throughput, and peak memory

Could the authors report end-to-end latency, peak memory usage, and throughput under the same hardware and input settings to support the efficiency claim?

Table 10 in the supplementary material reports latency and throughput at our lowest evaluated resolution, 224 × 224, where WWT shows no runtime advantage over DeiT. We are extending these measurements to higher resolutions and measuring peak GPU memory under matched hardware, precision, and batch sizes. We will report the results and measurement settings in a follow-up response and revise the efficiency claim accordingly.

Reviewer_vNSiOpenReview ↗

Official Comment by Reviewer vNSi

Thank you for your patient response. Your response addressed most of our concerns. We will increase the score to 3. Regarding the evaluation of efficiency, please provide more explicit details on the experimental setup and hardware specifications in the revised paper, as the results can vary significantly depending on the equipment used.

Thank you again for your reassessment and constructive feedback. We have added the hardware specifications and additional runtime and GPU memory measurements to the revised manuscript.

Reviewer_Rio8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose WWT, a method that separates what objects are from where they appear. It use tokens and attention maps to discover objects and improve localization and segmentation. There is however a performance drop in classification accuracy.

Critical strengths
  1. The idea is clear. Integrating slot-mask pairs directly into a ViT backbone is a simple and well-motivated design.
  2. The output is versatile and supports classification, segmentation and detection with lightweight heads.
Critical weaknesses
  1. Table 6b: Removing the reconstruction loss drops CorLoc significantly, but top1 barely changes. This suggests that localization mainly comes from the canonical OCL reconstruction and slot competition, which is not surprising, rather than from classification. Thus the narrative does not match the evidence.

  2. Equation 3 dose not normaliz slots thus lossing the typical slot competition as in Slot Attention the original version. As a result, most masks look almost identical as shown in Figure 5, 10 -- 15. The downstream heads must rely on thresholding, duplicate removal and a Herfindahl heuristic to select regions. The authors are suggested to justify this design choice through proper ablation.

  3. Table 5 only includes relatively old baselines, earlier than 2024, e.g., Slot Attention, SLATE and DINOSAUR. WWT-OCL showed no consistent superiority. Including newer and stronger Slot Attention based OCL methods, for example, SPOT and SmoothSA, should be covered.

  4. The classification gap to DeiT increases with model size. Increasing width brings almost no gain. It is thus unclear that if WWT could be a general-purpose backbone.

  5. Downstream performance is also rather limited. Supervised detection and segmentation clearly lag behind classifical methods DETR and FCN. In multi-object discovery, MOST is re-clustered to produce more candidates, so the evaluation protocol is not equivalent between the two methods.

  6. The claim that WWT is "the first ViT-style backbone that explicitly enforces what - where separation" is too strong. The differences from inverted-attention transformers (Wu et al., 2023) and from the concurrent RATS (Yang et al., 2026) and VECA (Song et al., 2026) should be clearly compared in the main text.

References:

  • SPOT: Self-training with patch-order permutation for object-centric learning with autoregressive transformers. CVPR 2024.
  • Smoothing Slot Attention Iterations and Recurrences. ICLR 2026.
  • Elastic attention cores for scalable vision transformers (VECA). arXiv 2026.
  • Inverted-attention transformers can learn object representations: Insights from slot attention. NeurIPS, 2023.
  • RATS! Patches talk through registers: Emergent parts in register attention transformers. arXiv 2026.
Questions for authors

See above.

4 comments

We thank Reviewer Rio8 for the detailed feedback.

W1. Contributions of RGB reconstruction and logit propagation

This suggests that localization mainly comes from the canonical OCL reconstruction and slot competition, which is not surprising, rather than from classification.

WWT learns recognition and grouping jointly within a classification backbone using class labels and auxiliary RGB reconstruction. While reconstruction-based grouping is established in Slot Attention, extending it to complex natural images has been challenging. DINOSAUR addresses this by reconstructing features from a pretrained SSL model. WWT instead learns backbone features and grouping together during ImageNet pretraining, without a pretrained SSL encoder or a separate Slot Attention module.

Table 6(b) shows that both reconstruction and logit propagation contribute. Removing reconstruction reduces CorLoc from 38.7% to 19.8%, with top-1 nearly unchanged (69.7% to 69.6%). Keeping reconstruction but recomputing logits at each block reduces CorLoc to 23.4%. Logit propagation therefore provides a 15.3-point gain under the same training objective.

DeiT with RGB reconstruction, using patch tokens as reconstruction units, achieves 25.7% CorLoc, below WWT's 38.7% under the ablation configuration. Adding reconstruction alone does not reproduce WWT's performance. Our contribution is to integrate recognition and grouping within the backbone, where persistent spatial assignments improve localization.

W2. Slot normalization, overlapping masks, and discovery heuristics

The downstream heads must rely on thresholding, duplicate removal and a Herfindahl heuristic to select regions. The authors are suggested to justify this design choice through proper ablation.

Eq. (3) normalizes across patches for each slot, allowing multiple slots to draw evidence from the same patch. Eq. (2) and the reconstruction head normalize across slots. This asymmetric design permits overlapping spatial support in WWT's overcomplete slot representation.

We tested Slot Attention-style competition under the full WWT-Tiny training configuration. We replaced the patch-axis softmax in Eq. (3) with slot-axis softmax followed by renormalization across patches for each slot before aggregating patch values. The remaining architecture and training recipe were unchanged.

Slot-update normalizationImageNet Top-1VOC12 CorLoc
Patch-axis normalization (WWT-Tiny)71.3%41.4%
Slot-axis softmax + patch-axis renormalization61.6%34.7%

Competition reduces top-1 by 9.7 points and CorLoc by 6.7 points under the same proposal-processing pipeline. This supports allowing overlapping slots within the backbone and resolving redundant predictions at the output stage, rather than imposing competition in every slot update. We use proposal filtering for zero-shot discovery and learned competition among retained slots at the reconstruction output for OCL.

For zero-shot discovery, masks are thresholded and split into connected components to generate box proposals. Duplicate proposals are removed, and concentration-based ranking selects the candidate for single-object discovery.

W3. Stronger object-centric baselines for Table 5

Including newer and stronger Slot Attention based OCL methods, for example, SPOT and SmoothSA, should be covered.

We provide the requested comparisons below. SPOT and SmoothSA values are published results under their respective training settings.

Method and mask readoutVOC12 instance mBOCOCO instance mBO
WWT-OCL39.125.2
DINOSAUR-MLP39.527.7
DINOSAUR-Trans44.032.3
SPOT, Slot Attention masks—33.7
SmoothSA, slot attention maps45.233.4

SPOT and SmoothSA achieve higher COCO scores. SPOT uses decoder-guided self-training, while SmoothSA introduces input-conditioned slot initialization. DINOSAUR-MLP remains our primary reference because it uses standard Slot Attention and the same decoder type as WWT-OCL. WWT-OCL achieves similar instance mBO on VOC12 and scores 2.5 points lower on COCO.

WWT-OCL starts from our classification-pretrained backbone and reconstructs frozen DINO features during OCL finetuning. Its results show that grouping can be supported by the backbone's own slots without a separate Slot Attention module, while the newer comparisons establish the gap to dedicated OCL methods.

W4. Width scaling and the scope of the backbone claim

The classification gap to DeiT increases with model size. Increasing width brings almost no gain. It is thus unclear that if WWT could be a general-purpose backbone.

The widening classification gap is a limitation. WWT-Small reaches 77.3%, and width-only WWT-Base reaches 77.4%, versus 79.8% and 81.8% for DeiT. WWT-Middle, which increases both width and depth relative to Small, reaches 77.9% at 15.8G FLOPs and 115M parameters, compared with 19.5G and 142M for WWT-Base. These preliminary results motivate further work on scaling. Our current evidence supports WWT as a localization-oriented backbone, with classification and efficiency trade-offs.

W5a. Supervised detection and segmentation lag dedicated models

Supervised detection and segmentation clearly lag behind classifical methods DETR and FCN.

Table 7 tests whether WWT's slot-mask representation transfers through lightweight heads. WWT-DET achieves 63.1 mAP, compared with 56.4 for DeiT using our detection head and 37.8 for the WWT patch-token readout. WWT-SEG achieves 61.3 mIoU, compared with 60.2 for DeiT using our segmentation head and 59.9 for the WWT patch-token readout.

These comparisons show useful localization information in the slot-mask representation, particularly for detection. DETR and ResNet-50 FCN achieve higher scores (72.2 mAP and 66.5 mIoU), but our question here is how directly the backbone's representation supports these tasks with lightweight heads. The gains over patch-token readouts suggest that WWT's slot-mask pairs provide a representation better aligned with downstream localization than the conventional patch-token representations..

W5b. Different proposal sets across K for MOST

In multi-object discovery, MOST is re-clustered to produce more candidates, so the evaluation protocol is not equivalent between the two methods.

We additionally evaluated MOST using one fixed proposal set per image at a fixed finer clustering granularity. Since the implementation does not provide candidate scores, we ranked proposals by the concentration index and applied each K cap to the same ranking. WWT uses its existing fixed proposal set and ranking.

DatasetMethod and protocolRecall@1Recall@5Recall@10Recall@15
VOC12MOST, granularity-specific18.9%34.0%34.2%33.5%
VOC12MOST, fixed proposals17.6%30.4%33.1%33.5%
VOC12WWT, fixed proposals15.8%35.2%38.3%42.4%
COCO20kMOST, granularity-specific6.7%12.4%13.0%13.6%
COCO20kMOST, fixed proposals6.9%10.6%12.4%13.6%
COCO20kWWT, fixed proposals4.8%11.3%17.2%19.4%

Under fixed proposals, MOST leads at K = 1, while WWT leads at K = 5, 10, and 15 on both datasets. At K = 15, WWT exceeds MOST by 8.9 points on VOC12 and 5.8 points on COCO20k. Thus, WWT's advantage at larger proposal budgets remains under a protocol that applies all K caps to a fixed ranked set.

W6. Novelty claim and relationship to inverted attention, RATS, and VECA

The claim that WWT is "the first ViT-style backbone that explicitly enforces what - where separation" is too strong.

The specific contribution is the joint refinement of patch features, slots, and an explicit patch-by-slot logit state across backbone blocks. We narrow our claim to this design rather than priority over all forms of what-where separation.

Wu et al.'s TF-Inv groups features from a separate image encoder into slots using Transformer-style updates. RATS routes patch communication through registers and reports emergent parts under self-supervised training. VECA uses persistent cores for patch communication and varies their number for scalable inference.

WWT carries the patch-to-slot logits across blocks, using them for both bidirectional communication and localization readouts. Table 6(b) shows the benefit of this persistent state within WWT: removing its propagation reduces CorLoc from 38.7% to 23.4%.

Thank the authors for the response and additional experiments. By the authors’ own numbers, WWT trails DeiT in classification with a gap that widens with scale (77.3 vs. 79.8, 77.4 vs. 81.8), trails the published results of recent OCL methods (COCO mBO 25.2 vs. 33.7 for SPOT and 33.4 for SmoothSA), and trails DETR and FCN (63.1 vs. 72.2 mAP, 61.3 vs. 66.5 mIoU). The authors have also narrowed the novelty claim and now position WWT as a localization-oriented backbone with classification and efficiency trade-offs. Since these limitations are acknowledged but not resolved, I maintain my original rating.

Thank you again for considering our responses. While we acknowledge the remaining limitations, we believe WWT offers a valuable proof of concept for a novel backbone architecture that jointly learns recognition features and reusable spatial assignments. We appreciate your feedback, which has helped us clarify the contribution and its scope.

Reviewer_6hSaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the What-Where Transformer, which biases the standard ViT to explicitly separate semantic information from spatial information. This is done by replacing full self-attention with bidirectional interactions between patch tokens and learnable slots, while keeping attention masks as separate representations across layers. The learned representations can be directly used for localization tasks, including weakly supervised semantic segmentation and zero-shot object discovery, with improvements over several ViT-based baselines.

Critical strengths
  • The proposed architecture is interesting and offers a novel approach to explicitly propagating and refining attention-mask states across backbone layers.
  • WWT improves over several supervised ViT baselines in localization tasks without dense supervision.
Critical weaknesses

Since a novel architecture is proposed, it should be shown that there is a real benefit to this idea compared to existing work, and there should be an in-depth analysis of the effect of the introduced biases (what-where separation). But the paper does not provide either of the two:

  • The performance gains are minimal. Although WWT improves over the supervised DeiT baseline in localization, stronger SSL models outperform it by far. In addition, WWT is worse in classification. That gap seems to scale with size (see Table 11), while WWT uses more parameters and has higher latency. Many experiments are also flagged as preliminary. This raises questions about the benefit of the model.

  • The central idea of a what-where separation is not studied well. The ablation in Table 6 even suggests that the reconstruction objective contributes more strongly to the separation than the architectural bias. The only real study is the translation invariance analysis in Fig. 7 using cosine similarity, which does not show that the representations are disentangled.

More systematic experiments, such as separate probes for semantic and positional information, and a more systematic evaluation are needed to support the claims.

  • While the framing of the what-where transformer seems quite novel, there is already work on alternatives to global patch-to-patch communication and their effects on visual tasks, i.e., GPViT (Yang et al., ICLR 2023) or the Native Segmentation ViT (Braso et al., NeurIPS 2025).

  • The spacing is too tight in some places, i.e., the section header in Section 5 or the captions of Figure 5 and Table 5. The authors should follow the template and not extensively use negative spacing to fit the page limit.

Questions for authors
  • Given the lower classification accuracy, higher computational cost, and stronger performance of existing SSL models, what is the main advantage of WWT over existing architectures?

  • Can the authors provide more direct evidence that semantic and spatial information are disentangled, for example through separate probing experiments?

  • How does WWT differ from existing grouping-based visual backbones such as GPViT and Native Segmentation ViT, and what specific benefit does the proposed architecture provide?

2 comments

We thank Reviewer 6hSa for the feedback on the architectural contribution, the evidence for what-where separation, and the scope of our evaluation.

W1 / Q1. Benefit relative to existing architectures

Given the lower classification accuracy, higher computational cost, and stronger performance of existing SSL models, what is the main advantage of WWT over existing architectures?

WWT’s demonstrated advantage is that a single backbone trained with ImageNet class labels and auxiliary RGB reconstruction can classify images and produce spatial masks that transfer to localization without task-specific retraining. On ImageNet-S50, these masks achieve 49.1 mIoU versus 35.9 for DeiT-Tiny with attention rollout (Table 2). For zero-shot object discovery, WWT achieves 41.4% and 30.2% CorLoc on VOC12 and COCO20k, respectively, versus 29.7% and 23.1% for our DeiT + LaSt reproduction. The SSL-based DINOv1 + LOST pipeline achieves higher CorLoc on both benchmarks (61.9% and 50.7%, Table 3).

The ablation in Table 6(b) shows that the architecture contributes to this capability. With the training objective held fixed, including RGB reconstruction, removing the propagation of patch-to-slot logits across blocks reduces CorLoc from 38.7% to 23.4%.

The current WWT models use more parameters and have higher batch-1 latency at both tested scales. Their ImageNet top-1 accuracy trails DeiT by 0.9 points at Tiny scale and 2.5 points at Small scale. These results establish a proof of concept for a localization-oriented backbone and quantify the trade-off observed in the tested models. Whether this design can retain its localization gains while closing the classification and efficiency gaps remains open.

W2 / Q2. Evidence for semantic and spatial separation

Can the authors provide more direct evidence that semantic and spatial information are disentangled, for example through separate probing experiments?

Figure 7 and Appendix Figure 9 show that slot embeddings are relatively stable when an object moves, while the masks track its movement. To complement this evidence, we ran a new experiment to measure how accurately object category and position can be decoded from each representation.

Additionally, we extracted objects from PASCAL VOC2012 using their instance segmentation masks and placed each cutout at all 16 positions on a 4 × 4 grid, giving every category the same distribution of grid positions. We froze the WWT backbone and trained separate probes to predict object category and position from slot embeddings and spatial masks. We trained the probes on images composed from objects in the VOC2012 training split and evaluated them on 54,576 images composed from 3,411 validation objects.

Category prediction accuracy was 55.2% from slots and 25.2% from masks. For position prediction, the root mean squared error between predicted and ground-truth coordinates was 0.201 from slots and 0.095 from masks. Squaring these values, the mean squared error from masks was 22% of that from slots. At every grid position, slots yielded higher category accuracy and masks yielded lower mean Euclidean distance to the ground-truth position.

W3 / Q3. Relationship to grouping-based visual backbones

How does WWT differ from existing grouping-based visual backbones such as GPViT and Native Segmentation ViT, and what specific benefit does the proposed architecture provide?

WWT is designed to produce recognition features and reusable localization masks from the same backbone. GPViT uses group tokens to exchange information and broadcasts the updated features back to the patch grid. Native Segmentation ViT progressively merges patches and derives hierarchical masks from those groupings. WWT preserves the patch grid and jointly refines slot features and patch-to-slot assignment logits across blocks. These assignments support bidirectional communication and provide the final localization masks.

We additionally evaluated ImageNet-supervised GPViT-L1 using box proposals extracted from its internal attention maps without additional training, ranked by the attention concentration in the same manner as WWT.

ModelImageNet Top-1VOC12 CorLocRecall@1Recall@5Recall@10Recall@15
GPViT-L180.5%17.2%12.5%23.6%25.0%25.0%
WWT-Tiny71.3%41.4%15.8%35.2%38.3%42.4%

GPViT-L1 achieves higher classification accuracy, while WWT-Tiny achieves higher CorLoc and recall at all evaluated proposal budgets under the respective readouts. This comparison demonstrates the usefulness of WWT’s explicit backbone masks for object discovery and illustrates the observed trade-off between classification accuracy and localization performance. In the qualitative examples we inspected, some GPViT groups covered multiple objects from different classes, limiting their usefulness as separate object discovery. This observation is consistent with the recall results and highlights the value of spatial masks that distinguish individual objects.

W4. Template and spacing

We will restore the template’s spacing around Section 5 and the captions of Fig. 5 and Table 5, and shorten the surrounding text where needed to meet the page limit.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback and the discussion. We have uploaded a revised manuscript incorporating the additional experiments and clarifications discussed in our responses. Changes are highlighted in blue:

  • Semantic and spatial specialization probes -- Reviewer 6hSa: W2/Q2; Reviewer vNSi: W1.
  • Slot-update normalization and competition ablation -- Reviewer Rio8: W2.
  • GPViT comparisons and architectural distinctions --Reviewer 6hSa: W3/Q3; Reviewer Rio8: W6; Reviewer vNSi: Q1a.
  • Comparisons with SPOT and SmoothSA --Reviewer Rio8: W3; Reviewer vNSi: Q1a.
  • MOST evaluation with fixed proposals -- Reviewer Rio8: W5b.
  • Higher-resolution throughput and GPU memory measurements, with hardware specifications -- Reviewer vNSi: Q1b.
  • Clarifications of reconstruction’s contribution and the classification, scaling, and downstream trade-offs -- Reviewer 6hSa: W1/Q1; Reviewer Rio8: W1, W4, W5a; Reviewer vNSi: W2.
  • Revised spacing around section headings, figures, and tables to improve readability -- Reviewer 6hSa: W4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.