- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes What-Where Transformer (WWT), a ViT-style visual backbone that aims to explicitly separate semantic information ("what") and spatial information ("where"). The key idea is to replace the conventional token-token self-attention with a slot-based architecture, where image tokens represent semantic features and attention masks are treated as spatial representations. By maintaining and updating slot-mask pairs throughout the network, WWT aims to learn localization-aware representations directly from the backbone without relying on additional localization modules. The authors evaluate WWT on several tasks, including ImageNet classification, weakly supervised semantic segmentation, object discovery, and object detection. The results show that WWT can generate more interpretable attention masks and achieve better localization-related performance compared with standard ViT/DeiT baselines.
Critical strengths
1. Interesting motivation and architecture design. The paper addresses an important limitation of current ViT-based models: localization information is often not explicitly modeled in the backbone and is usually handled by task-specific heads. The proposed what-where separation provides an interesting perspective for designing more localization-aware visual backbones.
2. The proposed architecture is reasonably novel. Instead of simply applying slot attention or adding an extra localization module after a pretrained ViT, WWT integrates slot-based representations into the backbone itself. The mutual attention mechanism between tokens and slots is a meaningful design choice.
3. The experiments show promising localization ability. The results on weakly supervised semantic segmentation and object discovery demonstrate that the learned attention masks contain useful spatial information. The qualitative visualizations are also interesting and support the intuition behind the method.
Critical weaknesses
W1. The main claim of what-where separation is not fully validated. The paper claims that WWT explicitly separates semantic representations ("what") and localization information ("where"). However, the current evidence mainly comes from visualization and downstream task performance. It is still unclear whether the slots truly capture semantic information while the masks mainly encode spatial information. For example, the translation experiment provides some evidence that slots are less sensitive to spatial changes, but this may not be sufficient to demonstrate a real disentanglement between semantic and spatial representations. More direct analysis would make the main claim more convincing.
W2. The contribution of the architecture and auxiliary reconstruction loss is unclear. The reconstruction objective seems to play an important role in learning localization ability. According to the ablation study, removing the reconstruction loss significantly decreases CorLoc performance. Therefore, it is unclear whether the localization capability mainly comes from the proposed slot-based architecture or from the additional reconstruction supervision. A more detailed analysis separating the effects of different components would help clarify the actual contribution of WWT.
Questions for authors
Q1. Comparisons and practical efficiency. Could the authors provide additional comparisons with recent object-centric methods and localization-oriented ViTs to better contextualize WWT’s performance? In addition, does the FLOPs reduction translate into practical speed or memory benefits when the additional slot and mask processing is taken into account? Could the authors report end-to-end latency, peak memory usage, and throughput under the same hardware and input settings to support the efficiency claim?
If you can satisfactorily address the weaknesses and questions, we would be very happy to increase the score.
4 comments
Part 2/2
Q1a. Comparisons with recent object-centric methods and localization-oriented ViTs
Table 3 includes DeiT + LaSt and Obj-DINOv2/v3 for zero-shot single-object discovery. WWT achieves 41.4%/30.2% CorLoc on VOC12/COCO20k, compared with 29.7%/23.1% for DeiT + LaSt and 36.0%/23.4% for Obj-DINOv3. DINOv1 + LOST achieves higher CorLoc at 61.9%/50.7%.
We additionally evaluated ImageNet-supervised GPViT-L1 without localization-specific training, extracting proposals from its internal 16-way group attention maps and ranking them by attention concentration.
| Model | ImageNet Top-1 | VOC12 CorLoc | Recall@1 | Recall@5 | Recall@10 | Recall@15 |
|---|---|---|---|---|---|---|
| GPViT-L1 | 80.5% | 17.2% | 12.5% | 23.6% | 25.0% | 25.0% |
| WWT-Tiny | 71.3% | 41.4% | 15.8% | 35.2% | 38.3% | 42.4% |
GPViT-L1 achieves higher classification accuracy, while WWT-Tiny achieves higher CorLoc and recall at all evaluated proposal budgets under the respective readouts. This comparison supports the usefulness of WWT’s backbone masks for zero-shot object discovery.
For target-dataset object-centric learning, we will add the published SPOT and SmoothSA results alongside our existing comparisons.
| Method and mask readout | VOC12 instance mBO | COCO instance mBO |
|---|---|---|
| WWT-OCL | 39.1 | 25.2 |
| DINOSAUR-MLP | 39.5 | 27.7 |
| DINOSAUR-Trans | 44.0 | 32.3 |
| SPOT, Slot Attention masks | — | 33.7 |
| SmoothSA, slot attention maps | 45.2 | 33.4 |
SPOT and SmoothSA report stronger results with additional mechanisms for object-centric learning, including decoder-guided self-training in SPOT and input-conditioned slot initialization in SmoothSA. DINOSAUR-MLP is our primary baseline because it combines DINO features with standard Slot Attention and uses the same decoder type as WWT-OCL. Against this baseline, WWT-OCL achieves nearly identical instance mBO on VOC12 and scores 2.5 points lower on COCO. These results support our proof of concept for in-backbone grouping: WWT’s backbone slots support object-centric reconstruction with an MLP decoder, without an SSL-pretrained encoder or a separate Slot Attention module.
Q1b. End-to-end latency, throughput, and peak memory
Could the authors report end-to-end latency, peak memory usage, and throughput under the same hardware and input settings to support the efficiency claim?
Table 10 in the supplementary material reports latency and throughput at our lowest evaluated resolution, 224 × 224, where WWT shows no runtime advantage over DeiT. We are extending these measurements to higher resolutions and measuring peak GPU memory under matched hardware, precision, and batch sizes. We will report the results and measurement settings in a follow-up response and revise the efficiency claim accordingly.
Official Comment by Reviewer vNSi
Thank you for your patient response. Your response addressed most of our concerns. We will increase the score to 3. Regarding the evaluation of efficiency, please provide more explicit details on the experimental setup and hardware specifications in the revised paper, as the results can vary significantly depending on the equipment used.
Thank you again for your reassessment and constructive feedback. We have added the hardware specifications and additional runtime and GPU memory measurements to the revised manuscript.
Part 1/2
We thank Reviewer vNSi for the feedback on the representation analysis, the role of reconstruction, and practical efficiency.
W1. Direct evidence for semantic versus spatial specialization
Fig. 7 and Appendix Fig. 9 examine how slot embeddings and masks respond to object translation. To complement that analysis, we conducted a new experiment in which probes predict object category and position from frozen WWT representations.
In a new experiment to further validate this, we constructed images by cutting objects out of PASCAL VOC2012 using their instance segmentation masks and placing each cutout at all 16 positions on a 4 × 4 grid. We trained separate linear probes to predict the object’s center coordinates from all 64 slots or masks jointly. The slot features were flattened into a 24,576-dimensional vector (64 × 384), and the masks into a 12,544-dimensional vector (64 × 14 × 14). Every category therefore had the same distribution of grid positions. We trained separate category and position probes on slot embeddings and spatial masks, using objects from the VOC2012 training split. We evaluated them on 54,576 composite images generated from 3,411 validation objects.
Category accuracy was 55.2% from slots and 25.2% from masks. For position prediction, the root mean squared error between predicted and ground-truth coordinates was 0.201 from slots and 0.095 from masks, giving masks 22% of the mean squared error obtained from slots. At every grid position, slots yielded higher category accuracy and masks yielded lower mean Euclidean position error. These results support preferential specialization under this protocol, with category more readily decoded from slots and position more accurately decoded from masks.
W2. Architecture versus auxiliary reconstruction
The strongest localization results come from combining the WWT architecture with RGB reconstruction, rather than from either alone. Table 6(b) shows that removing either reconstruction or logit propagation substantially reduces CorLoc.
Without reconstruction, WWT achieves 19.8% CorLoc, compared with 38.7% for the full model. Conversely, keeping reconstruction but recomputing patch-to-slot logits at each block reduces CorLoc to 23.4%, despite preserving the slot count and bidirectional token–slot interactions. Adding RGB reconstruction to DeiT also falls short, yielding 25.7% CorLoc with patch tokens used as reconstruction units.
These controls show that reconstruction alone does not account for WWT’s localization performance: propagating spatial assignments provides a further 15.3-point gain with reconstruction enabled. We will clarify that the reported localization performance depends on the combination of the architecture and the auxiliary objective.
In this sense, our contribution is to learn recognition and grouping jointly within a classification backbone, with spatial assignments propagated across blocks. This allows the same backbone to classify images and provide localization masks without a separate grouping module.