- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates knowledge distillation in Vision Transformer (ViT)-based pedestrian re-identification (Re-ID) from a spatial aggregation perspective. Through inference-time [CLS] token replacement interventions, the authors observe that large variations in individual image descriptors do not necessarily lead to a comparable degradation in retrieval performance; instead, changes in late-block attention align much more closely with retrieval behavior. Motivated by this insight, the proposed method distills the head-averaged, [CLS]-to-patch attention distribution across the latter half of Transformer blocks after excluding the [CLS] self-attention entry and re-normalizing. The distillation objective combines cross-entropy and reverse KL divergence while preserving standard Re-ID classification and triplet losses. Experiments span three Re-ID frameworks, five benchmarks, same-capacity refinement, and lightweight student distillation, alongside evaluations on multi-dataset joint training, zero-shot transfer, and random-seed stability.
Critical strengths
- The paper offers a novel motivation by emphasizing that open-set retrieval tasks such as person re-identification depend on relative rankings and geometric relationships among features, rather than their exact numerical values.
- The method is simple and easy to integrate into existing frameworks, without requiring additional large-scale annotated data.
- The method demonstrates strong robustness and generalization.
Critical weaknesses
- The [CLS] token intervention study has a potential confounding factor. Replacing [CLS] at Block 2 perturbs its interactions with patch tokens throughout the remaining ten blocks, potentially disrupting subsequent representation learning. In contrast, replacing it at Block 12 leaves access to patch tokens that have already acquired deep semantic information, allowing the final aggregation step to retain substantial retrieval-relevant information. Thus, the poor performance of early interventions may reflect disruption of the subsequent computation rather than deficiencies in early-layer attention patterns alone. The authors should clarify this distinction.
- The term “Spatial Aggregation” appears to introduce substantial conceptual framing around what is essentially distillation of late-layer [CLS] attention maps. Further explanation and analysis are needed to establish what this perspective contributes beyond conventional attention-map distillation.
- The comparison primarily includes relatively older baselines and lacks comparisons with more recent relevant methods.
Questions for authors
As indicated in Critical Weaknesses.