acceptodds
Withdrawn from ICLR 2027

Rethinking Distillation in Pedestrian Re-Identification: A Spatial Aggregation Perspective

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pedestrian re-identification (Re-ID) is an open-set retrieval task in which practical systems often require either improving retrieval performance under fixed computational constraints or reducing inference cost with limited accuracy degradation. To address these requirements through knowledge distillation, we revisit which teacher signal should be transferred for Re-ID. We find that large changes in individual descriptor values do not necessarily imply comparable degradation of retrieval-relevant structure, motivating supervision of how descriptors are formed rather than their exact values. In Vision Transformer (ViT)-based Re-ID models, late-block [CLS]-to-patch attention provides an observable proxy for how spatial evidence is aggregated into the global representation. We study spatial aggregation as an effective teacher signal for open-set Re-ID, focusing on how the retrieval representation is formed rather than on exact agreement with teacher outputs. Across multiple ViT-based Re-ID frameworks and benchmarks, spatial aggregation transfer consistently improves both same-capacity refinement and lightweight student distillation, requiring only a teacher model and standard Re-ID training data, without additional annotations or inference-time components.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Pedestrian Re-Identification, Open-Set Retrieval, Knowledge Distillation, Vision Transformers, Spatial Aggregation, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SQteOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates knowledge distillation in Vision Transformer (ViT)-based pedestrian re-identification (Re-ID) from a spatial aggregation perspective. Through inference-time [CLS] token replacement interventions, the authors observe that large variations in individual image descriptors do not necessarily lead to a comparable degradation in retrieval performance; instead, changes in late-block attention align much more closely with retrieval behavior. Motivated by this insight, the proposed method distills the head-averaged, [CLS]-to-patch attention distribution across the latter half of Transformer blocks after excluding the [CLS] self-attention entry and re-normalizing. The distillation objective combines cross-entropy and reverse KL divergence while preserving standard Re-ID classification and triplet losses. Experiments span three Re-ID frameworks, five benchmarks, same-capacity refinement, and lightweight student distillation, alongside evaluations on multi-dataset joint training, zero-shot transfer, and random-seed stability.

Critical strengths
  1. The paper offers a novel motivation by emphasizing that open-set retrieval tasks such as person re-identification depend on relative rankings and geometric relationships among features, rather than their exact numerical values.
  2. The method is simple and easy to integrate into existing frameworks, without requiring additional large-scale annotated data.
  3. The method demonstrates strong robustness and generalization.
Critical weaknesses
  1. The [CLS] token intervention study has a potential confounding factor. Replacing [CLS] at Block 2 perturbs its interactions with patch tokens throughout the remaining ten blocks, potentially disrupting subsequent representation learning. In contrast, replacing it at Block 12 leaves access to patch tokens that have already acquired deep semantic information, allowing the final aggregation step to retain substantial retrieval-relevant information. Thus, the poor performance of early interventions may reflect disruption of the subsequent computation rather than deficiencies in early-layer attention patterns alone. The authors should clarify this distinction.
  2. The term “Spatial Aggregation” appears to introduce substantial conceptual framing around what is essentially distillation of late-layer [CLS] attention maps. Further explanation and analysis are needed to establish what this perspective contributes beyond conventional attention-map distillation.
  3. The comparison primarily includes relatively older baselines and lacks comparisons with more recent relevant methods.
Questions for authors

As indicated in Critical Weaknesses.

Reviewer_35hcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper revisits knowledge distillation for ViT-based person re-identification from a spatial aggregation perspective. Rather than directly matching the teacher’s logits or final feature embeddings, the proposed method encourages the student to match the teacher’s late-block [CLS]-to-patch attention distributions. It averages attention across heads and combines cross-entropy with reverse KL divergence, alongside standard Re-ID supervision. The method is straightforward to implement, requires no additional annotations, and introduces no additional inference-time components. Experiments across multiple Re-ID frameworks and datasets, together with component ablations, provide a reasonably comprehensive evaluation. However, the methodological novelty appears limited, as the core approach remains teacher–student attention matching, with the main differences lying in the selection and construction of the attention targets. It also remains unclear to what extent the improvements reflect the transfer of teacher-specific spatial aggregation knowledge versus a beneficial regularization effect. Furthermore, replacing [CLS] affects both attention and subsequent representation formation, so the intervention experiments do not isolate the contribution of attention changes to retrieval performance. Overall, the paper presents a practical approach with encouraging empirical results, while the distinctiveness of its contribution and the evidence supporting its interpretation warrant further clarification.

Critical strengths

Strength 1:

The method is simple and practical. It uses attention maps already available within the model and introduces only an additional training constraint, without requiring extra annotations or inference-time components. This makes it straightforward to implement and allows the student to benefit from distillation without additional inference overhead, supporting its practical value for deployment.

Strength 2:

The experimental evaluation is reasonably comprehensive, covering multiple Re-ID frameworks and datasets, with ablation studies examining different components of the method. This breadth of evaluation provides evidence that the improvements are consistent across different settings and helps substantiate the main design choices.

Critical weaknesses

Weakness 1:

The methodological novelty appears limited relative to prior attention-distillation work. Li et al. [1] already investigate transferring ViT attention patterns while allowing the student to learn its own features, using attention supervision alongside a task objective. More directly, Wang et al. [2] distill [CLS]-query attention using KL divergence and aggregate attention heads to accommodate different teacher–student head counts. Although their setting and exact formulation differ from those of this submission, these precedents indicate that attention matching, [CLS]-query supervision, and head aggregation are established ideas. The present contribution therefore appears to lie mainly in the Re-ID-specific formulation, late-block selection, target construction, and loss combination. The paper needs to more clearly establish the substantive advance provided by these choices beyond existing attention-distillation approaches.

Weakness 2:

The intervention experiments discussed in the paper support a correlation, but are insufficient to conclude that retrieval performance is determined by these attention patterns. Replacing [CLS] simultaneously affects subsequent information interactions and feature formation. Changes in attention may be one cause of retrieval degradation, or they may co-occur with other changes.

References:

[1] Alexander C. Li, Yuandong Tian, Beidi Chen, Deepak Pathak, and Xinlei Chen. On the Surprising Effectiveness of Attention Transfer for Vision Transformers. NeurIPS, 2024.

[2] Kai Wang, Fei Yang, and Joost van de Weijer. Attention Distillation: Self-Supervised Vision Transformer Students Need More Guidance. BMVC, 2022.

Questions for authors

Question 1:

Is the contribution arising from the choice of attention in this work sufficient to support the new perspective emphasized in the paper? Does the student improve because it learns the teacher’s knowledge, or because it is subjected to a beneficial constraint? How does this work distinguish between these two sources of improvement?

Question 2:

Replacing [CLS] not only changes the attention distribution but may also affect information interactions and the formation of the final features. The association between attention changes and retrieval performance degradation is insufficient to establish the specific role of attention changes in this process. Could the authors further explain to what extent the existing experiments distinguish the effects of attention changes from those of other changes in the representations?

Reviewer_DbW1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits knowledge distillation for ViT-based person re-identification from a spatial aggregation perspective. Motivated by a [CLS] token intervention study, it proposes transferring the teacher’s late-block [CLS]-to-patch attention distributions. The method excludes the [CLS] self-attention entry, averages attention across heads, and re-normalizes the resulting patch distributions. A frozen teacher supervises the student through cross-entropy and reverse KL divergence, alongside standard identity classification and triplet losses. The approach supports both same-capacity refinement and lightweight student distillation without adding inference-time components. Experiments cover three Re-ID frameworks and five benchmarks, complemented by component ablations, multi-dataset training, zero-shot evaluation, and random-seed stability analysis.

Critical strengths

1.The evaluation spans three ViT-based Re-ID frameworks and five benchmarks, covering both same-capacity refinement and lightweight student distillation. It includes comparisons with several alternative distillation signals and ablations on distilled blocks, attention targets, and loss functions. Multi-dataset training, evaluation on an unseen dataset, and four-seed experiments further examine generalization and stability. This breadth provides substantial support for assessing the method across different models and evaluation settings.

2.The paper clearly motivates the question of which teacher signal should be transferred for open-set person re-identification. The [CLS] token interventions, together with analyses of descriptor changes and retrieval-structure preservation, provide a concrete empirical basis for investigating late-block attention as a distillation target. These analyses connect the motivating observations to the method design and make the reasoning behind the proposed supervision easy to follow.

Critical weaknesses

1.Attention Transfer[1] already introduced attention distillation to transfer information flow while allowing students to learn their own features and removing the teacher at inference. The present method retains this mechanism, with modifications involving late-block [CLS]-to-patch attention, head averaging, and cross-entropy plus reverse KL. The authors also acknowledge that their objective is equivalent to symmetric KL up to a student-independent constant. The methodological contribution therefore appears incremental, and the paper should more clearly establish its substantive technical advance beyond a Re-ID-specific adaptation.

2.Table 3 shows that full-attention distillation performs closely to the proposed target: both obtain 74.0 mAP for ViT-B→B on MSMT17, while ViT-B→S results are 71.3 versus 71.6 on MSMT17 and 52.2 versus 52.8 on NightReID. These results support [CLS]-only distillation as a compact, effective target, but do not isolate it as the main source of improvement. Since full attention also contains the [CLS] row, a patch-query-only control excluding that row would help distinguish global-token aggregation supervision from more general attention distillation.

3.Figure 1 and Appendix A.1 replace the entire [CLS] token at different depths and examine changes in descriptors, attention, and retrieval. Such interventions simultaneously alter the residual state, associated query/key/value computations, and subsequent network inputs; intervening at different depths also changes the remaining computation. Although Table 8 adds retrieval-geometry measurements, the experiment still does not isolate how much of the retrieval change is attributable to attention rather than other computational paths. This limits its explanatory strength as a rationale for selecting attention as the distillation signal.

[1]Alexander C. Li, Yuandong Tian, Beidi Chen, Deepak Pathak, and Xinlei Chen. On the Surprising Effectiveness of Attention Transfer for Vision Transformers. Advances in Neural Information Processing Systems (NeurIPS), 37, 2024.

Questions for authors

1.Could you clarify the substantive methodological advance over Attention Transfer, beyond its adaptation to Re-ID and the modifications to attention selection, head averaging, and loss formulation?

2.Could you add a patch-query-only distillation control that excludes the [CLS] query row, using comparable distilled blocks and loss settings, to test the specific contribution of [CLS] aggregation supervision?

3.Could you directly perturb the [CLS]-to-patch attention distribution at a fixed layer, while preserving the original value vectors and residual input, to isolate the effect of attention on retrieval performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.