acceptodds
Withdrawn from ICLR 2027

From Global to Local: Rethinking CLIP Feature Aggregation for Person Re-Identification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

CLIP-based person re-identification (ReID) methods typically compress spatial evidence into a single global [CLS] representation, which can mix identity-relevant and corrupted regions under occlusion and cross-camera variation. We propose SAGA-ReID, a feature aggregation framework that delays spatial compression by reconstructing each patch through a shared anchor basis before pooling: image patches act as cross-attention queries while learned anchors serve as keys and values, preserving patch-indexed representations rather than first compressing the image into a small set of anchor-indexed features. This limits direct mixing between clean and corrupted patches during refinement and enables selective aggregation afterward. We evaluate this mechanism under synthetic masking and human-distractor occlusion, where SAGA's advantage over global pooling grows with occlusion severity before declining as identity information is exhausted. On standard and occluded ReID benchmarks, SAGA consistently improves over CLIP-ReID, with gains of up to +9.1 Rank-1 on occluded benchmarks. Its reconstructed feature achieves higher Rank-1 than CLIMB-ReID’s dedicated Mamba feature despite starting from a weaker backbone, while integrating SAGA with the CLIMB backbone further improves the final fused system on both Rank-1 and mAP. Finally, sensitivity analysis shows that performance is largely invariant to anchor-basis size, indicating that the gains arise from the reconstruction geometry rather than increased anchor capacity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
CLIP, representation learning, feature aggregation, semantic anchors, person re-identification, occlusion robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fC31OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SAGA-ReID to improve CLIP-based person re-identification under occlusion by refining spatial features before pooling. It constructs shared anchors through a frozen CLIP text encoder, augments them with image-conditioned domain anchors, and uses intermediate visual patches as cross-attention queries over this anchor basis. The resulting patch-indexed features are pooled using attention-derived weights and fused with the global representation. Controlled masking and human-distractor experiments demonstrate improved robustness over the global baseline, while standard and occluded benchmarks show consistent improvements over CLIP-ReID. Integrating SAGA with a CLIMB-trained backbone further improves the fused representation.

Critical strengths
  1. Useful empirical improvements under occlusion. SAGA improves over CLIP-ReID on both reported occluded benchmarks while retaining improvements on standard datasets. This provides practical evidence that the additional refinement branch benefits retrieval beyond the synthetic corruption setting.

  2. Evaluation covers both missing and misleading evidence. The controlled experiments distinguish masking from human-distractor occlusion, with the backbone and gallery held fixed. This is consequential because suppressing absent information and resisting another person's appearance are different robustness challenges.

Critical weaknesses
  1. The comparisons do not establish a convincing competitive robustness advantage. Table 1 acknowledges substantially stronger clean-benchmark results from PromptSG, while the controlled robustness curves compare only against the global feature. On Occluded-Market, standalone SAGA-ReID also matches reproduced CLIMB-ReID in Rank-1 but trails it in mAP. The results establish improvement over CLIP-ReID, but leave the tradeoff against stronger alternatives inadequately characterized.

  2. Delayed compression has not been established as the cause of the improvements. The intermediate tokens already contain cross-patch information from ViT self-attention, and the domain anchors introduce additional coupling through mean pooling. Preserving token indices therefore does not establish separation of clean and corrupted information. Without matched adaptation and pooling controls on the same intermediate tokens, the robustness curves and anchor-count analysis do not isolate the proposed mechanism.

Questions for authors
  1. Evidence needed for competitive robustness: Under the same protocol, how does SAGA compare with stronger image-based baselines (especially after 2025)?

  2. How does SAGA compare with capacity- and training-matched token-wise adaptation followed by mean or learned pooling, using exactly the same frozen intermediate tokens and fusion-selection protocol?

  3. The selective-pooling reliability measure lacks sufficient validation. Eq. (9) measures the concentration of attention over anchors, rather than target-identity relevance. A distractor patch could also align confidently with an anchor. The paper reports neither a uniform-pooling ablation nor quantitative target-versus-distractor validation of these weights. Does max-anchor attention quantitatively distinguish target patches from distractor patches in the compositing experiments? How much of the improvement remains when Eq. (9) is replaced with uniform pooling?

Reviewer_2Ym1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces SAGA-ReID, which measures reliability of each patch and selectively performs spatial pooling. Therefore, it specifically improves robustness against occlusions.

Critical strengths

None

Critical weaknesses

W1. Notation clarity : In Introduction section, It is unclear what subscript kk or β\beta indicates, thus hard to follow. how different akia_{ki} and βik\beta_{ik} in terms of subscript? is β\beta a scalar? confusing.

W2. Visibility of Fig 2 : It is hard to read due to too small texts, which can be enhanced easily.

W3. Definition of Notation

  • In Eq 1, among α\alpha, VxV_x, tt, none of these is defined. can not understand.
  • In Eq 2, same issue. r, \beta, V_a, a^+.
  • Same issues exist in other equations. I cannot understand the algorithm fully.

W4. Lack of analysis

  • L_div is introduced to enhance diversity of anchors, however, there is no evidence it really does.
  • No analysis of how domain anchor D is determined.
  • How selective pooling is performed is missing.

Overall, the current manuscript is not ready to publish. Too many symbols are not defined and never explained, thereby too hard to follow. Furthermore, analysis of each proposed component is not sufficient, leading to poor explainability.

Questions for authors

Please see Weakness.

Reviewer_PfUPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SAGA-ReID (Structured Anchor Guided Aggregation for ReID), a feature aggregation framework designed to address the failure of CLIP-based ReID methods in occluded scenarios. Its core idea is to reverse the direction of cross-attention: image patches query shared anchor bases (rather than the traditional approach in which anchors query patches), thereby preserving patch-indexed representations before pooling and delaying spatial compression. Experiments were conducted on the Market-1501, MSMT17, Occluded-ReID, and Occluded-Market benchmarks, and the authors claim that favorable results have been achieved.

Critical strengths

(1) The design of reversing the cross-attention direction—having patches query the anchor basis rather than having anchors query patches—ensures that each patch is refined through the same anchor basis without direct patch-to-patch mixing, thereby preserving N patch-indexed representations. This perspective of "delayed spatial compression" provides a new direction for thinking about ReID feature aggregation. (2) The paper designs three levels of experiments: (a) controlled masking experiments verify SAGA's advantage when partial evidence is lost; (b) real-world occlusion experiments verify its robustness to semantically misleading evidence; and (c) a comparison with SOTA models on standard and occlusion benchmarks. These three levels progress stepwise from controlled variables to real-world scenarios, providing strong evidence for the approach. (3) The method is described accurately, the formulas are clearly defined, and the geometric comparison diagram in Figure 1 is intuitive and easy to understand. The appendix provides a comprehensive analysis of computational overhead, a configuration table, details on control experiments, and implementation details for polarity inversion, facilitating reproducibility.

Critical weaknesses

(1) The relevant work primarily covers ReID and CLIP-based methods; however, it does not address similar patch mixing/aggregation strategies used in visual transformers (such as Perceiver's cross-attention, MLP-Mixer in ViT, or recent works on efficient attention). This limits the paper's influence within the broader visual understanding community. (2) All experiments were conducted on ViT-B/16; performance on larger models such as ViT-L/14 or ViT-H/14 was not evaluated. Given that CLIP-based models exhibit considerable behavioral differences across different scales, it remains unclear whether the method retains its performance gains on larger backbones. Furthermore, the relative overhead of SAGA's trainable parameters on larger backbones is not discussed. (3) Ablation experiments show that freely learned anchors can already increase Rank-1 from 79.5 to 86.9, whereas structured anchors provide only a marginal additional mAP improvement (from 74.5 to 75.6). The paper acknowledges that "text-space parameterization serves as a secondary optimization prior rather than a reconstruction mechanism itself," but it does not sufficiently address the following question: since the core gain does not depend on textual semantics, why employ the complex CLIP text-space parameterization instead of a simple learnable vector? (4) The paper compares two aggregation strategies—global [CLS] and query-over-patch—but does not compare them against standard attention pooling methods (such as Softmax attention-weighted summation) or other fundamental approaches such as transformer pooling. These baseline methods can likewise preserve spatial information; the lack of comparison makes it unclear how significantly SAGA outperforms these simple methods. (5) The appendix provides GFLOPs (approximately 5.1 GFLOPs vs. 37.5 GFLOPs for the backbone) and training time (38 minutes vs. 60 minutes), but does not provide a comparison of actual inference latency (in the millisecond range). For practical deployment, absolute latency is more informative than relative FLOPs. It is recommended that the actual inference time across different batch sizes be included.

Questions for authors

Q1: In the paper, the SAGA anchor basis is parameterized via the CLIP text encoder; however, ablation studies show that freely learned anchors are already capable of capturing most of the gain. Have the authors attempted to introduce additional semantic structures into the text-space parameterization (e.g., predefining anchors by body part)? Compared with fully free learning, how do the convergence speed and final performance differ with structured initialization? Q2: The polarity inversion experiment (anchor patch suppression) employs an additional learnable attention pooling head (Equations 16–17), whereas SAGA uses a token-weighted sum by default. Could the difference between these two readout methods affect the fairness of the conclusions? Have the authors attempted to compare the two directions using the same pooling head? Q3: SAGA achieves a significant improvement over CLIP-ReID on Occluded-Market (+9.1 Rank-1), but a smaller improvement on clean Market-1501 (+0.8 Rank-1). Have the authors observed cases where SAGA's performance actually declines on certain specific categories or difficult samples? Have they conducted failure case analyses? Q4: The paper mentions that domain anchors are generated from mean patch representations, which introduces image-conditioned signals. How does the number M of domain anchors affect performance? Have you ever compared M=0 (purely shared anchors) with M>0? Could domain anchors potentially introduce noise in certain scenarios?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.