- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SWR, a CLIP-based framework for image-based person re-identification that combines image-conditioned semantic construction with identity-level relational supervision. Its In-Encoder Visual Semantic Tokenization (IVST) inserts learnable tokens into the visual Transformer, selects their intermediate states across layers, and converts these states together with a global identity representation into pseudo-words for the frozen text encoder. An inherited cross-attention module integrates the resulting textual and visual representations. Identity-Relational Ranking (IRR) then aligns relative orderings among hard competing identity prototypes using ListMLE, optionally expanding the candidate pool with detached cross-batch memory. The paper reports good retrieval performance across standard and challenging benchmarks, provides several component ablations, and demonstrates improved training efficiency relative to its IADT reimplementation.
Critical strengths
-
A coherent design for constructing image-conditioned pseudo-words. IVST uses states formed through interactions between learnable tokens and image patches throughout the visual hierarchy. Combining these states with the global identity pseudo-word provides a plausible way to enrich language-side representations, and the component ablation supports the usefulness of both sources.
-
Broad empirical effectiveness without additional textual annotations. The reported results are good across Market-1501, MSMT17, Occluded-Duke, and CUHK03-D, covering several distinct retrieval conditions. Achieving these results without externally generated captions or manually annotated attributes supports the framework’s practical value.
Critical weaknesses
-
The actual IRR objective remains incompletely specified. Figure 1 suggests a textual-relation target, but neither Section 3.3 nor Appendix A.3 explicitly enumerates the teacher-student pair set . Several loss coefficients are also left unspecified. These omissions affect which representations supervise and update one another, preventing faithful reproduction and a precise assessment of the central relational mechanism.
-
The reliability of the teacher ordering is insufficiently validated. IRR derives target orders from learned representations rather than relevance labels among competing negative identities. Its hard sorting discards teacher score gaps, allowing nearly tied similarities to impose an unqualified total order. The alignment and retrieval-boundary diagnostics show useful downstream effects, but do not establish the stability or informativeness of the transferred orders.
Questions for authors
-
Please enumerate : which of RR, TT, RT, and TR supervises which relation, and are any directions reciprocal? Please also provide the unspecified loss coefficients.
-
How stable are the hard-identity target orders across different observations, augmentations, and training updates, especially for nearly tied scores? Do learned target orders outperform shuffled orders over exactly the same candidate identities?
-
Appendix B.2 states that the remaining IVST configuration is unchanged. However, with , either four-layer restricted pool contains only eight states, fewer than the default . Reducing or increasing would change the comparison, so Table 5 does not yet establish the benefit of full-hierarchy collection independently of semantic capacity. What are and the actual retained for each Table 5 variant? Does full-hierarchy collection remain beneficial when the number of pseudo-words is matched?
-
The IVST–IRR complementarity is not clearly demonstrated. Table 4 does not specify whether IRR is enabled, and its combined MSMT17 result matches full SWR. Moreover, the Baseline+IRR in Table 10 outperforms full SWR on Occluded-Duke. The above requires a reasonable explanation.
6 comments
Q4. Clarifying IVST–IRR Complementarity and Correcting Table 4
IVST–IRR complementarity and the Occluded-Duke result
Thank you for pointing this out. We first found an error in Table 4. The MSMT17 result of the combined IVST configuration was mistakenly copied from full SWR. The correct IVST-only result is 88.3/92.8 (mAP/Rank-1), not 88.9/93.3. Table 4 does not enable IRR, and we will explicitly state this in the revised caption and text.
With the corrected numbers, IVST alone obtains 96.4/98.0 on Market-1501 and 88.3/92.8 on MSMT17, whereas full SWR reaches 96.6/98.3 and 88.9/93.3, respectively. Thus, adding IRR on top of IVST gives consistent gains on both standard benchmarks, particularly +0.6 mAP / +0.5 Rank-1 on MSMT17.
We view the complementarity mainly from their different roles. IVST improves within-instance representation formation by constructing image-conditioned pseudo-words from multi-layer visual states, whereas IRR improves between-identity organization by applying identity-level listwise supervision to hard competing identities. Therefore, IVST changes the semantic representation used by the model, while IRR regularizes how identities are relatively organized in the retrieval space.
The Occluded-Duke result reveals an important boundary of this complementarity. Baseline+IRR reaches 73.5/77.1, while full SWR obtains 72.7/75.4. We do not interpret complementarity as requiring strictly additive gains on every domain. IVST is image-conditioned and therefore depends on the visual evidence available in each observation. Under severe occlusion, different images of the same identity may expose substantially different or incomplete regions, making single-image semantic construction less reliable. IRR, in contrast, aggregates observations at the identity level and regularizes relations among competing identities, which can be more robust when individual observations are incomplete.
We will revise the manuscript to make two points explicit: (1) IVST and IRR provide complementary improvements on the standard ReID benchmarks, where their combination gives the strongest results; and (2) under severe occlusion, their contributions are domain-dependent rather than strictly additive, with IRR showing stronger robustness to incomplete visual evidence.
Q3. Layer-range comparison under matched semantic capacity
Q. Layer-range comparison and matched semantic capacity
Thank you for raising this point. We first clarify the actual configuration and purpose of Table 5.
The experiment was designed to examine what happens when the source layers of VST states are restricted. With , the restricted layer ranges provide relatively small candidate pools, so all available states were used directly:
- L1--4: 8 candidates, all 8 used;
- L9--12: 8 candidates, all 8 used;
- L1--4 + L9--12: 16 candidates, all 16 used.
For the full L1--12 hierarchy, however, produces 24 candidate states, and the default Top- selection was applied. This configuration obtains 96.4 mAP / 98.0 Rank-1. We additionally evaluated using all 24 states without Top- selection, which gives 96.2 / 97.8. Thus, simply exposing more VST states is not sufficient; the gain of the default configuration comes from constructing a broader multi-layer candidate pool and selectively retaining useful states.
To directly address the reviewer’s fixed-capacity question, we further performed a matched experiment on Market-1501 with , , full IRR, the same seed, and the same 60-epoch training protocol:
| Layer range | Candidates | Retained | Best mAP | Rank-1 |
|---|---|---|---|---|
| L1--4 | 8 | 8 | 96.5 | 98.0 |
| L9--12 | 8 | 8 | 96.4 | 97.9 |
| L1--4 + L9--12 | 16 | 8 | 96.4 | 97.8 |
| L1--12 | 24 | 8 | 96.4 | 97.9 |
At the strictly matched budget, broader layer coverage alone does not yield an additional accuracy gain. This indicates that the default IVST gain should not be attributed simply to using more visual layers.
Our default setting uses , following the empirical pseudo-word budget adopted in IADT and further supported by the -- sensitivity analysis in Appendix B.3 / Fig. 5. Under the complete SWR setting, full-hierarchy reaches 96.6 mAP / 98.3 Rank-1, the strongest configuration in our experiments. A four-layer range with contains only eight states, so enforcing there would necessarily require increasing and would change another capacity factor.
We also examined whether the full-hierarchy selector genuinely draws from multiple depths. Under the matched setting, 36.69% of the selected states come from L1--4, 38.74% from L5--8, and 24.56% from L9--12. Hence, the full candidate pool does not collapse to selecting only deep-layer states; shallow, intermediate, and deep representations all contribute.
Taken together, these results suggest that IVST benefits from the combination of multi-layer candidate construction and selective retention, rather than from layer coverage or candidate quantity alone.
Q1. Explicit teacher–student pairs and loss coefficients
Thank you for recognizing the image-conditioned semantic construction and the empirical effectiveness of SWR. We address the specification and validation questions below, including new controlled experiments.
Explicit teacher–student pairs and loss coefficients
Let R denote the fused pre-BN representation and T the textual representation. Using (student, teacher) as the convention, the implemented pair set is
P={(RR,TT),(TT,RR),(RT,TR),(TR,RT)}.
Thus, the supervision is reciprocal: TT supervises RR and RR supervises TT; TR supervises RT and RT supervises TR. The four terms have equal weight 1/4. For each direction, its teacher selects the Top-H non-self identities and determines their ordering, and the student is evaluated on exactly those candidate identities. Different teacher directions can select different candidate lists. ListMLE is averaged over anchors and list positions before averaging the four directional terms.
Prototypes are obtained by averaging raw observations within each identity and then applying L2 normalization. Relation scores are cosine similarities divided by tau. The teacher scores and the discrete candidate selection/order are detached. In the batch version, gradients propagate through both current-batch arguments of the student relation. In the memory version, candidate prototypes are detached, so gradients propagate through current anchors. Freezing the text encoder parameters still permits gradients through its input to the pseudo-word construction modules.
To make the coefficients explicit, let
\qquad \mathcal L_{\mathrm{orth}}=\frac{1}{B K^2}\sum_{i=1}^{B}\|\widetilde A_i\widetilde A_i^\top-I\|_F^2.$$ The implemented objective is $$\mathcal L=\mathcal L_{\mathrm{ID}}+\mathcal L_{\mathrm{triplet}}+0.2\mathcal L_{\mathrm{sup}}+\mathcal L_{\mathrm{orth}}+\lambda_{\mathrm{IRR}}(e)\mathcal L_{\mathrm{IRR}}.$$ With H=8 and tau=0.07, the IRR schedule is $$\lambda_{\mathrm{IRR}}(e)=0.1\min\{1,\max\{0,(e-5)/5\}\}.$$ Thus, it is zero for epochs 1–5, increases linearly during epochs 6–10, and is 0.1 thereafter. The orthogonality equation above explicitly includes the mean reduction used by the code. Equivalently, its coefficient would be 1/K² if the loss were defined as the batch-mean unnormalized Frobenius error. Market-1501 uses batch candidates; MSMT17 uses a detached identity memory of capacity 256. Current prototypes are enqueued before memory scoring and self-identity candidates are excluded. We will correct the enqueue position in Algorithm 1 and explicitly state these directions, reductions, and coefficients.