acceptodds
Desk rejected at ICLR 2027

Semantics Within, Relations Between: Structuring Vision-Language Representations for Person Re-Identification

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent CLIP-based person re-identification (ReID) methods increasingly exploit language-aligned semantics through textual prompts, descriptions, or image-derived pseudo-words. However, image-derived semantics are typically constructed only after visual encoding, limiting their interaction with visual representation formation, while conventional sample-level objectives do not explicitly organize the relative ordering of hard competing identities. These limitations can leave fine-grained visual evidence underexploited and ambiguous identity neighborhoods insufficiently structured for retrieval. We present SWR, a framework that addresses these two limitations through Semantics Within and Relations Between. Its In-Encoder Visual Semantic Tokenization (IVST) introduces learnable semantic tokens into the visual Transformer, allowing them to evolve jointly with image patches, and collects their intermediate states across visual layers to construct image-conditioned pseudo-word representations. Its Identity-Relational Ranking (IRR) aggregates observations into identity prototypes and applies listwise supervision over hard identity neighborhoods in the fused pre-BN representation. On Market-1501 and MSMT17, SWR improves the PromptSG baseline by 2.0 and 1.7 mAP points, reaching 96.6% and 88.9% mAP, respectively. It further achieves 72.7% mAP on Occluded-Duke and 86.0% on CUHK03-D. Compared with IADT under matched training settings, SWR improves mAP by 0.4 and 0.5 points on Market-1501 and MSMT17 while using 29.1% fewer parameters and reducing training time by 29.6–33.6%. Controlled analyzes further validate multi-layer state selection, relational supervision location, hard-identity mining, and retrieval-boundary behavior.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
person re-identification, vision-language learning, CLIP, semantic representation learning, relational ranking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_enz3OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SWR, a CLIP-based framework for image-based person re-identification that combines image-conditioned semantic construction with identity-level relational supervision. Its In-Encoder Visual Semantic Tokenization (IVST) inserts learnable tokens into the visual Transformer, selects their intermediate states across layers, and converts these states together with a global identity representation into pseudo-words for the frozen text encoder. An inherited cross-attention module integrates the resulting textual and visual representations. Identity-Relational Ranking (IRR) then aligns relative orderings among hard competing identity prototypes using ListMLE, optionally expanding the candidate pool with detached cross-batch memory. The paper reports good retrieval performance across standard and challenging benchmarks, provides several component ablations, and demonstrates improved training efficiency relative to its IADT reimplementation.

Critical strengths
  1. A coherent design for constructing image-conditioned pseudo-words. IVST uses states formed through interactions between learnable tokens and image patches throughout the visual hierarchy. Combining these states with the global identity pseudo-word provides a plausible way to enrich language-side representations, and the component ablation supports the usefulness of both sources.

  2. Broad empirical effectiveness without additional textual annotations. The reported results are good across Market-1501, MSMT17, Occluded-Duke, and CUHK03-D, covering several distinct retrieval conditions. Achieving these results without externally generated captions or manually annotated attributes supports the framework’s practical value.

Critical weaknesses
  1. The actual IRR objective remains incompletely specified. Figure 1 suggests a textual-relation target, but neither Section 3.3 nor Appendix A.3 explicitly enumerates the teacher-student pair set P\mathcal{P}. Several loss coefficients are also left unspecified. These omissions affect which representations supervise and update one another, preventing faithful reproduction and a precise assessment of the central relational mechanism.

  2. The reliability of the teacher ordering is insufficiently validated. IRR derives target orders from learned representations rather than relevance labels among competing negative identities. Its hard sorting discards teacher score gaps, allowing nearly tied similarities to impose an unqualified total order. The alignment and retrieval-boundary diagnostics show useful downstream effects, but do not establish the stability or informativeness of the transferred orders.

Questions for authors
  1. Please enumerate P\mathcal{P}: which of RR, TT, RT, and TR supervises which relation, and are any directions reciprocal? Please also provide the unspecified loss coefficients.

  2. How stable are the hard-identity target orders across different observations, augmentations, and training updates, especially for nearly tied scores? Do learned target orders outperform shuffled orders over exactly the same candidate identities?

  3. Appendix B.2 states that the remaining IVST configuration is unchanged. However, with M=2M=2, either four-layer restricted pool contains only eight states, fewer than the default K=16K=16. Reducing KK or increasing MM would change the comparison, so Table 5 does not yet establish the benefit of full-hierarchy collection independently of semantic capacity. What are MM and the actual retained KK for each Table 5 variant? Does full-hierarchy collection remain beneficial when the number of pseudo-words is matched?

  4. The IVST–IRR complementarity is not clearly demonstrated. Table 4 does not specify whether IRR is enabled, and its combined MSMT17 result matches full SWR. Moreover, the Baseline+IRR in Table 10 outperforms full SWR on Occluded-Duke. The above requires a reasonable explanation.

6 comments
AuthorsOpenReview ↗

Q1. Explicit teacher–student pairs and loss coefficients

Thank you for recognizing the image-conditioned semantic construction and the empirical effectiveness of SWR. We address the specification and validation questions below, including new controlled experiments.

Explicit teacher–student pairs and loss coefficients

Let R denote the fused pre-BN representation and T the textual representation. Using (student, teacher) as the convention, the implemented pair set is

P={(RR,TT),(TT,RR),(RT,TR),(TR,RT)}.\mathcal P=\{(RR,TT),(TT,RR),(RT,TR),(TR,RT)\}.

Thus, the supervision is reciprocal: TT supervises RR and RR supervises TT; TR supervises RT and RT supervises TR. The four terms have equal weight 1/4. For each direction, its teacher selects the Top-H non-self identities and determines their ordering, and the student is evaluated on exactly those candidate identities. Different teacher directions can select different candidate lists. ListMLE is averaged over anchors and list positions before averaging the four directional terms.

Prototypes are obtained by averaging raw observations within each identity and then applying L2 normalization. Relation scores are cosine similarities divided by tau. The teacher scores and the discrete candidate selection/order are detached. In the batch version, gradients propagate through both current-batch arguments of the student relation. In the memory version, candidate prototypes are detached, so gradients propagate through current anchors. Freezing the text encoder parameters still permits gradients through its input to the pseudo-word construction modules.

To make the coefficients explicit, let

\qquad \mathcal L_{\mathrm{orth}}=\frac{1}{B K^2}\sum_{i=1}^{B}\|\widetilde A_i\widetilde A_i^\top-I\|_F^2.$$ The implemented objective is $$\mathcal L=\mathcal L_{\mathrm{ID}}+\mathcal L_{\mathrm{triplet}}+0.2\mathcal L_{\mathrm{sup}}+\mathcal L_{\mathrm{orth}}+\lambda_{\mathrm{IRR}}(e)\mathcal L_{\mathrm{IRR}}.$$ With H=8 and tau=0.07, the IRR schedule is $$\lambda_{\mathrm{IRR}}(e)=0.1\min\{1,\max\{0,(e-5)/5\}\}.$$ Thus, it is zero for epochs 1–5, increases linearly during epochs 6–10, and is 0.1 thereafter. The orthogonality equation above explicitly includes the mean reduction used by the code. Equivalently, its coefficient would be 1/K² if the loss were defined as the batch-mean unnormalized Frobenius error. Market-1501 uses batch candidates; MSMT17 uses a detached identity memory of capacity 256. Current prototypes are enqueued before memory scoring and self-identity candidates are excluded. We will correct the enqueue position in Algorithm 1 and explicitly state these directions, reductions, and coefficients.
AuthorsOpenReview ↗

Q4. Clarifying IVST–IRR Complementarity and Correcting Table 4

IVST–IRR complementarity and the Occluded-Duke result

Thank you for pointing this out. We first found an error in Table 4. The MSMT17 result of the combined IVST configuration was mistakenly copied from full SWR. The correct IVST-only result is 88.3/92.8 (mAP/Rank-1), not 88.9/93.3. Table 4 does not enable IRR, and we will explicitly state this in the revised caption and text.

With the corrected numbers, IVST alone obtains 96.4/98.0 on Market-1501 and 88.3/92.8 on MSMT17, whereas full SWR reaches 96.6/98.3 and 88.9/93.3, respectively. Thus, adding IRR on top of IVST gives consistent gains on both standard benchmarks, particularly +0.6 mAP / +0.5 Rank-1 on MSMT17.

We view the complementarity mainly from their different roles. IVST improves within-instance representation formation by constructing image-conditioned pseudo-words from multi-layer visual states, whereas IRR improves between-identity organization by applying identity-level listwise supervision to hard competing identities. Therefore, IVST changes the semantic representation used by the model, while IRR regularizes how identities are relatively organized in the retrieval space.

The Occluded-Duke result reveals an important boundary of this complementarity. Baseline+IRR reaches 73.5/77.1, while full SWR obtains 72.7/75.4. We do not interpret complementarity as requiring strictly additive gains on every domain. IVST is image-conditioned and therefore depends on the visual evidence available in each observation. Under severe occlusion, different images of the same identity may expose substantially different or incomplete regions, making single-image semantic construction less reliable. IRR, in contrast, aggregates observations at the identity level and regularizes relations among competing identities, which can be more robust when individual observations are incomplete.

We will revise the manuscript to make two points explicit: (1) IVST and IRR provide complementary improvements on the standard ReID benchmarks, where their combination gives the strongest results; and (2) under severe occlusion, their contributions are domain-dependent rather than strictly additive, with IRR showing stronger robustness to incomplete visual evidence.

AuthorsOpenReview ↗

Q3. Layer-range comparison under matched semantic capacity

Q. Layer-range comparison and matched semantic capacity

Thank you for raising this point. We first clarify the actual configuration and purpose of Table 5.

The experiment was designed to examine what happens when the source layers of VST states are restricted. With M=2M=2, the restricted layer ranges provide relatively small candidate pools, so all available states were used directly:

  • L1--4: 8 candidates, all 8 used;
  • L9--12: 8 candidates, all 8 used;
  • L1--4 + L9--12: 16 candidates, all 16 used.

For the full L1--12 hierarchy, however, M=2M=2 produces 24 candidate states, and the default Top-1616 selection was applied. This configuration obtains 96.4 mAP / 98.0 Rank-1. We additionally evaluated using all 24 states without Top-KK selection, which gives 96.2 / 97.8. Thus, simply exposing more VST states is not sufficient; the gain of the default configuration comes from constructing a broader multi-layer candidate pool and selectively retaining useful states.

To directly address the reviewer’s fixed-capacity question, we further performed a matched experiment on Market-1501 with M=2M=2, K=8K=8, full IRR, the same seed, and the same 60-epoch training protocol:

Layer rangeCandidatesRetained KKBest mAPRank-1
L1--48896.598.0
L9--128896.497.9
L1--4 + L9--1216896.497.8
L1--1224896.497.9

At the strictly matched K=8K=8 budget, broader layer coverage alone does not yield an additional accuracy gain. This indicates that the default IVST gain should not be attributed simply to using more visual layers.

Our default setting uses K=16K=16, following the empirical pseudo-word budget adopted in IADT and further supported by the MM--KK sensitivity analysis in Appendix B.3 / Fig. 5. Under the complete SWR setting, full-hierarchy M=2,K=16M=2,K=16 reaches 96.6 mAP / 98.3 Rank-1, the strongest configuration in our experiments. A four-layer range with M=2M=2 contains only eight states, so enforcing K=16K=16 there would necessarily require increasing MM and would change another capacity factor.

We also examined whether the full-hierarchy selector genuinely draws from multiple depths. Under the matched M=2,K=8M=2,K=8 setting, 36.69% of the selected states come from L1--4, 38.74% from L5--8, and 24.56% from L9--12. Hence, the full candidate pool does not collapse to selecting only deep-layer states; shallow, intermediate, and deep representations all contribute.

Taken together, these results suggest that IVST benefits from the combination of multi-layer candidate construction and selective retention, rather than from layer coverage or candidate quantity alone.

Reviewer_G6X4OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SWR (Semantics Within, Relations Between), a vision-language framework for person re-identification that combines semantic representation learning with identity-level relational modeling. IVST integrates pseudo-word construction into the visual Transformer, using intermediate visual semantic token states from multiple layers to generate image-conditioned semantic representations. IRR aggregates retrieval and textual identity prototypes, constructs within-space and cross-space relations, and applies listwise ranking to organize hard competing identities in the fused pre-BN space. The components are complementary and clearly motivated. Experiments demonstrate strong performance on Market-1501, MSMT17, Occluded-Duke, and CUHK03-D, while reducing parameters and training cost compared with IADT.

Critical strengths
  1. IVST and IRR provide a coherent framework for semantic learning and identity relations. IVST simplifies post-hoc inversion by integrating VSTs into visual encoding and reusing their intermediate states.

  2. IRR introduces a meaningful identity-level listwise ranking formulation. Comparisons with alternative objectives support using ListMLE, while supervision-location ablations and correlation diagnostics justify applying IRR to fused pre-BN features.

  3. The evaluation demonstrates competitive performance across standard, large-scale, detector-noisy, and occluded ReID benchmarks.

Critical weaknesses
  1. Fig. 1 should correct the duplicated TT labels and typos, and clearly identify the teacher–student relations and the source of the Top-H ranking.

  2. The paper should explicitly distinguish IRR’s identity-level listwise ranking from triplet loss’s sample-level positive–negative constraints, while, key ranking-objective comparisons and layer-range ablations should be highlighted in the main text rather than confined to the appendix.

  3. The paper should clarify whether prototype aggregation is consistently used across datasets, with only the candidate source differing.

Questions for authors
  1. In Fig. 1, are the two “TT (target)” ranking rows in the IRR panel correctly labeled? Should one of them instead denote a student relation such as RR/RT/TR? Also, which relation is used to generate the Top-H hard-identity list shown on the right?

  2. Could the authors clarify the conceptual difference between IRR and hard triplet loss? In particular, is the key distinction sample-level pair/triplet ordering versus identity-level listwise ordering over an entire hard neighborhood, together with cross-representation relational transfer?

  3. Are identity prototypes constructed for all datasets, with the difference being that MSMT17 additionally uses Memory-256 while Market-1501 only uses current-context prototypes? If so, I suggest stating this explicitly.

  4. Have the authors analyzed the consistency of image-conditioned pseudo-words produced by different images of the same identity? Since viewpoint and occlusion may cause different VST states to be selected, I wonder whether textual prototype aggregation is sufficient to stabilize this variation.

  5. Is there a principled criterion for when cross-batch prototype memory is needed? For example, does it mainly depend on the ratio between the number of training identities and the number of identities exposed by the P×K sampler?

Reviewer_3fsZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses two overlooked limitations in existing CLIP-based pedestrian re-identification: (1) image-derived semantics are typically constructed only after visual encoding has been completed, which limits the interaction between semantic tokens and the process of forming visual representations; and (2) traditional sample-level objectives do not explicitly organize the relative ordering relationships among competing identities, resulting in underutilization of fine-grained visual evidence and insufficient structuring of identity neighborhoods. We propose two solutions: first, directly inserting a small number of learnable visual semantic tokens into the visual Transformer so that they participate in self-attention computation within the encoder alongside image patches; and second, aggregating multiple observations of the same identity into a retrieval prototype and a text prototype, constructing four directional relationship scores, and applying list-sorting supervision to the hard identity neighborhood. Experiments demonstrate that these approaches achieve favorable results.

Critical strengths

(1) By directly inserting learnable semantic tokens into the visual Transformer so that they interact with patch tokens within self-attention, this design breaks away from the previous paradigm in which pseudo-word methods construct semantics only after encoding. A cross-layer state collection and selection mechanism enables language-side representations to acquire information from visual hierarchies at different depths, rather than being confined to the output of the final layer. This approach to "semantic internalization" demonstrates considerable innovation. (2) Replacing sample-level metric constraints with list-based sorting is a reasonable approach—ReID is fundamentally a ranking task, and explicitly supervising the ranking of competing identities aligns more closely with evaluation metrics. The introduction of four directional relational spaces provides a rich set of relational perspectives. The prototype memory bank addresses the issue of insufficient candidate identities within batches; furthermore, the paper provides compelling evidence for its necessity through quantitative analysis (only 5.66–9.76% of Top-8 hard identities are found within a single batch). (3) The paper conducts comprehensive evaluations on four standard ReID benchmarks, covering standard scenes, large-scale scenes, occlusion scenes, and detection-noise scenes. Its comparative results against SOTA methods are competitive, and comparisons in training efficiency further demonstrate the practical advantages of the proposed method.

Critical weaknesses

(1) While a comparison between the paper and methods such as PromptSG, CLIP-SCGI, and IADT is meaningful, it lacks a comparison with more recent CLIP-based ReID methods, such as MP-ReID (AAAI '24,95.5% mAP on Market-1501) and FLaN-Net (IJCAI' 25,90.5% mAP). Furthermore, in Table 1, the results for MP-ReID and FLaN-Net on MSMT17 and Occluded-Duke are marked "—"; the authors did not provide the re-implementations of these methods on MSMT17 for a fair comparison. (2) IVST uses the raw dot product as the scoring criterion for candidate states, but the paper does not provide sufficient theoretical or experimental justification for why the dot product was chosen over cosine similarity or a learnable similarity function. Although Appendix B.4 uses feature norm-controlled experiments to rule out the possibility that feature magnitude alone explains the layer's preference, a systematic comparison between the dot product and cosine similarity within the IVST framework remains absent. Additionally, the dot product's sensitivity to feature magnitude may lead to unstable selection behavior across different datasets. (3) The paper does not provide an analysis of retrieval failure cases. Error analysis can help readers understand the scenarios in which the method performs poorly, which is highly valuable for assessing the method's practical limitations and guiding future improvements. It is recommended that case studies or discussions of failure modes be added, particularly regarding error analysis on challenging datasets such as Occluded-Duke. (4) As a relational regularizer during training, IRR does not directly participate in the retrieval process during the inference stage. This means that the method's core contribution—relational ranking supervision—is not explicitly leveraged during inference. Although the authors argue that IRR's effectiveness is concentrated on ambiguous queries and can propagate to the final retrieval boundary, designing a lightweight relational re-ranking module for inference could potentially further improve performance; this possibility is not discussed in the paper. (5) The choice of prototype memory size (256 entries) lacks sensitivity analysis. The paper reports only the results for Memory-256; it does not discuss the impact of different memory sizes (e.g., 128 or 512) on performance, nor does it analyze whether the FIFO eviction strategy for old prototypes in the memory is optimal. For datasets of different sizes, the optimal memory size may vary; however, the paper does not address this issue.

Questions for authors

Q1:Has the author systematically compared the differences among dot product, cosine similarity, and learnable similarity in state selection? Could the dot product's sensitivity to feature magnitude lead to unstable selection behavior across different datasets or different training stages? Is there any theoretical analysis that supports the advantage of the dot product under this setup? Q2:IRR's list-based ranking supervision applies only during the training stage; during inference, relational ranking information is not directly utilized. Has the author considered introducing a lightweight relational reordering module during the inference stage (e.g., by leveraging the relational structure learned during training)? If so, what magnitude of performance improvement would be expected? Would the resulting increase in inference latency remain within an acceptable range? Q3:The prototype memory employs a FIFO strategy to evict old prototypes, and prototypes in the memory do not receive gradients. Has the author explored alternative memory update strategies (e.g., exponential moving averages based on similarity weights, or importance sampling based on loss gradients)? Could a learnable memory (that permits gradient backpropagation) further enhance IRR performance?

7 comments
AuthorsOpenReview ↗

Q2. Inference-time relational re-ranking

Q2. Inference-time relational re-ranking

We considered whether the relational structure learned by IRR could be further exploited during inference. We therefore added a lightweight re-ranking control on Market-1501 using the fixed SWR checkpoint, without retraining the model.

For each query, we use its textual representation to retrieve the Top-8 gallery observations in the textual space, average their retrieval features to form a relational descriptor, and use this descriptor to re-score only the original Top-100 candidates. We also include a simpler text-score fusion control.

The results are:

Inference schememAPRank-1
Original retrieval96.698.3
Text-score fusion96.698.3
Relational re-ranking96.598.1

Neither direct text fusion nor relational re-ranking improves the SWR result. As an additional control, applying the same relational re-ranking to the model without IRR slightly increases mAP from 96.4 to 96.5 but decreases Rank-1 from 98.0 to 97.7, indicating that the re-ranking itself is not a generally beneficial post-processing trick.

These results suggest that IRR is more effective as a training-time representation regularizer than as an explicit inference-time ranking module. The relational supervision is absorbed into the learned retrieval geometry, allowing SWR to retain the standard retrieval pipeline at inference. We therefore keep IRR training-only rather than introducing an additional test-time re-ranking stage.

AuthorsOpenReview ↗

Q1. Dot product, cosine similarity, and learnable similarity for VST selection

Q. Dot product, cosine similarity, and learnable similarity for VST selection

Thank you for raising this question. We conducted a systematic comparison of five selection strategies under the same training configuration: raw dot product, cosine similarity, learnable similarity with straight-through (ST) estimation, dot product with ST, and cosine similarity with ST. All experiments use the same seed, 60-epoch schedule, batch size, M=2M=2, 12 visual layers, Top-16 selection, and IRR configuration.

SelectorMarket mAPMarket Rank-1MSMT mAPMSMT Rank-1
Raw dot product96.698.388.993.3
Cosine similarity96.698.288.592.7
Learnable similarity + ST96.698.287.992.5
Dot product + ST96.598.088.692.8
Cosine similarity + ST96.598.088.492.7

Under the current setting, the original dot-product selector remains the strongest and simplest choice. Cosine similarity, which explicitly removes feature-magnitude information, does not improve retrieval, and the learnable similarity also does not outperform the raw dot product. The learnable weights do change during training, confirming that the learnable scorer is active rather than degenerate.

We further examined whether the magnitude sensitivity of dot product leads to unstable Top-KK selection. For a candidate state ziz_i and query qq,

si=q⊤zi=∥q∥ ∥zi∥cos⁡θi.s_i=q^\top z_i=\|q\|\,\|z_i\|\cos\theta_i.

For a fixed image, ∥q∥\|q\| is shared by all candidates and therefore does not affect their relative ranking. Candidate magnitude can affect the ranking, so we directly tested its stability by independently rescaling candidate vectors while preserving their directions. With a radial perturbation magnitude of ϵ=0.1\epsilon=0.1, the Top-16 overlap remains 99.52% on Market-1501 and 98.83% on MSMT17; even at ϵ=0.2\epsilon=0.2, the overlap is still 98.60% and 97.42%, respectively. Thus, although magnitude participates in dot-product scoring, the learned Top-KK selection is highly robust to substantial magnitude perturbations.

The selection also becomes very stable in later training. On Market-1501, the dot-product Top-16 overlap between epochs 40/60 and epoch 50 is 99.87%/99.86%. On MSMT17, the overlap between epochs 40/50 and epoch 60 is 99.89%/99.94%. This indicates that the selector converges to a stable candidate set once the visual representation has matured.

Interestingly, dot product and cosine similarity select only about 86.41% and 85.53% common Top-16 states on Market-1501 and MSMT17, respectively. This shows that magnitude information does change which states are preferred, although removing it does not improve retrieval in our current experiments. We view this difference as an interesting direction for further study, particularly for understanding when candidate magnitude carries useful confidence information and when direction-only similarity may be preferable.

Overall, the experiments indicate that raw dot product is the most effective selector in our current setup, while remaining highly stable to magnitude perturbations and across later training stages. The analysis above explains why magnitude sensitivity does not necessarily imply unstable ranking; the empirical comparison, rather than a claim of universal theoretical superiority, supports our use of dot product here.

AuthorsOpenReview ↗

Q3. Alternative prototype-memory update strategies

Q3. Alternative prototype-memory update strategies

Thank you for this suggestion. We compared five memory strategies on MSMT17 under the same memory capacity (256), seed, 60-epoch schedule, and training configuration:

  1. FIFO: refresh the prototype when an identity reappears and evict the oldest identity when full;
  2. EMA: update repeated identities using a normalized 0.9/0.10.9/0.1 old/new prototype mixture;
  3. Similarity-weighted EMA: adapt the EMA coefficient according to the agreement between the old and new retrieval/textual prototypes;
  4. Gradient-priority retention: use the gradient magnitude of the fused representation as an identity-importance signal and preferentially retain higher-importance identities;
  5. Learnable memory: register the retrieval and textual prototype banks as trainable parameters, allowing the stored slots to receive relational gradients.

All five runs reached their best mAP at epoch 60:

Memory strategymAPRank-1
FIFO88.993.3
EMA88.593.1
Similarity-weighted EMA88.693.1
Gradient-priority88.793.2
Learnable memory88.993.4

The more complex update strategies do not consistently outperform FIFO. In particular, both EMA variants improve the consistency between cached and current prototypes, but this does not translate into better retrieval. At epoch 60, the retrieval-prototype memory/current mismatch decreases from 0.0349 with FIFO to 0.0282 with EMA and 0.0263 with similarity-weighted EMA; the textual prototypes show the same trend. This suggests that making cached prototypes closer to the current representation is not by itself sufficient to improve IRR, whose memory also serves to provide a broad and useful hard-identity candidate context.

The learnable-memory experiment directly addresses whether gradients through the memory can help. We use trainable retrieval and textual banks of size 256×512256\times512, and verified that both banks receive non-zero Adam updates. It achieves the same 88.9 mAP as FIFO and a small +0.1 Rank-1 improvement. Thus, learnable memory shows some potential, but the gain is currently marginal relative to the additional parameters, optimizer states, and memory-management complexity.

Overall, these experiments indicate that IRR benefits mainly from maintaining a sufficiently broad cross-batch candidate set, while more sophisticated smoothing, gradient-based retention, or learnable prototype updates do not yet provide a clear advantage over the simple FIFO design. The small gain from learnable memory is nevertheless an interesting direction for further investigation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.