acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Label-aware Spatial-Frequency Learning for Unsupervised Multi-modal Re-Identification

Abstract

Unsupervised multi-modal re-identification (UMM-ReID) aims to learn identity-discriminative representations from multi-modalities by constructing pseudo-labels through clustering. However, it is difficult to obtain reliable and globally consistent pseudo-labels from multi-modal representations, and global pseudo-label supervision itself cannot clearly guide the representation learning and strengthen the focus on identity-discriminative regions. To address these issues, we propose Contrastive Label-aware Spatial-frequency Learning, which is a framework that alternates between global pseudo-label and minibatch representation learning. At the beginning of each epoch, we use adaptive spectral clustering and spectral gap-MDL criterion to generate global adaptive pseudo-labels for subsequent minibatch optimization. Based on pseudo-labels, cross-modal contrastive learning selectively adds reliable intra-modal negative relations to strengthen representation learning. In addition, we use pseudo-labels to guide spatial discrimination, and combine complementary frequency cues to enhance the learning of discriminative regions and suppress background noise. Experiments on real datasets fully demonstrate the effectiveness of this method.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
label-aware, unsupervised learning, multi-modal, Re-ID

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tUvXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses unreliable pseudo-labels, modality discrepancies, and the difficulty of learning identity-discriminative regions in unsupervised multi-modal re-identification. It proposes a Contrastive Label-aware Spatial-Frequency Learning framework. At the beginning of each epoch, adaptive spectral clustering and the SG-MDL criterion estimate the number of clusters and generate global pseudo-labels and cluster prototypes. During the epoch, pseudo-labels guide cross-modal contrastive learning, while pseudo-label confidence is used to select intra-modal negative pairs and improve identity discrimination. The label-aware spatial-frequency learning module also uses pseudo-labels and frequency cues to guide spatial discrimination. Experiments, ablation studies, and visualizations on datasets including RGBNT201 demonstrate the effectiveness and competitiveness of the method.

Critical strengths
  1. The experimental evaluation is thorough, covering multiple datasets and showing strong performance. The ablation studies also demonstrate the contributions of the individual modules.
  2. The framework is comprehensive, and the method’s formulas and mathematical rationale are clearly presented. The paper also includes attention visualizations, analyses of cluster-number changes and feature-distance distributions, runtime analysis, and anonymous GitHub code, which help readers understand and verify the method.
Critical weaknesses

The frequency-domain component in Section 3.5, “Label-aware SF Learning,” is not explained in sufficient detail. In particular, the paper does not clearly explain the rationale or provide enough formulas for how Haar wavelets and high-frequency subbands provide complementary texture cues.

Questions for authors
  1. The paper analyzes the sensitivity of some hyperparameters, but the choices of key parameters such as (q), (α\alpha), and (β\beta) are not sufficiently justified. Could the authors provide sensitivity analyses for these parameters?
  2. Could the authors provide more implementation details for the frequency-domain component, including the selection of wavelet subbands, the relevant equations, and the normalization method?
1 comment
AuthorsOpenReview ↗

answer for reviewer tUvX

Thank you for your approval of the framework and experiment of our paper. Next, I will reply to your questions (the blue font in the updated pdf is our updated part):

  1. Thank you for your question about sensitivity analysis of parameters.. For the choice of q, we found in the references that the author mentioned that the general choice of q would not exceed 10, so we only set the threshold to 5, and did not conduct sensitivity analysis. The sensitivity analysis was supplemented in the appendix. As a balance coefficient, \alpha is generally set to 1/2, and the sensitivity analysis is supplemented in the appendix. According to the spectrum constructed by the current features, the \beta is recalculated at the beginning of each round without being fixed, and the specific updating rules have been supplemented in the appendix.

  2. Thanks to the reviewer for the details of frequency domain calculation. The specific implementation details have been supplemented in the appendix.

Reviewer_L9gWOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores unsupervised multimodal re-identification by improving pseudo-label reliability and learning identity-discriminative representations across modalities. It first identifies limitations of clustering methods such as DBSCAN and K-Means when handling complex multimodal feature distributions and introduces spectral clustering. To address its sensitivity to Gaussian-kernel distance scales and reliance on a predefined cluster count, the method uses an adaptive affinity matrix and a spectral gap–minimum description length criterion (SG-MDL), respectively, to generate global pseudo-labels and cluster prototypes. Network training then combines three losses: a cross-modal contrastive loss incorporating confidence-weighted intra-modal negatives; a prototype-label loss encouraging consistency between samples and their assigned prototypes; and a spatial–frequency loss selecting and learning local features using pseudo-label-guided spatial scores and Haar-wavelet frequency cues. Training alternates between global clustering and minibatch optimization. The method is evaluated through performance comparisons, ablations, and visualizations across six datasets.

Critical strengths
  1. The paper explores adaptive spectral clustering, automatic cluster-number estimation, and label-aware spatial–frequency learning for unsupervised multimodal re-identification. Bringing pseudo-label generation and local representation learning into one framework offers potentially useful directions for further investigation in this task.

  2. Evaluation spans six datasets covering pedestrians, vehicles, and different modality combinations. The reported retrieval results are competitive across multiple datasets; for example, RGBNT201 mAP reaches 68.33%, exceeding the listed CHCR result by 9.02 percentage points, providing encouraging empirical evidence.

Critical weaknesses
  1. The motivation behind the method is unclear, making it difficult to understand why the proposed designs are needed. Additionally, the paper repeatedly emphasizes the global-to-minibatch pipeline, although this is a common training paradigm in unsupervised ReID.

  2. The paper frequently refers to experimental configurations using non-descriptive labels such as A1 and A2. Readers must repeatedly consult their definitions to understand the differences, making the experimental results difficult to follow.

  3. The main comparison tables do not specify each method’s backbone, and the paper does not identify its ViT variant or pretraining source, making comparison fairness difficult to assess. Including unsupervised visible–infrared person re-identification baselines is reasonable given the limited literature on unsupervised multimodal re-identification, but their original task categories should be explicitly indicated.

Questions for authors
  1. The paper argues that clustering methods such as K-Means and DBSCAN are sensitive to nonconvex structures and nonuniform densities in multimodal representations. Could you provide targeted experiments demonstrating that these issues arise in the studied task and that the proposed adaptive spectral clustering method effectively mitigates them?

  2. A1 in Table 2 changes both the clustering scope and algorithm, making their effects on performance difficult to distinguish and preventing isolation of the proposed clustering method’s contribution. Furthermore, Figure 6 does not clearly specify whether fixed-k and DBSCAN use minibatch or global clustering, leaving the experimental setup ambiguous.

  3. Please clarify whether the patch-level frequency scores described after Eq. (16) in Section 3.5 are computed through a fixed image-processing procedure that depends solely on the input images and contains no learnable parameters. Additionally, how are training augmentations such as random cropping and flipping handled to ensure alignment between these scores and the patches input to the ViT?

  4. Please specify each method’s backbone and pretraining source in the comparison tables to clarify comparison fairness and distinguish gains from the backbone versus the proposed method.

1 comment
AuthorsOpenReview ↗

answer for reviewer L9gW

Thank you for affirming the innovation and experimental sufficiency of our paper, and then I will reply to your questions (the blue font in the updated pdf is our updated part):

  1. In fact, Figure 2 in this paper is a t-SNE diagram comparing the traditional clustering method with our method. We choose DBSCAN and the traditional Gaussian kernel SC method as a comparison. Figure 2 shows that our method can effectively solve the problems of non-convex structure and non-uniform density.

  2. A1 you mentioned is not an ablation method with a fixed variable, which we didn't notice. In the updated paper, we added a set of comparison groups A1-G of DBSCAN algorithm using global pseudo-labels. In addition, both fixed-k and DBSCAN in figure 6 use global clustering, which is our negligence in writing the appendix. This experimental setup is supplemented in the appendix.

3.Thank the reviewers for pointing out this detail that needs clarification. Yes, the frequency score is calculated by a fixed image processing flow, which does not contain learnable parameters and only depends on a corresponding set of input images. We only sample each image once for random cropping and flipping, and use the same geometrically enhanced image for frequency branch and ViT at the same time. The reconstructed frequency response graph keeps the same spatial size as the enhanced image, and then the frequency score of each region is calculated according to the same size of ViT.

4.Thank the reviewers for asking the benchmark question of the comparison method. We supplement the backbone network of comparison methods with footnotes in the relevant experimental tables.

5.Thank the reviewer for the problem of global to mini-batch paradigm mentioned in the weakness. We agree that it is not a new training paradigm to generate pseudo-labels globally and optimize minibatch. The contribution of this paper mainly lies in the combination of adaptive SC and SG-MDL, and how pseudo-labels are used in confidence-weighted contrastive learning and space-frequency learning. We will modify the relevant expressions in the introduction section to avoid taking the general training process itself as an independent innovation.

Reviewer_HJh3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses unsupervised multi-modal re-identification using adaptive spectral clustering and SG-MDL to estimate the number of identities and generate global pseudo-labels. It combines confidence-aware cross-modal contrastive learning, prototype label learning, and label-aware spatial-frequency learning. Experiments cover six person or vehicle re-identification datasets.

Critical strengths

The experiments include ablations supporting the contributions of confidence weighting and frequency cues. The method reports substantial mAP gains over the compared unsupervised baselines on RGBNT201 and MSVR310, with additional gains on RGBNT100 in the appendix. These results support the framework’s effectiveness in exploiting multi-modal identity information.

Critical weaknesses

1.It averages patch scores and combines wavelet outputs across modalities, but Section 4.1 only explains how shared pseudo-labels determine cross-modal positive and negative pairs. How spatial locations correspond across different images remains unclear.

2.A1 in Table 2 changes both the clustering scope, from global to minibatch, and the clustering algorithm, from adaptive spectral clustering to DBSCAN. Thus, this comparison alone cannot attribute the performance gap specifically to the proposed clustering algorithm.

Questions for authors

1.Is S symmetrized before constructing the Laplacian? What is the candidate range of k in SG-MDL, and what objective and update rule are used to learn \beta?

2.How are spatial scores averaged and wavelet outputs combined on datasets without explicit cross-modal shot correspondence?

3.Do the GT identity counts in Figure 5 refer to the training splits? Please clarify the training protocol and data split for each dataset.

1 comment
AuthorsOpenReview ↗

answer for reviewer Hjh3

Thank you for your approval of our paper, and then I will reply to your questions (the blue font in the updated pdf is our updated part):

  1. Actually, S is asymmetric. In the traditional SC, the similarity is determined by Gaussian kernel, while our method is determined by the neighbor value of data points. j is the neighbor of i, but i may not be the neighbor of j. I checked and found that there was a step S = (S+S^T)/2 in the code that was not put into the paper, and now the corresponding description has been added to the article. Because it is unsupervised learning, the value of k cannot depend on the number of real identities, so we usually choose the range of [2,n-1] in calculation, where n is the number of data. The update rule of \beta is determined by the ratio of the change amplitude of the first term and the second term of SG-MDL, and the specific formula derivation is shown in the updated appendix.

  2. Among the six data sets used, the datasets of non-shot-level corresponding modes include sysu-mm01 and hitsz-vcm. We introduce these two datasets to observe the robustness of the method to this type of datasets, so we don't introduce the experimental settings in detail in the appendix. Without clear shot correspondence, the spatial scores are not averaged, but calculated one by one, and then the loss is calculated the same. This experimental setup is supplemented in the appendix.

3.Your question is very detailed. In the experiment in Figure 5, we fused the training set, verification set and test set in order to test that we can perform well in more categories. GT represents the identity categories contained in the whole dataset, but this experiment is only used for independent post-event analysis and has nothing to do with network training. This experimental setup has been explained in the appendix.

4.A1 you mentioned is not an ablation method with a fixed variable, which we didn't notice. On the basis of A1, we set variants A1-G, which represent the comparison group of DBSCAN algorithm running on the whole training division. Thank you for pointing out the problems in our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Total answer

We thank the reviewers for their constructive feedback. We have added a global DBSCAN control and sensitivity analyses for (q) and (\alpha), and we have clarified the clustering procedure, the spatial and frequency branches, and the evaluation protocols. We summarize our responses below.

  1. Scope of the contribution

We agree that generating global pseudo-labels and optimizing the model in mini-batches is a standard training procedure. We will revise the introduction to focus on our contributions: combining an adaptive affinity graph with SG-MDL and using the resulting pseudo-labels for confidence-weighted contrastive learning and local learning in the spatial and frequency domains.

  1. Affinity graph and parameter selection

The initial affinity matrix (S) can be asymmetric because (j) can be a neighbor of (i) without (i) being a neighbor of (j). Our implementation symmetrizes it as (S \leftarrow (S+S^\T)/2), and the revised manuscript now states this step explicitly. The candidate values of (k) range from (2) to (n-1), where (n) is the number of data points, so the search does not require the ground-truth identity count. At the beginning of each epoch, we recompute (\beta) from the graph built using the current features. Its update depends on the ratio of the magnitudes of change in the first two SG-MDL terms. The revised appendix provides the derivation and sensitivity analyses for the default settings (q=5) and (\alpha=0.5).

  1. Clustering comparisons and ablation controls

The original DBSCAN ablation changed more than one factor. We added a DBSCAN baseline that runs on the entire training split to provide a comparison under the same global clustering setup. We also clarified that both the fixed-(k) and DBSCAN variants in Figure 6 use global clustering. Figure 2 provides a qualitative t-SNE comparison with DBSCAN and conventional Gaussian-kernel spectral clustering, illustrating the behavior of the methods on non-convex structures and under non-uniform density. The revised experiment tables identify the baseline backbones in footnotes.

  1. Spatial and frequency score computation

Frequency scores are computed from the input images using a fixed pipeline with no trainable parameters. For each image, we sample the random crop and flip once and pass the same augmented image to both the frequency branch and the ViT. The reconstructed frequency response map has the same spatial dimensions as the augmented image, and the frequency scores are computed over regions aligned with the ViT patches. For SYSU-MM01 and HITSZ, where camera-level correspondence across modalities is unavailable, we compute spatial scores per image and use those scores jointly to compute the loss. The revised appendix now documents both procedures.

  1. Use of ground truth in Figure 5

Figure 5 is a standalone post hoc analysis that pools the training, validation, and test splits. The ground-truth reference corresponds to the identities in the full dataset. This pooled analysis was used only for Figure 5 and played no role in network training. We have clarified this protocol in the appendix.

Thank you for considering these revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.