acceptodds
Under review as a conference paper at ICLR 2027

OSFAL: Orthogonal Cross-Spectral Feature Alignment and Layer-Localized Cross-View Learning for Any-Scenario Person Re-Identification

Abstract

In Any-Scenario Person Re-Identification (AS-ReID), a unified model must retrieve identities across spectral modalities and aerial–ground views. However, existing methods handle these variations through joint scenario alignment without explicitly accounting for their distinct characteristics, potentially limiting retrieval performance. To address this limitation, we propose OSFAL, which combines modality-specific orthogonal alignment with layer-localized cross-view learning to mitigate spectral and view discrepancies through distinct mechanisms. Specifically, Orthogonal Cross-Spectral Feature Alignment (OCSFA) combines orthogonal rotations with mean alignment to map features from different spectral modalities into a shared reference coordinate system while preserving within-modality Euclidean distances. We learn these transformations by encouraging the modality–view prototypes of each identity to agree with a identity consensus prototype. Layer-Localized Cross-View Learning (LLCVL) compares within-view and cross-view similarities across encoder blocks, together with cross-view identity matching accuracy, to locate a key supervision boundary. Bidirectional cross-view contrastive learning is then applied at the selected block output within each modality. Experiments on WHU-MARS show that OSFAL improves retrieval performance across different evaluation protocols, achieving +5.0% gain in Rank-1 accuracy over the baseline on WHU-MARS-1000.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Any-scenario person re-identification, cross-spectral, cross-view, feature alignment

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_rBZBOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OSFAL, a framework for AS-ReID that addresses the challenges arising from heterogeneous spectral modalities and camera views. OCSFA aligns features from different spectral modalities through modality-specific orthogonal transformations and mean translations, with identity-wise modality-view prototype consistency used to guide the alignment. LLCVL analyzes layer-wise feature statistics to automatically identify a suitable intermediate layer for applying cross-view contrastive supervision, without introducing additional modules at inference time. Experiments on WHU-MARS under multiple evaluation protocols demonstrate the effectiveness of the proposed framework. OCSFA provides substantial improvements over the baseline, and LLCVL brings further gains when combined with OCSFA.

Critical strengths
  1. A well-motivated and complementary approach to two distinct sources of heterogeneity. The paper explicitly addresses both cross-spectral discrepancies and cross-view variations, which are important challenges in AS-ReID. OCSFA reduces spectral discrepancies through structured feature alignment, and LLCVL adaptively determines where cross-view supervision should be applied.
  2. On WHU-MARS-1000, OCSFA improves Rank-1 accuracy from 29.3% to 33.4%, and the complete OSFAL framework further reaches 34.3%. The component ablations suggest that OCSFA accounts for a substantial portion of the improvement and that LLCVL provides complementary gains. The evaluation across multiple protocols and spectral-modality pairs also offers a broader assessment of the method than a single aggregate result. Overall, the empirical findings support the practical value of the proposed design.
Critical weaknesses

None

Questions for authors
  1. Although the ablation studies show that LLCVL improves the baseline and provides further gains when combined with OCSFA, a comparison with applying the same supervision at fixed intermediate layers would more directly demonstrate the value of the adaptive selection mechanism. This limitation does not undermine the overall effectiveness of OSFAL demonstrated by the experiments.
  2. Could the authors report the additional computational overhead introduced by OSFAL compared with the baseline, such as training time, GPU memory consumption, and any additional computational costs associated with OCSFA and LLCVL? This information would help readers better understand the practical trade-off between the performance improvements and the additional cost of the proposed framework.
1 comment
AuthorsOpenReview ↗

Response to Reviewer rBZB

Thank you for the constructive questions. We conducted a fixed-layer comparison and a computational-overhead profile.

Q1. How does LLCVL compare with supervision at fixed layers?

We applied the same supervision separately at each of the 12 ViT blocks on WHU-MARS-1000. All results below are from Epoch 120 under matched training settings.

Supervision layermAP (%)Rank-1 (%)Rank-5 (%)Rank-10 (%)
Fixed Block 112.040933.224050.195258.3450
Fixed Block 212.074733.661250.694858.6105
Fixed Block 312.032633.442650.507458.6105
Fixed Block 412.047833.442650.320158.5948
Fixed Block 512.083433.536350.288858.4075
Fixed Block 612.130833.864250.804158.6573
Fixed Block 712.202934.004750.741658.6417
Fixed Block 812.327734.248251.334958.9696
Fixed Block 912.308634.004750.819759.3443
Fixed Block 1012.293133.841850.710458.7822
Fixed Block 1112.246433.848650.897758.9383
Fixed Block 1212.189733.676850.835359.0164
Adaptive LLCVL12.328934.342851.381758.8915

Adaptive LLCVL demonstrates superior and more balanced performance across multiple key metrics. It achieves the highest scores in mAP, Rank-1, and Rank-5, outperforming all fixed-block strategies. While Fixed Block 9 yields a slightly higher Rank-10 score, LLCVL maintains a strong Rank-10 performance while significantly excelling in the more critical top-ranking metrics.

Q2. Computational overhead. We profiled matched variants on WHU-MARS-1000 using four RTX 3090 GPUs and a global batch size of 256.

MethodAccounted TFLOPs/global training stepPeak memory (GiB/GPU)
UAD baseline52.136611.849
UAD + OCSFA52.697612.312
UAD + LLCVL52.136611.850
OSFAL (fixed Block 8)52.697612.313

The measured FLOP and memory overheads are modest. Relative to UAD, OSFAL increases the accounted computation by 0.5610 TFLOPs per global training step (+1.08%) and peak allocated memory by 0.464 GiB per GPU (+3.9%). LLCVL adds almost no counted FLOPs or peak memory; most of the increase comes from OCSFA. Even during the adaptive selector's startup phase, the observed peak was 12.517 GiB per GPU, only about 0.668 GiB (+5.6%) above UAD. These FLOP figures include an estimate of the forward Cayley solves but not their backward solves or other uncounted operations, so we report them as accounted subtotals rather than complete training FLOPs.

Reviewer_KKanOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Targeting the distinct characteristics of spectral and view discrepancies in Any-Scenario Person Re-Identification (AS-ReID), this paper proposes the Orthogonal Cross-Spectral Feature Alignment and Layer-Localized Cross-View Learning (OSFAL) framework, which addresses spectral and view discrepancies via two distinct mechanisms: Orthogonal Cross-Spectral Feature Alignment (OCSFA) and Layer-Localized Cross-View Learning (LLCVL). OCSFA maps modality-specific features into a shared spectral coordinate system while preserving intra-modal geometric structure. LLCVL identifies the block boundary at which view-related representations in the network undergo a significant transition, and enforces cross-view supervision at this boundary. The method comprehensively outperforms state-of-the-art methods under multiple evaluation protocols on WHU-MARS, and thoroughly validates the effectiveness of each component through ablation studies and visual analysis.

Critical strengths
  1. The authors adopt different handling strategies according to the source of the discrepancy, thereby achieving multimodal pedestrian image processing. Unlike some existing AS-ReID methods that generally follow a joint scenario alignment paradigm—using a single feature aggregation and contrastive learning mechanism to simultaneously handle spectral and view discrepancies, this paper demonstrates from the imaging origin that the two constitute fundamentally different types of variation: spectral discrepancy originates from different sensor spectral band responses and is modeled in part through modality-specific rigid transformations in the learned feature space, while pedestrian pose, spatial structure, and scene layout remain relatively intact; in contrast, aerial-ground view discrepancy is accompanied by pose distortion, scale variation, local occlusion, and resolution differences, forming a compound spatial distortion that evolves progressively with network depth and is difficult to fit with a single global transformation. Therefore, the authors design two dedicated mechanisms: the global orthogonal transformation in OCSFA handles spectral shift, while the layer-localized supervision in LLCVL handles view discrepancy; the two are independent yet complementary. Ablation studies show that OCSFA provides the primary performance gain, and LLCVL further improves retrieval on top of it, supporting the complementary roles of the two mechanisms.
  2. The design of this paper also balances interpretability and practicality. The spectral alignment has explicit geometric constraints, and the cross-view learning maintains a concise inference pipeline: the rotation and translation in OCSFA preserve intra-modal Euclidean geometry and covariance eigenvalues before normalization, providing a clear and verifiable interpretation for the alignment operation. The localization statistics in LLCVL do not propagate gradients, and cross-view supervision is used only during training, adding no extra inference branch and introducing few parameters. This avoids the problem of pursuing metric improvements by stacking modules and increasing inference overhead, thereby improving accuracy while retaining the inference efficiency of the baseline model.
Critical weaknesses
  1. OCSFA appears to work well for NIR, but its performance on TIR is less convincing. The method uses a single orthogonal transform to align each modality while preserving within-modality distances (Eq. 6), with the transform learned by pulling modality–view prototypes of the same identity toward a shared consensus (Eq. 10). This design seems reasonable for NIR, whose appearance is relatively close to RGB. TIR differs much more substantially from RGB due to the loss of color and texture information, making alignment through a single rigid transformation potentially less effective. The results in Table 6 are consistent with this concern, as several TIR retrieval directions perform slightly worse than the baseline. This limitation is particularly relevant given that cross-spectral alignment is a central contribution of the paper. The overall 5.0 points Rank-1 gain of AS-ReID should therefore be broken down by modality pair. Such an analysis would help establish how much of the improvement comes from cross-spectral alignment, particularly in the more challenging TIR setting, rather than from the easier NIR and same-spectral cases.
  2. LLCVL brings only a modest improvement over OCSFA. Table 4 shows a Rank-1 gain of about 0.9 points. The layer localization procedure also involves several steps, including block-wise scoring, voting across folds and seeds, a consensus gate, and a multi-epoch stability check. The benefit of this added complexity remains unclear. The current experiments do not show whether automatic layer localization performs better than selecting a fixed and reasonable layer and applying the same cross-view loss there. Layer localization is presented as one of the paper's two main contributions. A comparison with a fixed-layer baseline under the same training settings is needed to isolate its contribution. Without this comparison, the improvement may come from the cross-view loss itself rather than from the localization strategy. The authors should include this baseline to better support the claimed contribution of LLCVL.
Questions for authors
  1. Table 6 shows that some TIR retrieval directions perform worse than the baseline. The paper reports an overall Rank-1 gain of 5.0 points for AS-ReID. A breakdown by modality pair is needed to clarify how much of this gain comes from cross-spectral alignment rather than from NIR and same-spectral cases.
  2. LLCVL improves Rank-1 by about 0.9 points over OCSFA. A comparison with a fixed supervision layer would help establish whether automatic layer localization provides additional benefits. The same cross-view loss should be used in both settings to isolate the effect of the localization mechanism.
  3. The consistency loss in Eq. 10 relies on cross modality prototypes. The paper does not clearly explain how each batch covers different modality view conditions. If TIR samples are less frequent, the supervision for the TIR transform may be weaker. The effect of such modality imbalance on cross-spectral alignment should be clarified.
2 comments
AuthorsOpenReview ↗

Response to Reviewer KKan

We thank the reviewer for these questions. The additional analyses help distinguish what each component is designed to achieve from what can be inferred from an aggregate retrieval score.

Q1. Where does the overall improvement come from, particularly for TIR?

The modality-pair results are complementary evidence, rather than a decomposition of the overall Rank-1 gain. Seven of the nine directed pairs improve, including both RGB-NIR directions and both RGB-TIR directions; NIR -> TIR and TIR -> NIR remain challenging. The larger gains in same-modality retrieval are understandable: these searches do not require matching across a spectral gap, and end-to-end training can improve the shared identity representation even though the orthogonal transform itself preserves within-modality geometry.

More importantly, each modality-pair protocol restricts the gallery to a single modality. The main AS-ReID protocol instead places RGB, NIR, and TIR images together in one gallery, where a query must rank its true matches above distractors from all three modalities. The pairwise results therefore cannot be added together or used to assign a fraction of the overall gain to one modality pair. Under this mixed-gallery protocol, the matched Rank-1 results are 29.32% for UAD, 33.40% for UAD+OCSFA, and 34.24% for OSFAL. This direct comparison establishes the overall benefit, while the pairwise table shows where that benefit generalizes and where further cross-spectral improvement is needed.

Q2. How does LLCVL compare with supervision at fixed layers?

We applied the same supervision separately at each of the 12 ViT blocks on WHU-MARS-1000. All results below are from Epoch 120 under matched training settings.

Supervision layermAP (%)Rank-1 (%)Rank-5 (%)Rank-10 (%)
Fixed Block 112.040933.224050.195258.3450
Fixed Block 212.074733.661250.694858.6105
Fixed Block 312.032633.442650.507458.6105
Fixed Block 412.047833.442650.320158.5948
Fixed Block 512.083433.536350.288858.4075
Fixed Block 612.130833.864250.804158.6573
Fixed Block 712.202934.004750.741658.6417
Fixed Block 812.327734.248251.334958.9696
Fixed Block 912.308634.004750.819759.3443
Fixed Block 1012.293133.841850.710458.7822
Fixed Block 1112.246433.848650.897758.9383
Fixed Block 1212.189733.676850.835359.0164
Adaptive LLCVL12.328934.342851.381758.8915

Adaptive LLCVL demonstrates superior and more balanced performance across multiple key metrics. It achieves the highest scores in mAP, Rank-1, and Rank-5, outperforming all fixed-block strategies. While Fixed Block 9 yields a slightly higher Rank-10 score, LLCVL maintains a strong Rank-10 performance while significantly excelling in the more critical top-ranking metrics.

Q3. Are TIR samples underrepresented in the consistency loss?

No. The PKM sampler is balanced per identity and per spectral modality. On each GPU, a batch selects 16 identities and four images of each identity from each of RGB, NIR, and TIR. It therefore contains 64 RGB, 64 NIR, and 64 TIR images; the four-GPU global batch contains 256 images of each modality. The TIR transform does not receive fewer training samples simply because the dataset contains fewer TIR images.

For Eq. 10, we first form a prototype for each identity-modality-view cell observed in the current batch, then average the available cell-level terms within each identity. Consequently, a cell with more images does not receive proportionally more weight in the consistency objective, and an unobserved view cell is not treated as a zero prototype. The sampler does not force Ground and Aerial to occur equally often, nor does equal sampling create additional distinct TIR information. These are different issues from the proposed TIR-frequency imbalance; we will clarify the sampling and prototype construction in the implementation details.

The authors' response addressed some of my concerns and I will keep my score.

Reviewer_hbLmOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors present OSFAL to tackle Any-Scenario Person Re-Identification (AS-ReID). Instead of treating different modalities (RGB, NIR, TIR) and camera perspectives (ground vs. aerial) as a single unified domain gap, the method handles them separately through two dedicated components: OCSFA for spectral alignment and LLCVL for view-invariant learning. The approach is tested extensively on the WHU-MARS dataset and shows solid performance gains over the UAD baseline across various test settings.

Critical strengths
  1. Treating cross-spectral shifts as a combination of feature centering, rotation, and translation is intuitive and mathematically clean. Centering the features prior to applying the Cayley transform effectively avoids the origin-shift issue during rotation, and the paper provides a solid derivation showing why within-modality relative distances are preserved.

  2. The training-only design of LLCVL is very practical. By restricting the intermediate-layer contrastive loss to the training phase, the network avoids extra branches or heavy computation during deployment, keeping the inference pipeline just as fast as the baseline except for a minor linear projection.

Critical weaknesses
  1. OCSFA uses RGB as the fixed reference space for cross-spectral alignment without sufficiently justifying this choice or evaluating alternatives such as NIR or TIR. The method’s sensitivity to the reference modality and its dependence on richer initial RGB representations remain unclear.

  2. The mini-batch sampling strategy is insufficiently specified. The proposed losses appear to require same-identity samples across multiple spectral modalities and aerial/ground views, but it is unclear whether the standard PK sampler ensures this coverage or how missing modality–view combinations are handled. These details are important for reproducibility and understanding the training requirements.

  3. Despite overall improvements, Table 6 reports lower Rank-1 accuracy than UAD on NIR-to-TIR retrieval, with insufficient discussion of this regression. Further analysis is needed to clarify whether RGB-based alignment introduces a trade-off for cross-infrared retrieval or whether other factors explain the decline.

Questions for authors
  1. In OCSFA, what happens if you set NIR or TIR as the reference modality instead of RGB? Does the alignment accuracy remain consistent, or does the method rely on RGB having richer initial representations?

  2. Calculating both the OCSFA and LLCVL losses seems to require each mini-batch to contain multiple spectral modalities and both aerial/ground views for the same person ID. Did you modify the standard PK sampler during training to force every sampled identity to include all modality and view combinations?

  3. Despite the overall improvements, Table 6 shows a drop in Rank-1 accuracy for the NIR-to-TIR retrieval task compared to the baseline. What is the main reason behind the degradation in this specific cross-infrared setting?

2 comments
AuthorsOpenReview ↗

Response to the reviewer hblm

We thank the reviewer for the constructive feedback and positive assessment of our work. We address the three questions below.

Q1. Reference modality.

We evaluated all three reference modalities in a controlled OCSFA-only ablation, with LLCVL disabled and other settings unchanged. Results at epoch 120 on WHU-MARS-1000 are:

Reference modalitymAP (%)Rank-1 (%)
Baseline10.9629.32
RGB12.0433.40
NIR11.8932.57
TIR11.8032.44

All three references outperform the baseline, with RGB performing best. The reason is that our ViT backbone is initialized from pretraining on RGB images, providing a stronger starting representation for RGB than for NIR or TIR. Using RGB as the reference may therefore provide a more reliable anchor for spectral alignment, consistent with the observed results.

Q2. Sampling strategy.

We retain the baseline’s PKM sampler, which samples four images per identity per modality, ensuring RGB/NIR/TIR coverage without forcing all six modality–view combinations. OCSFA uses only observed condition prototypes and omits missing cells. LLCVL includes an identity within a modality only when both Ground and Aerial observations are available; a modality contributes only if at least two identities qualify. Otherwise, that modality is skipped, and the loss is zero if none qualifies. Explicit synchronized pairing is unnecessary.

Q3. NIR-to-TIR retrieval.

A plausible explanation lies in the different image-formation mechanisms. RGB and NIR primarily capture reflected radiation, whereas TIR primarily captures emitted thermal radiation. RGB and NIR therefore share a more similar imaging mechanism, while TIR appearance depends on surface temperature and emissivity, providing different identity cues. This makes TIR harder to align with the other modalities, so improvements are not uniform across retrieval directions. For NIR-to-TIR, Rank-1 drops slightly, although mAP and mINP improve.

Thanks to the authors for their detailed responses, which have addressed most of my concerns. After reviewing the comments from the other reviewers and the authors‘ corresponding responses, I have decided to maintain my initial positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.