acceptodds
Under review as a conference paper at ICLR 2027

Joint Modality- and View-Aware Positional Residuals for Any-Scenario Person Re-Identification

Abstract

Any-Scenario Person Re-Identification (AS-ReID) aims to match individuals across diverse combinations of sensing modalities and camera viewpoints. Although existing transformer-based methods have achieved promising performance, they commonly share positional embeddings across sensing modalities. However, this shared positional prior does not explicitly account for modality-dependent spatial cues or the changes in body-part layout caused by aerial-ground viewpoint shifts. To address these limitations, we propose Modality- and View-Aware Positional Residuals (MVPR), a framework comprising two complementary components: Modality-Aware Positional Residual Correction (MPRC) and Text-Guided View Alignment (TGVA). MPRC learns modality-specific positional residuals and injects differences between consecutive layer-specific residuals at successive transformer blocks, enabling layer-wise, position-dependent corrections within a shared visual encoder. TGVA uses learnable text prompts to supervise paired forward passes with MPRC disabled and enabled. For aerial images, the two passes use aerial- and ground-view text anchors, respectively, whereas for ground images both use ground-view anchors. These asymmetric target assignments guide aerial-to-ground adaptation while maintaining ground-view supervision for ground images. Experiments on WHU-MARS-1000 demonstrate that MVPR surpasses UAD by 3.4% and 7.7% in mAP and Rank-1 accuracy, respectively, under the AS-ReID setting. The code will be available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Any-Scenario Person Re-Identification, Cross-Modal Representation Learning, Modality-Aware Positional Residuals, Text-Guided View Alignment, Prompt Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GmtPOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies any-scenario person re-identification across RGB, NIR, and TIR images captured by aerial and ground cameras. It proposes Modality- and View-Aware Positional Residuals (MVPR), based on a shared CLIP visual encoder. The method includes two components. Modality-Aware Positional Residual Correction learns separate positional residuals for each modality and transformer layer. It adds the differences between consecutive residuals before each block to adjust token features. Text-Guided View Alignment uses learnable text anchors to supervise two forward passes, with the corrections disabled and enabled. For aerial images, these passes use aerial-view and ground-view targets, respectively. Ground images use ground-view targets in both passes. This guides aerial features toward a ground-view reference. The model is trained with identity classification, triplet, and text-guided losses. Experiments on WHU-MARS show improved performance across any-scenario, aerial-ground, and visible-infrared retrieval settings. Ablation studies further show the benefits of both components.

Critical strengths

1.The paper addresses a useful and challenging ReID problem involving both modality and viewpoint changes. A single model handles RGB, NIR, and TIR images from aerial and ground cameras, supporting retrieval across different combinations of these conditions.

2.The method has a clear design and reasonable. MPRC learns modality-specific positional corrections at different transformer layers, while TGVA provides explicit text targets for aerial-to-ground alignment. The two components have clear roles and work together within a shared visual encoder.

3.The reported results show consistent improvements in mAP and Rank-1 across the tested WHU-MARS settings.

Critical weaknesses

1.The motivation for using the ground view as the common reference needs further explanation. ReID requires features of the same person to support matching across views, but this goal does not by itself require one view to serve as the reference. The paper clearly defines the aerial-to-ground training targets but should better explain why this direction is helpful and what makes the ground view a suitable reference.

2.The added value of position-dependent corrections is not clearly isolated. Earlier methods such as TransReID and CMTR already use learnable embeddings to encode camera, view, or modality information. MPRC learns separate corrections for different layers and token positions. However, the paper does not compare it with a modality vector shared across positions at the same layers, using the same CLIP backbone and supervision. Table 6 tests sharing across modalities, which does not establish the benefit of position-dependent corrections.

3.This paper uses a static positional correction. MPRC shares the same residual parameters across all images of a modality. However, poses, viewpoints, and visible body regions can vary greatly within the same modality. So, I think whether using a image-based correction strategy will show a better performance?

Questions for authors

Please refers to the weaknesses.

1 comment

Q1: The motivation for using the ground view as the common reference needs further explanation. ReID requires features of the same person to support matching across views, but this goal does not by itself require one view to serve as the reference. The paper clearly defines the aerial-to-ground training targets but should better explain why this direction is helpful and what makes the ground view a suitable reference.

A1: Thank you for your question. We chose ground views as the reference because they often show more details of a person’s appearance and are less affected by foreshortening. We compared ground and aerial targets in the forward pass with residuals enabled. In both settings, the pass with residuals disabled used the original-view target, and all other settings were fixed. As shown in Table A, using ground-view targets improves mAP by 1.13% and Rank-1 by 2.50% compared with using aerial-view targets. Table F in our response to Reviewer gevp also shows that paired supervision with original-view and ground-view targets improves retrieval compared with using ground-view targets in a single forward pass.

Table A: Ground versus aerial reference under paired supervision on WHU-MARS-1000.

Common referencemAP (%)Rank-1 (%)Rank-5 (%)Rank-10 (%)mINP (%)
Aerial13.2934.6651.3359.480.58
Ground14.4237.1654.1061.830.65

Q2: The added value of position-dependent corrections is not clearly isolated. Earlier methods such as TransReID and CMTR already use learnable embeddings to encode camera, view, or modality information. MPRC learns separate corrections for different layers and token positions. However, the paper does not compare it with a modality vector shared across positions at the same layers, using the same CLIP backbone and supervision. Table 6 tests sharing across modalities, which does not establish the benefit of position-dependent corrections.

A2: Thank you for your suggestion. We added a control with one correction vector per modality and layer, shared across token positions. This control uses the same CLIP backbone and paired text supervision as MPRC. As shown in Table B, learning separate corrections at each token position improves mAP by 0.28% and Rank-1 by 0.38% over this control. MPRC also outperforms the variant that applies modality-specific positional corrections only at the input.

Table B: Controls for the additional value of position-dependent correction.

Correction designmAP (%)Rank-1 (%)
Modality-specific input positional correction14.0836.44
Per-layer correction shared across token positions14.1436.78
Per-layer, position-dependent MPRC14.4237.16

Q3: This paper uses a static positional correction. MPRC shares the same residual parameters across all images of a modality. However, poses, viewpoints, and visible body regions can vary greatly within the same modality. So, I think whether using a image-based correction strategy will show a better performance?

A3: Thank you for your suggestion. We kept the supervision fixed and replaced static MPRC with an MLP that uses the CLS token to predict corrections for each token. As shown in Table C, static MPRC yields 0.31% higher mAP and 3.45% higher Rank-1 than the dynamic variant. These results favor sharing corrections across images of the same modality in this setting.

Table C: Image-dependent versus static correction.

CorrectionmAP (%)Rank-1 (%)
CLS-conditioned dynamic MLP14.1133.71
Static MPRC14.4237.16
Reviewer_gevpOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MVPR for any-scenario person re-identification. The method uses a shared CLIP visual encoder and two modules. MPRC learns correction vectors for each modality, layer, and token position. These vectors are shared across images of the same modality. Differences between adjacent-layer corrections are added before each transformer block. TGVA uses six modality-view text anchors generated from learnable prompts and a frozen text encoder. Each training image passes through the visual encoder twice, with the corrections disabled and enabled. The two passes use the original-view and ground-view targets, respectively. This guides aerial features toward ground-view references. Identity classification and triplet losses support identity matching. Inference uses only the visual encoder with corrections. Experiments on WHU-MARS show improved performance in any-scenario, aerial-ground, and visible-infrared retrieval.

Critical strengths

1.The motivation of this work is straightforward and interesting. I think this design well address one of the important challenges of this topic.

2.This work is easy to understand and achieves a good performance on related settings.

3.I appreciate the interesting analyses in the paper, particularly the comparison between injecting differences of adjacent-layer residuals and directly learning the bias added at each layer.

Critical weaknesses
  1. The overall methodological advance appears relatively modest. The method is straightforward and includes several interesting design choices. Its simplicity is a practical strength, but the contribution mainly lies in combining modality-specific feature corrections with text-guided view supervision.

  2. The motivation and limitations of the static corrections are not fully discussed. MPRC shares the same residual parameters across all images of a modality, despite differences in pose, viewpoint, and visible body regions. Although the transformer remains input-dependent, the corrections themselves do not change with image content. The current analysis provides limited insight into the tradeoff between shared corrections and image-dependent adaptation.

  3. The experimental evidence is limited to one dataset. The paper evaluates several splits and retrieval settings, but all experiments use WHU-MARS. These results provide useful evidence within this benchmark, while the effectiveness of the method on independently collected data remains unverified.

Questions for authors
  1. What motivates learning a separate correction for each token position, instead of using one correction shared across positions within each modality and layer? What common spatial patterns are these position-specific parameters expected to capture?

  2. Why are the residuals shared across all images of the same modality? Have the authors considered a lightweight variant that generates or adjusts the corrections based on each image? What benefits or costs would such a design introduce?

  3. Table 7 uses shared residuals and two RGB text anchors, while the final model uses modality-specific residuals and six anchors. Why were this simplified setting chosen, and how should its ablation results be interpreted in relation to the final model?

4 comments

Q1: The overall methodological advance appears relatively modest. The method is straightforward and includes several interesting design choices. Its simplicity is a practical strength, but the contribution mainly lies in combining modality-specific feature corrections with text-guided view supervision.

A1: Thank you for your constructive feedback. Aligning modality centers or adding embeddings at the input does not directly account for how spatial cues vary across modalities and layers. MPRC learns positional residuals for each modality and transformer block, and TGVA uses paired text supervision to guide adaptation across aerial and ground views. MPRC adds the corrections before each block, so it can be integrated without changing the attention or feed-forward layers. The text branch is used only during training. At inference, the visual encoder extracts features in a single forward pass without viewpoint labels. Compared with the CLIP baseline, MVPR adds approximately 3.57M parameters. Computing and adding the positional residual corrections costs an additional 0.0048 GFLOPs per image. In Table A, MVPR improves mAP and Rank-1 by 2.43% and 6.06%, respectively, over the CLIP baseline. The corresponding gains over the reported UAD results are 3.42% and 7.66%.

Table A reports parameter counts for visual feature extraction, including the residuals for all three modalities. We count FLOPs for one full forward pass of a 256 × 128 image, including residual differencing and addition. We count two FLOPs per MAC, one multiplication and one addition.

Table A: Inference cost and retrieval performance on WHU-MARS-1000.

MethodParams (M)FLOPs (G)mAP (%)Rank-1 (%)
CLIP visual baseline85.7522.74911.9931.10
UAD85.7522.74411.0029.50
MVPR89.3122.75414.4237.16

Q2: The motivation and limitations of the static corrections are not fully discussed. MPRC shares the same residual parameters across all images of a modality, despite differences in pose, viewpoint, and visible body regions. Although the transformer remains input-dependent, the corrections themselves do not change with image content. The current analysis provides limited insight into the tradeoff between shared corrections and image-dependent adaptation.

A2: Thank you for your suggestion. We share positional residuals across images of the same modality to capture systematic differences in spatial cues among RGB, NIR, and TIR images. The transformer still extracts features from each input, but the corrections are fixed within each modality and cannot adapt to changes in pose, viewpoint, or visible body regions. To assess this limitation, we compared static MPRC with a variant that uses the CLS token to predict corrections for each image. As shown in Table B, static MPRC yields 0.31% higher mAP and 3.45% higher Rank-1. These results support sharing corrections within each modality in this setting.

Table B: Effect of sharing corrections across images within a modality.

CorrectionmAP (%)Rank-1 (%)Rank-5 (%)Rank-10 (%)
CLS-conditioned dynamic MLP14.1133.7151.1959.20
Static MPRC14.4237.1654.1061.83

Q3: The experimental evidence is limited to one dataset. The paper evaluates several splits and retrieval settings, but all experiments use WHU-MARS. These results provide useful evidence within this benchmark, while the effectiveness of the method on independently collected data remains unverified.

A3: Thank you for raising this point. To the best of our knowledge, WHU-MARS is currently the only AS-ReID dataset that supports retrieval across arbitrary combinations of RGB, NIR, and TIR imagery and across aerial and ground viewpoints. This coverage makes it suitable for evaluating the task studied in our paper.

Q4: What motivates learning a separate correction for each token position, instead of using one correction shared across positions within each modality and layer? What common spatial patterns are these position-specific parameters expected to capture?

A4: Thank you for your question. Separate corrections at each token position allow MPRC to capture systematic differences in spatial cues within each modality. We kept the CLIP backbone and text supervision fixed and compared MPRC with two variants. One shares corrections across token positions, and the other applies corrections only at the input. As shown in Table C, MPRC improves mAP by 0.28% and Rank-1 by 0.38% over the variant that shares corrections across positions. It also outperforms the variant that applies corrections only at the input, supporting the use of corrections at multiple layers.

Table C: Controls for position-dependent and layer-wise correction.

Correction designmAP (%)Rank-1 (%)
Modality-specific input positional correction14.0836.44
Per-layer correction shared across token positions14.1436.78
Per-layer, position-dependent MPRC14.4237.16

Q5: Why are the residuals shared across all images of the same modality? Have the authors considered a lightweight variant that generates or adjusts the corrections based on each image? What benefits or costs would such a design introduce?

A5: Thank you for your suggestion. Shared residuals capture spatial patterns common to images of the same modality. To test whether corrections tailored to each image are beneficial, we replaced MPRC with an MLP that predicts corrections for individual tokens from the CLS token and embeddings for the layer and modality. The MLP has a 32-dimensional hidden layer and shares weights across transformer blocks. As shown in Table D, static MPRC yields 0.31% higher mAP and 3.45% higher Rank-1. The dynamic variant can adapt to individual images, but it requires an extra prediction step at inference and yields lower retrieval performance in this comparison.

Table D: Static versus CLS-conditioned dynamic correction.

CorrectionmAP (%)Rank-1 (%)Rank-5 (%)Rank-10 (%)
CLS-conditioned dynamic MLP14.1133.7151.1959.20
Static MPRC14.4237.1654.1061.83

Q6: Table 7 uses shared residuals and two RGB text anchors, while the final model uses modality-specific residuals and six anchors. Why were this simplified setting chosen, and how should its ablation results be interpreted in relation to the final model?

A6: Thank you for pointing out the difference in configurations. The original Table 7 used shared residuals and two RGB text anchors to study residual injection locations and view supervision targets. We repeated the ablations with the final configuration, using modality-specific residuals and six text anchors whenever the respective components were enabled. As shown in Table E, injecting residuals at all 12 blocks yields the highest scores on all reported metrics. Compared with injection at six blocks, it improves mAP by 0.12% and Rank-1 by 0.30%.

Table E: Ablation of residual injection locations under the final configuration. Block numbering starts at 1.

Blocks receiving correctionsmAP (%)Rank-1 (%)Rank-5 (%)mINP (%)
114.0736.1152.830.64
1213.7235.3852.040.60
1, 4, 7, 1014.1736.3253.610.61
1, 3, 5, 7, 9, 1114.3036.8653.750.62
All 1214.4237.1654.100.65

As shown in Table F, with modality-specific residuals, paired supervision improves mAP by 1.34% and Rank-1 by 3.19% compared with ground-view supervision in a single forward pass. Under the latter setting, adding positional residuals reduces retrieval performance. These results show that the gains from residual correction depend on the choice of view supervision. We will replace the original Table 7 with these results and update the discussion.

Table F: Ablation of view supervision targets under the final configuration.

Visual parametersTargetsmAP (%)Rank-1 (%)Rank-5 (%)mINP (%)
BackboneNone11.9931.1048.570.62
BackboneGround, single pass13.4835.8252.460.59
Backbone + modality-specific residualsNone12.7533.4750.070.64
Backbone + modality-specific residualsGround, single pass13.0833.9751.370.61
Backbone + modality-specific residualsPaired14.4237.1654.100.65

I appreciate the authors’ response, which has addressed all my concerns. I have therefore raised my score to 4.

Reviewer_fzsLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MVPR for person re-identification across sensing modalities and aerial–ground viewpoints. MPRC introduces modality-specific positional residuals through adjacent-layer differences, while TGVA supervises residual-disabled and residual-enabled forward passes using learnable modality–view text anchors.

Critical strengths

MVPR achieves consistent mAP and Rank-1 improvements across the reported WHU-MARS settings, including the larger 2,337-identity split. The framework retains a single visual encoder at inference and requires neither text processing nor viewpoint labels.

Critical weaknesses
  1. The benefit of the differential parameterization is not convincingly established. Appendix C shows that it represents the same function class as directly learned per-layer biases; the distinction concerns optimization and regularization. In the full model, direct biases achieve 14.45% mAP versus 14.42% for adjacent differences, with only a small Rank-1 difference.
  2. The interpretation of TGVA as semantic view alignment remains insufficiently supported. Its objective assigns different view targets to the two passes, but satisfying these targets does not establish identity-preserving alignment between aerial and ground features. The bound in Appendix C addresses identical-pass predictions rather than cross-view alignment. A comparison with learnable non-text prototypes under the same paired-pass objective is needed to distinguish the contribution of textual semantics from auxiliary classification supervision.
  3. Does TGVA retain its advantage when the six text anchors are replaced with learnable non-text prototypes under the same paired-pass objective and training budget?
Questions for authors

See Weaknesses. I would be willing to raise my rating if the authors adequately address these concerns.

1 comment

Q1: The benefit of the differential parameterization is not convincingly established. Appendix C shows that it represents the same function class as directly learned per-layer biases; the distinction concerns optimization and regularization. In the full model, direct biases achieve 14.45% mAP versus 14.42% for adjacent differences, with only a small Rank-1 difference.

A1: Thank you for your constructive feedback. The two parameterizations have the same parameter count and nearly identical computational costs. We chose adjacent differences based on the Rank-1 accuracy reported in lines 403 to 405 of the submitted manuscript. As shown in Table A, in the full model, adjacent differences yield 0.27% higher Rank-1 accuracy, while directly learned per-layer biases yield 0.03% higher mAP. The two parameterizations have similar expressive capacity (Appendix C). Our contribution is to learn modality-specific, position-dependent corrections and guide them with paired text supervision. We will clarify the basis for this choice and report the results for both metrics.

Table A: Adjacent-layer differences versus directly learned per-layer biases.

SettingParameterizationmAP (%)Rank-1 (%)
Shared residuals, no text supervisionDirectly learned per-layer biases12.7632.27
Shared residuals, no text supervisionAdjacent-layer differences13.2934.10
Modality-specific residuals, six text anchorsDirectly learned per-layer biases14.4536.89
Modality-specific residuals, six text anchorsAdjacent-layer differences14.4237.16

Q2: The interpretation of TGVA as semantic view alignment remains insufficiently supported. Its objective assigns different view targets to the two passes, but satisfying these targets does not establish identity-preserving alignment between aerial and ground features. The bound in Appendix C addresses identical-pass predictions rather than cross-view alignment. A comparison with learnable non-text prototypes under the same paired-pass objective is needed to distinguish the contribution of textual semantics from auxiliary classification supervision.

A2: Thank you for your suggestion. TGVA assigns view targets to the two forward passes, and the identity classification and triplet losses provide identity supervision. The bound in Appendix C describes the view loss when the two passes produce identical features. It does not establish identity-preserving alignment across views. To evaluate the contribution of the text anchors, we replaced them with six learnable non-text prototypes and kept the paired objective, normalization, logit scale, loss weight, and training budget fixed. As shown in Table B, the text anchors improve mAP by 0.76% and Rank-1 by 1.56% over the prototype baseline.

Table B: Text anchors versus learnable non-text prototypes on WHU-MARS-1000.

Supervision anchorsmAP (%)Rank-1 (%)Rank-5 (%)Rank-10 (%)
Six learnable non-text prototypes13.6635.6052.3260.12
Six text anchors, MVPR14.4237.1654.1061.83

Q3: Does TGVA retain its advantage when the six text anchors are replaced with learnable non-text prototypes under the same paired-pass objective and training budget?

A3: Thank you for your suggestion. As shown in Table B, the model with text anchors achieves better retrieval performance than the model with learnable non-text prototypes under the same paired objective and training budget. Please let us know if you have any further questions or if we have misunderstood your concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.