acceptodds
Under review as a conference paper at ICLR 2027

MoMI-RC: Identity-Aware Reliability Calibration and Multi-Order Modality Interaction for Multi-Modal Object Re-Identification

Abstract

Multi-modal object Re-Identification (ReID) exploits complementary RGB, near-infrared, and thermal-infrared information to retrieve the same person or vehicle under challenging imaging conditions. However, fusion weights predicted solely from individual inputs may lack a stable identity-discriminative reference, while merging interactions from different modality combinations can obscure their distinct sources. To address these issues, we propose MoMI-RC, a multi-order modality interaction network with identity-aware reliability calibration. The reliability calibration module learns global modality anchors and bounded sample-wise corrections under supervision based on intra-class compactness and inter-class separation. The anchors provide a stable reference, while the corrections adapt modality contributions to individual inputs. The multi-order modality interaction module constructs compact pairwise and three-way interactions in a shared low-rank space, preserving each interaction in a separate coordinate block alongside calibrated unimodal features. Modality-subset simulation and consistency regularization further stabilize the descriptor during training. Extensive experiments demonstrate the effectiveness of MoMI-RC, which achieves 84.2%, 89.6%, and 70.9% mAP on RGBNT201, RGBNT100, and MSVR310, respectively, without task-specific text or segmentation priors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-modal Object Re-Identification, Reliability Calibration, Multi-order Modality Interaction, Low-rank Fusion, Multi-modal Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7FpnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes MoMI-RC for multi-modal object re-identification using RGB, near-infrared, and thermal images. It combines identity-supervised global modality weights with sample-specific adjustments, and preserves pairwise and three-way modality interactions in separate feature blocks. Modality-subset training and consistency regularization further support robust representation learning. Experiments on three person and vehicle benchmarks report competitive retrieval performance, with component ablations conducted on RGBNT201.

Critical strengths

The proposed reliability calibration and multi-order interaction modules achieve strong retrieval performance across three benchmarks, demonstrating the practical effectiveness of the overall framework for both person and vehicle re-identification.

Critical weaknesses
  1. The introduction does not clearly explain the concrete challenges in existing methods or why the proposed RC and MoMI designs address them. The reliance on abstract terminology makes the motivation and intended contributions difficult to assess.

  2. Beyond improvements in Rank-1 and mAP, the main text provides limited direct evidence supporting the core mechanisms. In particular, it remains unclear whether the global anchors reliably capture modality reliability and whether the separate interaction blocks preserve complementary information. This limits confidence in the explanation for the reported gains.

Questions for authors
  1. The introduction does not clearly explain the concrete limitations of existing methods or how RC and MoMI address them. Could you illustrate these limitations with a concrete example? A schematic comparing existing fusion approaches with the proposed design would help clarify the motivation.

  2. The component ablations are limited to RGBNT201. Could you report the main component ablations on RGBNT100 and MSVR310 as well, to establish whether the benefits of RC, MoMI, subset training, and consistency regularization hold across both person and vehicle ReID?

  3. The main text does not clearly demonstrate whether the proposed mechanisms work as motivated except rank and mAP performance. But I want to have a look more direct evidence supporting the core designations. Could you display the other evidence?

  4. Could you distinguish modalities within each identity in the t-SNE visualization, for example using colors for identities and marker shapes for modalities? If the plotted descriptor is a fused tri-modal representation, a separate visualization of modality-specific features would be more informative.

I will improve the rating and confidence if the rebuttal provides clearer explanations and convincing supporting results.

3 comments
AuthorsOpenReview ↗

Clarified motivation and additional supporting evidence

Thank you for the constructive comments. We address each point below.

[W1] Concrete limitations and motivation. RC and MoMI address the two limitations below, and Figure 1 of the revised PDF illustrates both.

Limitation 1: overall modality reliability and per-image degradation are not modeled separately. A modality's weight depends on two different factors. The first is whether the current image is degraded; e.g., an occluded RGB image should lower the RGB weight for that image only. The second is how well the modality distinguishes identities overall; e.g., NIR has the lowest single-modality mAP on RGBNT201 (28.2%, Table 7). Input-adaptive methods such as DeMo predict weights from the current input, but have no identity-supervised parameter for the second factor. RC models the two factors separately. A global anchor, supervised by identity distances in training batches, encodes the overall reliability of each modality, and the learned NIR anchor is the lowest (Fig. 4). A bounded sample residual then adjusts the weight for the current image; for example, occluding 75% of an RGB image lowers its weight from 1.42 to 0.94 (Fig. 9).

Limitation 2: cross-modal interactions are not organized by source. Direct concatenation, common in multi-modal ReID, keeps only first-order features. If the interactions are summed or averaged into one vector, it may no longer be clear which modality pair the evidence comes from. For example, if sample A has a strong RGB–NIR and a weak RGB–TIR interaction and sample B the reverse, averaging makes them identical. MoMI places each of the RN, RT, NT, and RNT interactions in its own compact block and concatenates them with the first-order features. In Table 3, with the same products, separate blocks give 84.2% mAP and a single merged block gives 83.1%. Under the same setting, MoMI also outperforms our bilinear-, TFN-, and LMF-style heads (82.4–82.9%) with fewer head parameters (see our response to Reviewer M9zo).

[W2] Ablations on RGBNT100 and MSVR310. Following Table 2, mAP (mean ± std over 3 runs):

DatasetBaseline+MoMI+RC+MoMI+RC+SubsetFull
RGBNT10085.4±0.487.9±0.386.4±0.188.5±0.489.1±0.289.6±0.3
MSVR31065.8±0.468.8±0.869.0±0.669.4±0.669.9±0.470.5±0.4

Both datasets follow the RGBNT201 trend: MoMI and RC each improve over the baseline, their combination outperforms either alone, and subset training and consistency regularization raise mAP further. The full model's mean mAP is 4.2 and 4.7 points above the baseline, so the components give consistent gains for both person and vehicle ReID.

[W3] Direct evidence beyond Rank and mAP.

Global anchors reflect modality reliability. The anchors learned on RGBNT201 are 1.178 for TIR, 1.026 for RGB, and 0.827 for NIR (Fig. 4). They are learned only from training-set identity distances, yet follow the same order as single-modality test mAP: TIR 56.1%, RGB 44.5%, NIR 28.2% (Table 7).

Sample weights respond to degradation. Following Fig. 9, we occlude the central 75% of one modality for each of the 836 RGBNT201 queries, once per modality (2,508 pairs in total):

OccludedMean weight before → afterSamples with lower weight
RGB1.069 → 0.86399.9%
NIR0.832 → 0.68599.9%
TIR1.119 → 0.97997.7%

Separate interaction blocks. Structurally, each block corresponds to one modality combination. When a modality is missing, only the blocks involving it are zeroed, and the pairwise block of the other two modalities remains usable; with R+T input, for example, the full model reaches 79.4% mAP (Table 7). Empirically, with the same products, separate blocks outperform merging (Table 3: 84.2 vs. 83.1), and keeping both second- and third-order interactions gives the best mAP (Table 8).

Descriptor-level separation. On the same positive and negative query–gallery pairs, the mean cosine-similarity gap rises from 0.250 for the baseline to 0.332 with MoMI and RC, and to 0.403 for the full model (Fig. 5).

[W4] Modality-wise t-SNE. Each point in Fig. 3 is a fused tri-modal descriptor, so modalities cannot be separated there. We therefore visualize the RGB, NIR, and TIR features separately, with color denoting identity; for each modality, the baseline and MoMI-RC panels use the same samples. For the displayed identities, MoMI-RC gives more compact clusters than the baseline in every modality, which qualitatively indicates that the single-modality features also improve. This figure is included as Figure 10 in Appendix A.7 of the revised PDF.

Thanks for your response. The response has not fully addressed my concerns. So I will not improve rating.

AuthorsOpenReview ↗

Follow-up on the remaining concerns

Thank you for reading our response. Since the discussion period is nearly over, we would appreciate knowing which concern remains open, so that we can answer it specifically. For the AC's convenience, the evidence for each of your weaknesses and questions is summarized below; [W1]–[W4] in our first response correspond to Q1–Q4.

  • Weakness 1 / Q1 (concrete limitations and a schematic). Figure 1 of the revised PDF illustrates the two limitations we address with concrete examples: overall modality reliability and per-image degradation are not modeled separately, and cross-modal interactions are not organized by source. Table 3 isolates the second point: with the same products, separate blocks give 84.2% mAP and a merged block gives 83.1%.
  • Q2 (ablations on RGBNT100 and MSVR310). The component ablations on RGBNT100 (85.4 → 89.6) and MSVR310 (65.8 → 70.5) show the same trend as RGBNT201, with mean ± std over 3 runs.
  • Weakness 2 / Q3 (direct evidence for the mechanisms). Global anchors: the learned anchors (TIR 1.178 > RGB 1.026 > NIR 0.827) follow the single-modality test ranking, although they are learned only from training-set identity distances (Fig. 4); under 75% occlusion, RC lowers the weight of the occluded modality for 97.7–99.9% of the 836 queries (Fig. 9). Separate interaction blocks: with a missing modality, the remaining pairwise block keeps the full model at 79.4% mAP for R+T input (Table 7), and separate blocks outperform merged ones (Table 3). Descriptor level: the positive–negative cosine gap rises from 0.250 to 0.403 (Fig. 5).
  • Q4 (modality-wise t-SNE). Figure 10 in Appendix A.7 visualizes RGB, NIR, and TIR features separately, with colors for identities, and shows more compact clusters for MoMI-RC in each modality.

If any of these points does not match what you had in mind, a brief note would let us respond before the deadline.

Reviewer_M9zoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoMI-RC for multi-modal object re-identification using RGB, near-infrared, and thermal-infrared images. The framework combines identity-aware reliability calibration, which learns global modality anchors and bounded sample-specific adjustments, with multi-order modality interaction, which preserves pairwise and three-way feature products in separate descriptor blocks. Modality-subset training and consistency regularization further support representation learning. Experiments on three person and vehicle benchmarks demonstrate competitive retrieval performance.

Critical strengths

The method achieves strong results across three benchmarks, particularly on MSVR310, while introducing limited additional inference cost. The distinction between global modality reliability and sample-specific adjustments is intuitive, and the formulation is clearly presented. Selected backbone-matched comparisons, training-budget controls, missing-modality evaluations, and component ablations provide useful evidence for the framework’s effectiveness.

Critical weaknesses
  1. The methodological advance over established bilinear and low-rank fusion approaches is not sufficiently demonstrated. Comparisons under identical backbone and training settings would help establish the value of retaining interaction sources separately.

  2. The calibration ablations do not independently evaluate the global reliability loss and gating ranking loss, leaving the contribution of the proposed reliability target entangled with additional metric supervision.

  3. The interaction results are mixed: adding MoMI to RC yields only a small gain in Table 2, while adding second-order interactions reduces mAP in Table 8. These findings require further explanation of the modules’ complementarity.

  4. Most ablations focus on RGBNT201, although the largest improvement occurs on MSVR310. Repeated-run statistics are also missing, making the stability of several modest gains difficult to assess.

Questions for authors
  1. MoMI constructs fused representations through low-dimensional projection, second- and third-order element-wise products, and block-wise concatenation. These operations are closely related to existing bilinear pooling, TFN, and low-rank multimodal fusion approaches. The manuscript does not sufficiently establish its substantive methodological advance over these formulations. Comparisons with representative fusion methods under the same backbone and training settings would help clarify the necessity of preserving interaction sources separately.

  2. Table 4 does not separately ablate the global reliability loss LrelL_{\mathrm{rel}} and the gating ranking loss LgateL_{\mathrm{gate}}. It is therefore difficult to determine whether the improvement comes from the proposed reliability calibration mechanism or the additional identity-based metric supervision. Each loss should be removed independently, with an additional control using the same auxiliary ranking supervision, to establish the independent contribution of the reliability target.

  3. The complementarity between the two core modules requires further explanation. In Table 2, RC alone achieves 80.5% mAP, while adding MoMI increases it only to 80.8%. In Table 8, adding second-order interactions to first-order features reduces mAP from 82.5% to 81.1%, and the first-plus-third-order variant achieves higher Rank-1 accuracy than the complete model. These results suggest that additional interaction orders do not consistently help. Further analysis is needed to explain when they are beneficial and whether they introduce redundancy or noise.

  4. Most ablations are conducted on RGBNT201, whereas the largest performance advantage is observed on MSVR310. It remains unclear whether the individual components contribute consistently across datasets. Moreover, several key design choices yield gains of only 0.7–1.1 percentage points, without means and standard deviations over multiple runs. Key ablations on MSVR310 and repeated-run statistics would strengthen the experimental conclusions.

  5. The main paper contains only the framework diagram and a t-SNE visualization, without an introductory figure that clearly illustrates the motivation. Such a figure could show how modality reliability varies under occlusion or low illumination, together with concrete examples of information potentially lost when different interaction sources are merged. This would better motivate the two core modules. Moving selected weight analyses from the appendix into the main paper would also make their behavior easier to understand.

3 comments
AuthorsOpenReview ↗

Additional fusion comparisons, loss ablations, and repeated runs

Thank you for the detailed review.

[W1] Comparison with bilinear, TFN, and LMF. MoMI differs from these heads in how interactions are organized: each pairwise product and the three-way product keeps its own block next to the calibrated first-order features, instead of sharing coordinates with products from other modality combinations. We retrain the alternative heads on RGBNT201 with the same DINOv3-B backbone, RC, subset training, consistency regularization, and schedule as MoMI. Each concatenates the three calibrated first-order features with a 512-d interaction output. MLB-style, TFN, and LMF heads map the products into one shared 512-d block; MoMI uses four separate 128-d blocks. Head parameters include input projections.

HeadmAPR-1Head params
MoMI84.285.5102K
MLB-style82.485.0414K
TFN (16-d projection)82.485.82.62M
LMF (rank 4)82.585.0895K
LMF (rank 1, 4-d bottleneck)82.985.5110K

MoMI reaches the highest mAP with the fewest head parameters, 1.3 points above the parameter-close LMF variant (110K, 82.9%). Because these heads also differ in projection and parameterization, Table 3 gives the controlled comparison of interaction organization: with the same products, merging them into one block yields 83.1% mAP, while keeping them in separate blocks yields 84.2%.

[W2] Separate ablation of LrelL_{\mathrm{rel}} and LgateL_{\mathrm{gate}}. On RGBNT201, all settings except the losses match the full model. Equal-weight ranking fixes g,q=1g,q=1 and applies the same batch-hard loss to the unweighted features. Uniform target keeps both losses and the learnable weights, and only sets the target of LrelL_{\mathrm{rel}} to tm=1t_m=1 for all modalities.

SettingmAPR-1
Full84.285.5
w/o LrelL_{\mathrm{rel}}82.083.9
w/o LgateL_{\mathrm{gate}}78.079.1
w/o both76.878.5
Equal-weight ranking78.980.3
Uniform target (tm=1t_m=1)81.983.9

Both losses contribute (−2.2 and −6.2 mAP). Applied to fixed weights, the same ranking loss gives 78.9%; used to train the learnable reliability, it gives 84.2%, so most of its gain comes through the reliability weights. With both losses kept, replacing the identity-derived target with a uniform one lowers mAP from 84.2% to 81.9%, so the reliability target contributes beyond the extra metric supervision.

[W3] Complementarity and interaction orders. The 80.5→80.8% step in Table 2 is measured without subset training and consistency regularization, which Section 3.4 introduces to stabilize the higher-order products. With both enabled (Table 8), adding MoMI to the first-order features raises mAP from 82.5% to 84.2%. Adding only the third-order block gives 83.5%, while adding only the second-order blocks gives 81.1%. The best result comes from keeping both orders: relative to first-plus-third, mAP rises from 83.5 to 84.2, R-5 from 91.5 to 92.0, and R-10 from 94.0 to 94.6. When a modality is missing, the second-order block of the two available modalities remains usable while the three-way block is zeroed; e.g., the model keeps the RT block under R+T input (79.4% mAP overall, Table 7). The first-plus-third variant has higher R-1 (86.6 vs. 85.5), but the full model is best in mAP, R-5, and R-10, so we keep both orders.

[W4] MSVR310 ablation and repeated runs. Following Table 2, mAP (mean ± std over 3 runs):

Baseline+MoMI+RC+MoMI+RC+SubsetFull
65.8±0.468.8±0.869.0±0.669.4±0.669.9±0.470.5±0.4

Every component raises the mean, consistent with RGBNT201, and the full model is 4.7 points above the baseline. Using the mean, MoMI-RC remains 6.9 points above the second-best method in Table 1. The main and ablation tables of FUSE, Signal, and MDReID report single values without mean ± std, and we followed this format. The 0.7–1.1-point gaps arise between strong variants that differ in a single factor, whereas the main designs give large gains (Table 2: +4.7 for MoMI and +7.3 for RC; Table 4: −7.4 without calibration supervision; Table 10: +8.7 under the same budget). Gaps of this size are also typical between recent methods on RGBNT201 (e.g., FUSE vs. Signal +1.1, MDReID vs. FUSE +0.7).

[W5] Motivation figure. Due to the page limit, these analyses were placed in the appendix. The revised PDF adds a motivation figure to the introduction (Figure 1). The left half illustrates RC: occluding 75% of the RGB image of ID 275 lowers its weight from 1.42 to 0.94. The right half contrasts direct concatenation, which keeps only first-order evidence, with MoMI, which adds the RN, RT, NT, and RNT interactions as separate blocks. The RC weight statistics and the cosine-similarity analysis have also been moved from the appendix into the main text (Section 4.7, Figs. 4 and 5).

Reviewer_M9zoOpenReview ↗

Response to author

The authors have made efforts to address the concerns, but some issues remain:

  1. The methodological novelty of MoMI over existing bilinear and low-rank fusion methods is still not sufficiently established.
  2. The performance degradation caused by adding second-order interactions remains insufficiently explained.

Overall, the revision has improved the manuscript, but it may still not fully meet my expectations. I therefore maintain my original rating and leave the final decision to the Area Chair.

AuthorsOpenReview ↗

Response to the two remaining concerns

Thank you for the follow-up and for noting that the revision improved the manuscript. We understand that the loss ablations, repeated runs, and motivation figure addressed your earlier points, and we respond to the two that remain.

1. What MoMI adds beyond bilinear and low-rank fusion. We do not claim the low-rank element-wise product as new; MLB and LMF established it. Our contribution is how the interactions are organized in the ReID descriptor, which these methods do not address:

  • MLB models a single pair of inputs. LMF appends a constant 1 to each modality before multiplying the projections, so each output coordinate mixes first-, second-, and third-order terms from all modality combinations, and the source of the evidence is lost.
  • MoMI keeps RN, RT, NT, and RNT in separate blocks next to the calibrated first-order features, and the reliability weights act only on the first-order blocks.

This organization produces measurable effects that the operator alone does not. (i) With the same products, separate blocks give 84.2% mAP and one merged block gives 83.1% (Table 3). (ii) Retrained with our backbone and training, MLB-, TFN-, and LMF-style heads reach 82.4–82.9% with 110K–2.62M head parameters, versus 102K for MoMI. (iii) When a modality is missing, only the blocks involving it are zeroed and the rest of the descriptor stays valid (Table 7, R+T: 79.4%); a single fused vector cannot be partially removed in this way.

2. The drop with second-order blocks alone. We agree this result needs a clear position. Table 8 shows that the pairwise and three-way blocks are complementary rather than additive: the pairwise blocks lower mAP on their own (82.5 → 81.1) but raise it once the three-way block is present (83.5 → 84.2), together with R-5 (91.5 → 92.0) and R-10 (94.0 → 94.6). MoMI is defined as this combination, so "first + second" is a partial configuration that we report for completeness, not a variant of our method. Non-additive effects between components are common in fusion designs, and we do not claim that every order must help in isolation.

There are two reasons for keeping the pairwise blocks despite this partial result. First, they are the only interaction evidence available when one modality is missing, because the three-way block requires all three inputs; without them, R+T input would fall back to first-order features only. Second, the gain of the full model over "first + third" is consistent across mAP, R-5, and R-10.

Importantly, the main conclusions do not depend on the second-order result. Even the "first + third" variant (83.5% mAP) is above the best competing method in Table 1 (82.7%), and RC alone raises the baseline by 7.3 points (Table 2). In the final version we will add this discussion to Appendix A.3.

We hope these clarifications resolve the two remaining concerns.

Reviewer_BTD4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses multi-modal object re-identification (RGB + NIR + TIR) by proposing MoMI-RC, comprising two core modules: Identity-Aware Reliability Calibration (RC) and Multi-Order Modality Interaction (MoMI). RC learns global modality anchors supervised by training-sample identity relationships and calibrates unimodal features with bounded sample-wise corrections. MoMI constructs pairwise and three-way interactions in a shared low-rank space, where each interaction occupies a separate coordinate block and is concatenated with calibrated unimodal features. Modality-subset simulation and consistency regularization are further introduced during training to stabilize the descriptor.

Critical strengths

This paper has three main strengths. First, it proposes a clear two-module framework of "reliability calibration + higher-order interaction" that addresses two core challenges in multi-modal fusion with clearly separated module responsibilities that can be combined flexibly. Second, the proposed reliability calibration does not require additional modality inputs at inference time (such as text or segmentation). The proposed method obtains global modality anchors solely through identity-based supervision during training, making itself fully visual at inference and promising for practical deployment. Third, by preserving pairwise and three-way interactions as separate coordinate blocks in a shared low-rank space, the method retains complementary evidence from different modality combinations while controlling parameter size, offering more structural advantages than simple tensor concatenation or low-rank decomposition.

Critical weaknesses
  1. Although the inference phase is indeed free of text or segmentation priors (visual inference), the training phase clearly relies on such auxiliary information as described in Figure 1 and Section 3.3, e.g., text encoders and segmentation paths are used to assist global anchor estimation in the RC module. The abstract claims "without task-specific text or segmentation priors," but Table 1 shows that nearly all compared methods (marked with * or ‡) use task-specific priors of varying degrees. The authors should clarify: (1) the specific way and extent to which text/segmentation aids are used in training; (2) whether these training-time aids are necessary for performance gains; (3) whether the method's performance drops significantly under a training regime without text/segmentation priors. This directly affects the validity of the "no external priors" selling point.

  2. Nearly all compared methods in Table 1 (marked * or ‡) use varying degrees of task-specific priors (e.g., text descriptions, segmentation masks), while the proposed method also relies on auxiliary information during training (see Concern #1). Declaring "without task-specific priors" and achieving SOTA under this asymmetric comparison setup raises fairness concerns. It is recommended that the authors 1) provide detailed comparison with baselines that similarly use priors; 2) retrain and report the proposed method under a setting without task-specific priors; 3) or at least more carefully annotate the comparability of baselines.

  3. The MoMI module constructs interactions in a shared low-rank space with an embedding dimension of 128 (mentioned in Section 4.2). However, the paper provides no theoretical analysis or hyperparameter sensitivity experiment for this choice. In principle, the low-rank dimension affects interaction block capacity and computational cost, but whether 128 dimensions is optimal and whether interaction information is lost due to insufficient dimensionality has not been sufficiently discussed. It is recommended to include ablation experiments or theoretical analysis on the low-rank dimension.

Questions for authors

Please refer to the Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.