acceptodds
Under review as a conference paper at ICLR 2027

Aligning without Homogenizing: Jointly Explainable Common-Specific Decomposition for Multi-Modal Object Re-Identification

16 views

Abstract

Multi-modal object Re-Identification aims to reduce cross-modal discrepancies while preserving identity-relevant modality-specific cues. Previous methods often align representations from heterogeneous modalities uniformly; however, treating common and modality-specific information alike may over-smooth useful modality-specific cues and cause information loss. To address this issue, we propose JESD-ReID, which first distinguishes these two types of evidence using a joint-explainability criterion and then coordinates each according to its role. Jointly Explainable Semantic Decomposition (JESD) uses shared semantic queries to form compact observations with shared query-slot indexing and reconstructs them jointly from a single latent code through constrained modality-conditioned dictionaries. The reconstructed evidence forms the common components, while the remaining token-level evidence is retained as modality-specific residuals. Low-volume regularization organizes the common span, whereas relational optimal transport coordinates local relational structure in the residuals without forcing feature values to coincide. Six lightweight modality–role experts use sample-dependent gates to control how strongly each branch contributes, while a jointly predicted shared grid controls where complementary evidence is sampled for aggregation. JESD-ReID achieves 82.4%, 88.0%, and 66.6% mAP on RGBNT201, RGBNT100, and MSVR310, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Modal Object Re-Identification, Cross-Modal Representation Learning, Common-Specific Decomposition, Multi-Modal Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NP4GOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JESD-ReID to reduce cross-modal discrepancies while preserving complementary information for multi-modal object re-identification. Shared semantic observations are jointly reconstructed from one latent code through constrained modality-conditioned dictionaries. The reconstructed components receive low-volume regularization, while token residuals receive relational optimal transport. Gated modality-role experts and deformable aggregation produce the retrieval descriptor. Experiments on RGBNT201, RGBNT100, and MSVR310 report 82.4%, 88.0%, and 66.6% mAP, with component and property-assignment ablations.

Critical strengths
  1. The paper addresses a relevant problem: reducing modality discrepancy while retaining complementary information. JESD gives the decomposition an explicit shared-code formulation, while allowing the reconstructed components to differ across sensors through constrained dictionaries.

  2. The experiments cover both person and vehicle ReID. The component ablations show consistent improvements, Table 4 supports the proposed regularizer assignment, and Table 6 confirms that combining the two components improves retrieval. The results are competitive, particularly on MSVR310 dataset.

Critical weaknesses
  1. The advantage of the shared-code criterion over existing decomposition strategies is not sufficiently established. Table 3 adds JESD as a whole, leaving unclear whether the gains come from sharing the code or from the surrounding observation and reconstruction design. This limits the evidence for the paper's main contribution.

  2. The contribution of additional inputs and evaluation choices is unclear. Captions and camera metadata are used during both training and inference, and reported scores also use test augmentation. Image-only and single-view results are needed to clarify how much of the improvement comes from the proposed architecture.

Questions for authors
  1. Could the authors include more methods from 2026 to better support the claimed advantage over the state of the art?

  2. How does JESD compare with independently estimated modality codes when the remaining architecture, supervision, and regularization strength are matched?

  3. How does relational transport compare with feature-distance transport under the same JESD and OT settings, with comparable tuning of the loss weight and transport temperature?

  4. What are the separate contributions of the modality-role experts and learned deformable sampling when the remaining components are unchanged?

  5. How sensitive is retrieval performance to the ridge coefficient when the shared-slot number and dictionary-adaptation rank are fixed?

Reviewer_dkoQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores the issue of potential information loss during cross-modal recognition in multi-modal object re-identification (Re-ID). The authors propose a method called JESD-ReID, which employs a joint-explainability criterion to distinguish between shared information and modality-specific information. This criterion is then used to reconstruct the shared information and regulate the contribution of each branch. The authors report strong experimental results.

Critical strengths
  1. The research problem is clearly stated that modal alignment can indeed lead to information loss. The paper investigates this issue, and the core narrative effectively outlines the authors' approach to overcoming the challenges of cross-modal fusion.
  2. The experiments are well-targeted and demonstrate strong overall performance.
Critical weaknesses
  1. The paper attempts to reconstruct different modalities, this approach lacks a solid theoretical foundation, and the experimental results do not convincingly demonstrate that the reconstructed shared information holds meaningful significance.
  2. The training process is not described with sufficient clarity, and key details are omitted. This hinders reproducibility and makes it difficult to verify the authors' underlying hypotheses.
Questions for authors
  1. The authors argue that the initial common information is inadequate and requires further mining to uncover shared semantics. Is there a specific objective for this mining process? Is it targeted at a particular challenge within Re-ID or a specific type of data? Furthermore, how do the authors determine when the shared semantics have been sufficiently extracted? Is there any theoretical or experimental validation for this?
  2. Figure 3 is quite unusual, it features a structural diagram in the top-left corner, while the rest consists entirely of mathematical formulas. This is an unconventional way to present information; what exactly are the authors trying to convey? Could Figure 3 be integrated with Figure 2?
  3. Do the three modalities visible (RGB), near-infrared (NIR), and thermal infrared (TIR) contribute equally to the Re-ID task? Would it be possible to use one of the more significant modalities as a primary reference (anchor) for the fusion process?
Reviewer_6d32OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JESD-ReID for object retrieval using synchronized RGB, NIR, and TIR observations. The implementation also uses per-image captions and camera metadata. Shared, text-conditioned queries convert visual tokens into compact semantic observations. A joint ridge problem estimates one latent code, which approximately reconstructs the observations through modality dictionaries formed from a shared basis and low-rank corrections. The reconstructions are decoded into common token maps, and subtraction from the original tokens defines residual branches. A volume loss regularizes pooled common descriptors, while optimal transport matches local neighbor-distance profiles of residual tokens. Six gated bottleneck experts and a shared deformable sampling grid aggregate the representations. Identity and triplet supervision train the retrieval descriptor. Experiments on RGBNT201, RGBNT100, and MSVR310 report strong performance, with additional component, loss-assignment, correspondence, and input-removal analyses.

Critical strengths
  1. The paper addresses a meaningful problem. Reducing modality differences can suppress useful sensor-dependent evidence, and explicitly assigning different roles to shared and residual features is a reasonable approach. The joint ridge formulation gives the decomposition a clear operational definition. I also appreciate that the authors distinguish this definition from physical factor identifiability and acknowledge that residuals may contain nuisance information.
  2. The reported results are competitive across both person and vehicle retrieval. On RGBNT201, the complete model improves the common baseline from 77.4 to 82.4 mAP and from 81.2 to 87.2 Rank-1 in Table 3. On MSVR310, it reaches 66.6 mAP compared with 64.8 for the strongest listed baseline. These results suggest that the complete framework is useful.
Critical weaknesses
  1. The novelty of this method is somewhat incremental. DeMo [1] already combines feature decomposition with expert aggregation, and MDReID [2] separates shared and specific features with differentiated constraints. IDEA [3] already predicts a sampling grid jointly from the modalities and shares it across streams. Signal [4] uses volume-based global alignment and deformable local alignment, while DecAlign [4] applies different alignment strategies to common and unique features, including OT for the latter in a different task. The proposed restricted dictionaries and joint ridge decomposition are distinct, but their contribution within this combination is not sufficiently isolated.

  2. The performance improvement is marginal in realted datasets. For example compared to STMI, the improvements on MSVR310 and RGBNT100 are limited. Why it work better in person than vehicle datasets?

  3. The evidence for the central joint-explanation mechanism remains indirect. Table 3 removes the complete JESD block, rather than comparing its shared-code criterion with a simpler matched alternative. Figure 9 varies a nonzero dictionary-correction rank but does not test a fully shared dictionary. Table 5 establishes sensitivity to identity-inconsistent groups, which would also affect ordinary multimodal fusion. Its reconstruction error changes only from 0.12365 to 0.12531 when both non-RGB modalities are shuffled, despite a large retrieval drop. The authors interpret this observation cautiously, but it does not establish that the proposed criterion is responsible for preserving complementary identity cues. Moreover, the decoded common map in Eq. 6 also depends on the original tokens through its queries. The operational decomposition is well defined, yet the advantage over a learned low-rank fusion or independently estimated codes remains unverified.

[1] DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification. AAAI, 2025.

[2] MDReID: Modality-decoupled learning for any-to-any multi-modal object re-identification. NeurIPS 2025.

[3] IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identification. CVPR, 2025.

[4] Singnal: Signal: Selective Interaction and Global-local Alignment for Multi-Modal Object Re-Identification. AAAI, 2026.

[5] DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning. ICLR, 2026.

Questions for authors

Please refers to the weakness. I think this paper is somewhat interesting, but show limited novelty and experimental results improvments. Therefore, I think this paper not good enough for ICLR, until now.

1 comment

As of now, I have not received a rebuttal. If there is no rebuttal, I will keep my negative score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.