acceptodds
Under review as a conference paper at ICLR 2027

Disentangled Representation Learning with Dual-Branch Enhancement and Adaptive Cross-modal Interaction

Abstract

Multimodal Sentiment Analysis (MSA) aims to capture human sentiment by jointly leveraging heterogeneous information from text, speech, and vision. While existing feature disentanglement methods partially separate modality-specific and modality-invariant representations, there remain two key limitations: (i) the disentangled representations lack targeted enhancement. Residual redundant temporal dependencies and intra-modal noise impair the discriminative capability, while insufficient cross-modal semantic interaction result in incomplete shared semantics, and (ii) the interaction features generated by cross-modal transformers are typically treated equally, which may amplify redundant and noisy dependencies while diluting the contribution of critical sentiment cues. To address these issues, we propose a disentangled representation learning framework with dual-branch enhancement and adaptive cross-modal interaction. Specifically, we first design a Dual-Branch Masked Representation Enhancement (DBMRE) module, which employs intra-modal masked attention and inter-modal masked attention to enhance modality-specific and modality-invariant representations, respectively. We then propose an Adaptive Cross-modal Interaction Gating (ACIG) mechanism, which dynamically estimates channel-wise importance weights based on the global semantic context of each interaction branch, selectively enhancing sentiment-relevant cross-modal dependencies while suppressing redundant interactions. Furthermore, we introduce multi-level knowledge distillation and dictionary-guided contrastive learning to provide auxiliary supervision at different representation levels. Extensive experiments demonstrate that our proposed framework achieves competitive performance. Furthermore, comprehensive ablation studies and training dynamics analyses validate the synergistic effectiveness of DBMRE and ACIG in enhancing both discriminative representation learning and model convergence.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Feature Disentanglement, Dual-Branch Masked Representation Enhancement, Adaptive Cross-modal Interaction Gating, Knowledge Distillation, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_oiKSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper’s architecture diagrams and experimental results exhibit extensive signs suggestive of “autoresearch”-style generation. I consider this disrespectful to both ICLR and the reviewers, and therefore believe the paper should be rejected outright.

Critical strengths

None

Critical weaknesses

The paper’s architecture diagrams and experimental results exhibit extensive signs suggestive of “autoresearch”-style generation. I consider this disrespectful to both ICLR and the reviewers, and therefore believe the paper should be rejected outright.

Questions for authors
  1. Clarification on AI/“autoresearch” usage. Several parts of the manuscript, including the architecture diagrams, experimental analyses, and highly templated descriptions, raise concerns about possible broader use of AI-assisted or “autoresearch”-style tools. I could not find a clear AI-use disclosure in the manuscript. Could the authors explicitly clarify whether AI tools were involved in research ideation, method design, implementation, experimental design, data analysis, result interpretation, or figure/table generation? ICLR 2027 requires disclosure of such AI-assisted research activities.

  2. Validity of the orthogonality constraint. The paper defines Lort=∑mmax⁡(0,cos⁡(Sm,Cm))L_{\mathrm{ort}}=\sum_m\max(0,\cos(S_m,C_m)) to encourage orthogonality between modality-specific and modality-invariant representations. However, when cos⁡(Sm,Cm)<0\cos(S_m,C_m)<0, the loss becomes zero even if the two representations remain strongly correlated in opposite directions. Could the authors explain how this objective actually enforces orthogonality or decorrelation, and whether alternative symmetric penalties were considered?

Reviewer_Mp3FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a multimodal sentiment analysis framework built upon DHMD. It introduces Dual-Branch Masked Representation Enhancement (DBMRE), which applies intra-modal and inter-modal attention to modality-specific and modality-invariant representations, respectively. Adaptive Cross-modal Interaction Gating (ACIG) further reweights cross-modal interaction features through channel-wise gating. The framework also incorporates hierarchical knowledge distillation and dictionary-guided contrastive learning. Experiments on CMU-MOSI and CMU-MOSEI report improvements over the authors' reproduced DHMD baseline, with ablation studies evaluating the contributions of DBMRE and ACIG.

Critical strengths
  1. The paper provides experiments on two standard multimodal sentiment analysis benchmarks, CMU-MOSI and CMU-MOSEI. The component ablations and alternative design comparisons offer some evidence of the proposed modules' effectiveness.
Critical weaknesses
  1. There are serious inconsistencies between the proposed method and its implementation. In the released code, the intra-modal and inter-modal attention masks are not actually applied to the attention computation, raising concerns about whether the experiments validate the core DBMRE mechanism. Moreover, the reproduced DHMD baseline performs substantially worse than the originally reported results, further weakening the experimental conclusions.

  2. The methodological novelty is limited. The framework largely extends DHMD by adding two masked attention branches and a conventional channel-wise gating mechanism. These components are closely related to existing techniques, and the ablation studies do not adequately establish their distinct contributions, particularly the benefit of the proposed masking strategy over standard self-attention.

Questions for authors
  1. I checked the original DHMD paper [1] and noticed substantial differences in the reported results. For example, DHMD reports 87.8% and 87.1% Acc-2 on CMU-MOSI and CMU-MOSEI, respectively, compared with 83.69% and 84.06% in Table 1. I also could not find an official DHMD code release, only the publicly available DMD implementation [2]. Did the authors use the official DHMD code or reproduce it independently? What explains these large performance gaps? I also suggest including DMD as an additional baseline.

[1] Li et al., Decoupled Hierarchical Distillation for Multimodal Emotion Recognition. IEEE TPAMI, 2026.

[2] Li et al., Decoupled Multimodal Distilling for Emotion Recognition. CVPR, 2023.

  1. I examined the implementation of the two proposed modules and found a potential issue with DBMRE. In model/dmd.py, intra_mask and inter_mask are passed as the second argument to TransformerEncoder, but its forward(self, x_in, x_in_k=None, x_in_v=None) treats this argument as x_in_k rather than an attention mask. The masks therefore appear unused, which contradicts Eqs. (10) and (13). Could the authors clarify how these masks are applied and whether the ablation results actually validate DBMRE? Additionally, the linked GitHub repository appears to reveal author-identifying information, which may violate ICLR's double-blind policy. I suggest providing an anonymous version.

  2. The proposed framework largely follows DHMD, with two additional modules based on masked attention and channel-wise gating. Similar intra-/inter-modal interaction and gating strategies have been explored in Semi-IIN [3]. Could the authors clarify the main methodological novelty beyond integrating these existing techniques into DHMD?

[3] Lin et al., Semi-IIN: Semi-Supervised Intra-Inter Modal Interaction Learning Network for Multimodal Sentiment Analysis. AAAI, 2025.

  1. I could not find the AI use statement required by ICLR 2027. Could the authors provide this statement? Additionally, Figures 1 and 2 have a strongly AI-generated visual style, and I suggest replacing them with clearer technical diagrams. This point does not affect my rating.
Reviewer_zskxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a disentangled representation learning framework for multimodal sentiment analysis, aiming to improve the discriminative quality of modality-specific and modality-invariant representations while mitigating redundant and noisy cross-modal interactions. Specifically, the authors introduce Dual-Branch Masked Representation Enhancement (DBMRE), which applies intra-modal and inter-modal masked attention to refine the two types of representations, respectively. Furthermore, Adaptive Cross-modal Interaction Gating (ACIG) dynamically reweights the channels of cross-modal interaction features based on their global semantic context. The framework also incorporates multi-level knowledge distillation and dictionary-guided contrastive learning to provide auxiliary supervision for intermediate representations. Experiments suggest the effectiveness of the model.

Critical strengths
  1. The paper addresses a relevant issue in multimodal feature disentanglement. Separating modality-specific and modality-invariant representations does not necessarily ensure that both are sufficiently discriminative, while subsequent cross-modal interaction may introduce redundant dependencies and obscure important sentiment cues. To address these issues at different stages, the authors introduce DBMRE, which employs intra-modal and inter-modal masked attention to refine the two types of representations, and ACIG, which adaptively reweights interaction features based on their global semantic context. The overall design is reasonably well motivated, with each module addressing a distinct aspect of representation learning and cross-modal interaction.

  2. The experiments cover two widely used MSA benchmarks, CMU-MOSI and CMU-MOSEI. In addition to evaluating overall performance, the authors conduct ablation studies to examine DBMRE, ACIG, and alternative masking and gating strategies. In particular, comparisons with static linear gating and random feature masking provide further evidence for the contribution of the proposed design choices. The results also indicate that DBMRE and ACIG play different roles, with their combined use yielding improvements on several evaluation metrics.

Critical weaknesses

1.Although the manuscript lists the authors as anonymous and states that it is under double-blind review, the Introduction includes a GitHub URL associated with the account “lin20021113”. This may compromise reviewer anonymity and raise concerns about compliance with the venue's double-blind submission policy.

2.The manuscript contains several inconsistencies with the prescribed ICLR formatting conventions. In particular, the in-text citations mix author–year and numeric styles, as illustrated by “[Li and Liu, 2025; He et al., 2026; Xu et al., 2025]” and “CMU-MOSI [30] and CMU-MOSEI [31]”. Moreover, the body paragraphs appear to use first-line indentation, which is inconsistent with the flush-left paragraph formatting specified by the template. These issues indicate insufficient attention to the required submission format.

3.The mathematical notation is not consistently defined across the manuscript. For instance, the reconstruction objective is described using F_m in the main text, whereas Appendix B.1 formulates it in terms of Z_m, without clearly establishing the relationship between these representations. In addition, the reuse of \lambda_1,\lambda_2,\lambda_3 across different levels of the loss formulation makes the roles of these coefficients difficult to distinguish.

4.Appendix B.4.2 refers to a “cross-entropy term” in the heterogeneous distillation objective, whereas Eq. (26) defines an absolute-error objective for the auxiliary prediction. Given that the main task involves continuous sentiment regression, the supervision objective for this component remains unclear.

Questions for authors

1.Regarding the dictionary-guided contrastive learning module, how is the shared dictionary constructed and updated, and how are positive and negative samples identified during training?

2.The ablation results show that ACIG alone does not consistently improve performance, whereas the full model achieves better results on several metrics. What accounts for this difference, and how sensitive is ACIG to the quality of its input representations?

3.How exactly are the heterogeneous distillation loss and its auxiliary prediction objective implemented, given the discrepancy between the “cross-entropy term” in Appendix B.4.2 and the absolute-error objective in Eq. (26)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.