acceptodds
Withdrawn from ICLR 2027

When Modalities Disagree: Learning adaptive Task-Aware Multimodal Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human-centered multimodal understanding integrates text, audio, and visual cues to infer human sentiment and intent. However, semantic discrepancies, conflicting evidence, and cross-modal redundancy make it challenging to identify task-relevant information and determine appropriate modality contributions during fusion. In particular, predictive disagreement alone cannot determine which evidence should guide the final decision. To address this challenge, we introduce InfoMind, a multimodal representation learning framework that combines shared-private semantic decomposition, asymmetric information-bottleneck regularization, and disagreement-aware latent-state reasoning. Specifically, InfoMind learns compact shared and modality-specific representations and jointly interprets task-level evidence and predictive disagreement through a latent task state, which guides sample-specific fusion of the original multimodal features. Comprehensive experiments across four benchmarks, including CMU-MOSI and CMU-MOSEI for sentiment prediction and MIntRec and MIntRec2.0 for intent recognition, demonstrate the effectiveness and generality of the proposed framework.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Modality Disagreement, Task-Aware Multimodal Learning, Adaptive Multimodal Fusion, Latent State Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Tmz7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

InfoMind proposes a task-oriented, adaptive multimodal representation learning framework designed to dynamically determine the contribution of different modalities to a final task, particularly when these modalities provide inconsistent evidence. The method first constructs shared and residual representations for non-textual modalities using a text-conditioned diffusion model, while employing an asymmetric information bottleneck and task-conditioned residual dependency regularization to learn compact, task-specific representations. Subsequently, the model feeds the shared context, modality-specific task evidence, and pairwise prediction discrepancies into a latent task-state inference module. Through a single state update, it derives sample-specific modality weights, which are then used to fuse the shared interactive representations with modality-private representations. The model demonstrates strong performance across various tasks, and the roles of its core components are validated through loss ablation, structural ablation of the state inference mechanism, and qualitative analysis.

Critical strengths
  1. The approach of using both task-level evidence and prediction discrepancies to guide the weighting of modalities is well-motivated. Rather than equating inter-modal differences directly with modal reliability, InfoMind first maps auxiliary predictions from various modalities into a common task space; it then infers the contribution of each modality by incorporating shared context and pairwise prediction discrepancies. This design is conceptually sound and distinct from simple gating mechanisms that calculate weights based solely on raw features.

  2. The experiments cover a diverse range of tasks and include ablation analyses for key components.

Critical weaknesses
  1. The originality of the proposed method relative to existing approaches is not sufficiently established. There are clear architectural similarities between InfoMind and DiffuFuse (MM 2025), specifically regarding text-conditioned diffusion-based shared/private decomposition, residual construction, dual-stream fusion, and projection-based redundancy control. While InfoMind introduces task-level evidence, an asymmetric information bottleneck, and latent task-state inference, the current experiments fail to adequately isolate the contributions of these added components. Furthermore, the study does not sufficiently demonstrate that these additions provide a significant, independent advancement over existing diffusion-based fusion architectures. Additionally, the authors should clarify the distinctions between their work and EMOE (CVPR 2025).

  2. In Table 1 of InfoMind, the results for EMOE are reported as 46.5% Acc-7 and 0.719 MAE on the CMU-MOSI dataset, and 53.55% Acc-7 and 0.527 MAE on the CMU-MOSEI dataset. However, the original EMOE paper reports different figures for CMU-MOSI: 47.7% Acc-7 and 0.710 MAE for the aligned setting, and 47.8% Acc-7 and 0.697 MAE for the unaligned setting. The results for CMU-MOSEI also do not fully align with those presented in the InfoMind table. The authors should provide a reasonable explanation for these discrepancies. Furthermore, while the original EMOE paper reports ACC and F1 scores on the MIntRec dataset, EMOE is omitted from Table 2 in InfoMind. Given that InfoMind claims to be generalizable to intent recognition tasks, the authors should include this baseline, evaluated under the same experimental settings.

Questions for authors
  1. The originality of the proposed method relative to existing approaches is not sufficiently established. There are clear architectural similarities between InfoMind and DiffuFuse (MM 2025), specifically regarding text-conditioned diffusion-based shared/private decomposition, residual construction, dual-stream fusion, and projection-based redundancy control. While InfoMind introduces task-level evidence, an asymmetric information bottleneck, and latent task-state inference, the current experiments fail to adequately isolate the contributions of these added components. Furthermore, the study does not sufficiently demonstrate that these additions provide a significant, independent advancement over existing diffusion-based fusion architectures. Additionally, the authors should clarify the distinctions between their work and EMOE (CVPR 2025).

  2. There remains a substantial gap between the core semantic assumptions and the experimental evaluations. Since the actual shared representation is derived from random samples within the conditional diffusion distribution, rather than using the conditional mean directly, there is no guarantee that this representation contains only the shared semantics predictable from the text. Moreover, the paper does not sufficiently account for the impact of sampling randomness on final predictions, nor does it include systematic stress tests for modal conflicts.

  3. Regarding baseline comparisons, the numerical values ​​reported for EMOE are inconsistent with those in the original paper, and the MIntRec results for EMOE are omitted from the intent recognition comparison table. Furthermore, variations in feature extraction pipelines make it difficult to fairly assess the validity of the primary results.

  4. The baseline comparisons lack coverage of the latest research from 2026; the authors should consider conducting a comprehensive comparison with papers published at top conferences in 2026 (e.g., CVPR 2026, ICML 2026, ICLR 2026).

Reviewer_GRifOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes InfoMind for multimodal sentiment analysis and intent recognition. It uses text-conditioned diffusion to construct shared and residual representations for audio and vision. A compact reasoning branch combines asymmetric information bottleneck regularization, task-conditioned dependence regularization, and auxiliary task supervision to produce modality-associated evidence. Pairwise disagreement is computed from the auxiliary predictions in a common task space. A latent task state is initialized from shared context and updated once using the evidence and disagreement. The updated state determines modality weights, which guide the fusion of retained full-width shared and private features for final prediction. The authors evaluate InfoMind on CMU-MOSI, CMU-MOSEI, MIntRec, and MIntRec2.0, report improvements over the baselines, and provide structural ablations to examine its main components.

Critical strengths

Strong empirical results across sentiment prediction and intent recognition. The authors report gains on four benchmarks covering both continuous sentiment prediction and multiclass intent recognition. On MIntRec, for example, it achieves 77.35% accuracy and 74.78% macro-F1, compared with 73.71% and 70.33% for the strongest listed baseline. Results across these two task settings provide useful evidence for the predictive value of the framework. The loss and structural ablations also provide initial evidence that the training objectives and routing components contribute to its performance.

Critical weaknesses

W1. The benefit of latent task-state refinement over Direct Gating is not convincingly established.

The paper presents a shared-context-initialized state as the intermediate step separating “evidence interpretation” from “contribution assignment.” but the advantage of this organization over direct gating remains unclear. Section 3.4 argues that predictive disagreement alone cannot determine modality contributions, motivating the use of context and evidence. However, Direct Gating already receives the same context, evidence, and disagreement inputs. The key question is what explicit state refinement adds beyond direct access to this information. Equation (12) summarizes the update as Ψ\Psi, while the scoring and update networks remain insufficiently specified to assess this distinction.

Table 4 reports gains over Direct Gating, including MIntRec ACC from 75.82% to 77.35% and MOSI MAE from 0.581 to 0.561. However, it is still difficult to attribute the gains to the proposed separation of evidence interpretation and contribution assignment. Please explain the modeling advantage, specify both implementations, and compare them under similar model capacity.

W2. The claimed benefit when multimodal information conflict occurs is not directly demonstrated.

Handling conflicting evidence is central to the paper’s motivation. The conclusion explicitly links the results to separating evidence interpretation from contribution assignment “when multimodal cues conflict.” However, the experiments mainly establish improvements averaged over the full test sets. It does not show how state refinement improves decisions on conflicting samples.

Figure 4 presents three successful examples, such as Sample 311. The final prediction follows the text branch despite agreement between the audio and visual branches. This shows that the model need not follow a majority vote, but does not establish that refinement produces the correct decision. Other variants may yield the same result.

Please compare the full model, Direct Gating, and the model without refinement on the same samples, partitioned using a clearly defined disagreement criterion. Report both corrections and new errors introduced by refinement. This would directly test whether the proposed state refinement provides the claimed benefit for conflicting evidence.

Questions for authors

1. Question about using a generated sample as the shared representation.

Section 3.2 defines the ideal shared component as the conditional mean of the non-textual representation given text. The implemented shared stream, however, uses a sample generated from the learned conditional distribution. Why is a conditional sample used instead of an estimate of the conditional mean, and what makes that sample suitable for representing the text-predictable shared component?

2. Incomplete specification of the state-update network and routing scorer.

Equations (12)–(13) introduce the state-refinement function and the contribution scorer through their inputs and outputs. Please provide the architecture of the state-update network fupdf_{\mathrm{upd}} and the routing scorer sϑs_{\vartheta}. In particular, how does the scorer combine the state, modality evidence, and disagreement?

3. Missing definitions and settings for the training objective.

Equation (16) includes the final task loss LtaskL_{\mathrm{task}} but doesn't specify what losses are used for final sentiment regression and intent classification. The authors should also report what the four auxiliary loss weights in Equation (16) are for each dataset and how to choose them through parameter adjustment.

4. Insufficient definitions of the structural ablations in Table 4.

The variant names do not fully specify the computational changes. For “w/o Disagreement Input,” is disagreement removed from both the state update and the contribution scorer? For “w/o Shared-Context Init.,” what replaces the shared-context initialization? For “w/o State Refinement,” how are the final modality weights computed?

5. Limited evidence for conflict handling in Figure 4.

In all three examples in Figure 4, the text branch already predicts the correct label, and the full model produces the same prediction. What evidence distinguishes the proposed conflict-handling mechanism from this explanation? In particular, do the reported analyses show cases where the text branch is incorrect, and the model successfully uses audio or visual evidence to correct the prediction?

Reviewer_CB1oOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces InfoMind, a multimodal fusion framework that formulates fusion as sample-specific latent task-state inference. The method uses text-conditioned diffusion to decompose non-textual modalities (audio, visual) into shared and private streams, applies asymmetric information-bottleneck regularization to construct compact task-relevant representations, computes pairwise predictive disagreement from auxiliary task predictions, and refines a shared-context-initialized latent state to determine normalized modality contributions. The contributions are converted into feature gates that modulate full-width shared and private representations for final prediction. Experiments are conducted on CMU-MOSI, CMU-MOSEI (sentiment analysis) and MIntRec, MIntRec2.0 (intent recognition).

Critical strengths

Thorough ablation and analysis are provided. The paper provides both training-objective ablations (Table 3) and structural ablations (Table 4), together with evidence reliability visualizations (Figure 3) and qualitative case studies on conflicting samples (Figure 4), which help verify the contribution of individual components.

Critical weaknesses
  1. The complex system lacks clear and direct motivation. The proposed framework combines many components and loss terms, but the paper does not clearly explain the specific problem each addresses or why their combination is necessary. Also, the concrete model architecture is not clear.
  2. This paper is pretty hard to follow. Essential symbols are not clearly defined in the main text. Key symbols in the method section, such as u_m, v_m (line 188), s_ϑ(·) (Eq. 13), c_sh (line 264), and f_init/Ψ/f_upd (Eq. 12), are never defined in the main text, making the pipeline incomprehensible without the appendix.
  3. Table 1 has extensive missing entries with unclear reproduction protocol. Many baselines have missing metrics, strongly suggesting results are copied from original papers rather than reproduced under identical features. Since InfoMind uses T5 while older baselines may use different text encoders, the comparison is potentially unfair, yet the paper provides no clarification.
Questions for authors
  1. Are the baseline results in Table 1 reproduced using the same feature extractors as InfoMind, or are they directly taken from the original papers? If the latter, how do you ensure fair comparison?
  2. Could the authors clarify the concrete model architecture used for the diffusion denoisers, latent-state initialization and update, and modality contribution scoring?
  3. What are the training and inference costs of InfoMind compared with representative baselines, in terms of runtime and memory usage? In particular, how much computational overhead does the diffusion module introduce?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.