acceptodds
Withdrawn from ICLR 2027

Robust Multimodal Sentiment Analysis via Calibrated Dynamic Fusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Sentiment Analysis (MSA), which integrates linguistic, acoustic, and visual modalities, is crucial for understanding human emotions. Since modality quality often varies in real-world scenarios, dynamic fusion methods have been introduced to adaptively weight modalities according to their estimated reliability. However, existing methods still suffer from two key limitations: they often confuse degradation caused by intrinsic noise with that caused by insufficient optimization, inducing erroneous modality modulation. Besides, they usually rely on simple probabilistic assumptions that cannot capture the complex, multi-peak structure of real modality features. To address these issues, we propose a Calibrated Dynamic Fusion (CDF) framework for robust MSA. Specifically, we introduce Degradation Source Rectification (DSR) to reduce optimization-induced uncertainty bias, preventing informative but under-optimized modalities from being unduly downweighted. We further propose Multi-Peak Uncertainty Modeling (MPUM) to capture heterogeneous local patterns in modality representations and provide more faithful uncertainty estimates under complex feature distributions. The calibrated uncertainty estimates are then used to assess modality reliability and guide sample-adaptive dynamic fusion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Analysis, Dynamic Fusion, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YXLgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Calibrated Dynamic Fusion (CDF) for robust multimodal sentiment analysis. It combines Degradation Source Rectification (DSR), which uses unimodal supervision and gradient decoupling to reduce optimization-induced representation bias, with Multi-Peak Uncertainty Modeling (MPUM), which employs sample-conditional Gaussian mixtures to estimate within-component and between-component predictive uncertainty. The resulting uncertainty estimates guide sample-adaptive modality fusion. Experiments on CMU-MOSI, CMU-MOSEI, and MVSA-Single report competitive performance, supported by component ablations, calibration analyses, and controlled degradation experiments.

Critical strengths
  1. The paper addresses a meaningful issue in uncertainty-aware multimodal learning by distinguishing representation degradation caused by insufficient optimization from that caused by corrupted inputs.

  2. The experimental analysis provides useful support for the proposed framework.

Critical weaknesses
  1. In Eqs. (11)–(13), the proposed aleatoric and predictive uncertainty terms correspond to the within-component and between-component variance of a Gaussian mixture. While this decomposition is mathematically valid, between-component disagreement does not necessarily represent uncertainty about modality quality, and a larger predictive variance does not always imply less task-relevant information. Although Table 3 reports improved expected calibration error and Figure 4 shows sensible uncertainty trends under image corruption, these results do not directly establish whether the estimated uncertainty can reliably distinguish informative from degraded modalities at the sample level. In particular, the ECE evaluation concerns predictive calibration rather than the correctness of the modality reliability rankings used in fusion. I would like to see more targeted analyses, such as evaluating the relationship between estimated uncertainty and unimodal prediction errors under matched corruption conditions, or comparing uncertainty-based modality rankings against independently assessed modality quality.

  2. The main performance gains are established on the tri-modal CMU-MOSI and CMU-MOSEI benchmarks, whereas the corruption experiments are conducted primarily on the image-text MVSA-Single dataset. This leaves an important gap in evaluating whether DSR and MPUM improve reliability-aware fusion when acoustic and visual quality vary in the intended tri-modal setting. Furthermore, the comparisons omit closely related recent approaches such as EBMC [1], which addresses modality competition and instance-level reliability estimation, and QA-MoE [2], which uses uncertainty-aware expert routing for robust sentiment analysis.

Refs:

[1] Kang He et al. Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis. CVPR 2026.

[2] Yitong Zhu et al. QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis. ACL 2026.

Questions for authors
  1. How do the authors establish that the within-component and between-component variances of MPUM accurately reflect modality reliability rather than general predictive ambiguity? Is there evidence that lower estimated uncertainty consistently corresponds to more informative or more accurate unimodal predictions under comparable corruption conditions?

  2. Table 3 reports improvements in ECE, while the fusion weights are computed from the mixture uncertainty in Eq. (13). How closely do these uncertainty estimates align with independently measured modality quality or prediction errors, beyond conventional probability calibration?

  3. Appendix A.3.3 demonstrates that under-optimization and input corruption can produce similar uncertainty responses. After applying DSR, does the model become better at distinguishing these two sources, or does it primarily reduce uncertainty through improved unimodal optimization? Can the authors clarify the evidence supporting the claimed rectification mechanism?

  4. I would like to see robustness experiments on CMU-MOSI or CMU-MOSEI under controlled modality-specific corruption, ideally including uncertainty estimates, fusion-weight changes, and downstream performance across different noise severities. This would more directly test the proposed mechanisms in the main tri-modal setting.

  5. Please consider expanding the comparisons to closely related recent methods, particularly EBMC [1] and QA-MoE [2], under matched protocols where feasible. Although these methods are relatively recent, their close relevance to modality competition, uncertainty estimation, and reliability-aware fusion makes a comparison or more detailed discussion valuable.

Reviewer_dNwSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces CDF, a dynamic fusion framework for multimodal sentiment analysis with two modules. Degradation Source Rectification (DSR) uses auxiliary unimodal losses and stop-gradient decoupling: unimodal losses train encoders while the fusion loss optimises only the fusion branch, to avoid spuriously high uncertainty for modalities that remain under-optimised. Multi-Peak Uncertainty Modelling (MPUM) constructs a sample-conditioned 3-component Gaussian mixture for each modality’s sentiment score, using an Entmax‑1.5 router and Gaussian experts. Training uses NLL together with a KL loss that aligns routing weights to label-guided component likelihoods. Modality uncertainty combines aleatoric dispersion within components and predictive disagreement between component means, and fusion weights scale inversely with uncertainty. CDF obtains the best results across all six metrics on CMU-MOSI and CMU-MOSEI, and shows the strongest robustness to Gaussian and salt-and-pepper corruption on MVSA-Single. Even so, the method largely assembles known techniques. The mixture-based uncertainty “decomposition” is, in essence, the standard total-variance identity rephrased. The central claim about calibration has limited support, relevant statistical results are not provided, and ablation experiments are conducted only on the smallest dataset.

Critical strengths
  1. Clear writing with an honest note on non-monotonic acoustic weights in the controlled analysis.
  2. The claim that optimization induced and corruption induced degradation produce indistinguishable uncertainty signatures is clean.
Critical weaknesses

Several issues raised. The main issues it the novelty of the paper is rather limited.

  1. This sample‑conditional Gaussian mixture corresponds to a classical mixture density network (Bishop, 1994). Unimodal auxiliary supervision and gradient modulation are well‑established techniques (Peng et al., 2022; Wei et al., 2024/2025 — cited yet not clearly differentiated). Most critically, Equations 11 and 12 exactly recover the two terms from the law of total variance: the sum of aleatoric uncertainty and predictive uncertainty under unit weights equals the total predictive variance of the mixture, a standard textbook identity. The paper frames this “complementary aleatoric and predictive components” decomposition as a design contribution, without acknowledging that it merely reweights the total variance. The routing‑consistency loss (Eqs. 8–9) represents the only genuinely novel component, and its novelty remains modest.
  2. The motivation conflates dataset-level and per-sample multimodality. Figures 1, 7 and 8 justify K>1 by fitting GMMs to dataset-level PCA projections of features (NLL 1815 vs. 1983 and so on). Yet MPUM assumes a multi-peaked conditional distribution per sample over a scalar sentiment score. Any mixture model will fit a heterogeneous dataset better than a single Gaussian; this does not demonstrate that individual samples themselves require multiple components. This distinction is important, as the per-sample mixture forms the core mechanism of the method. Furthermore, NLL comparisons on two-dimensional PCA projections offer only anecdotal evidence.
  3. Calibration is assessed only via ECE on MVSA‑Single for two modalities, and solely within the internal 2×2 ablation study shown in Table 3. There are no reliability diagrams, no calibration results reported for MOSI or MOSEI, and no ECE comparisons against external baselines. For a paper that includes “Calibrated” in its title, this evidence base is rather thin.
  4. Three seeds are used, yet standard deviations are not reported anywhere. The reported performance gains over MODS range from 0.71 to 1.39 points, which correspond to a small number of test samples on MOSI (n=686). No significance tests are provided. Most ablation studies (Tables 4, 5, 8; Figures 3, 5, 6) are conducted only on MOSI, even though MOSEI is ten times larger. The robustness table copies baseline values from CPSC instead of re-running the baselines themselves. In addition, MVSA-Single is a two-modality dataset (image plus text), so it cannot evaluate the three-modality scenario that this method aims for.
  5. Section 3.1 defines \mathcal as cross-entropy, yet Table 7 lists the task loss as MSE applied to continuous labels in the range [−3,3][-3,3]. The routing‑ambiguity coefficient γ\gamma appears in Table 7 but is never defined or utilised within the method section. Figure 4’s core supporting evidence for DSR relies on just one single test image (n=1n=1). The caption for Figure 2 contains grammatical error: “DSR reduce the optimization basis”. There are also multiple awkward passages throughout the manuscript, for instance repetitive sentence structure around Equations 7 and 10.
Questions for authors
  1. Can you provide evidence that the per‑sample conditional distributions are genuinely multi‑peaked, for example via post‑hoc analysis of the learned mixtures on ambiguous samples, rather than merely dataset‑level feature clusters?

  2. Why does γ appear in Table 7 without any definition, and is the task loss cross‑entropy or MSE?

  3. What behaviour does DSR show on CMU‑MOSEI and MVSA‑Single; does the reported ablation ordering (full decoupling > US > PD/SD) hold for those datasets?

  4. Can the analysis shown in Figure 4 be aggregated across many images with confidence intervals, given that it currently relies on only a single sample?

  5. How sensitive are the observed MOSI performance gains to random seeds, as no variance measures are reported?

Reviewer_u4fVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Calibrated Dynamic Fusion (CDF) for multimodal sentiment analysis. Its Degradation Source Rectification module uses unimodal sentiment supervision and gradient separation to reduce under-optimization of modality encoders. Its Multi-Peak Uncertainty Modeling module fits a sample-conditional Gaussian mixture over sentiment predictions and combines within-component dispersion with disagreement among mixture components to derive modality reliability weights. The resulting weighted proxy representation is passed to an attention-based multimodal predictor. Experiments on CMU-MOSI and CMU-MOSEI report competitive sentiment prediction performance; results on MVSA-Single examine image corruption, and ablations study mixture components, uncertainty terms, and gradient-decoupling strategies.

Critical strengths
  1. The paper targets a concrete failure mode in uncertainty-guided multimodal fusion. Poor unimodal predictions can result from insufficiently optimized encoders as well as corrupted input signals. Treating both as equally unreliable may suppress useful modalities. The distinction is meaningful and is illustrated by the controlled under-optimization and corruption analyses in Appendix A.3.3.
  2. The submission provides useful empirical support for the overall system. CDF improves sentiment metrics on CMU-MOSI and CMU-MOSEI and performs well under the tested Gaussian and salt-and-pepper corruptions on MVSA-Single. Table 8 distinguishes full gradient decoupling from unimodal supervision alone, partial decoupling, and staged decoupling; Table 3 reports calibration improvements from the mixture model. These experiments are more informative than relying only on the full model's final accuracy.
Critical weaknesses
  1. The proposed reliability weights do not clearly control the information reaching the final predictor. Equation (2) constructs a weighted proxy from modality features, but the subsequent attention layer also directly receives the original language, acoustic, and visual representations. These direct paths may allow the predictor to use a degraded modality even when its estimated reliability weight is low. The proxy can still influence attention, so this is not proof that the weights are ignored; nevertheless, the architecture does not establish that noise robustness is caused by uncertainty-guided suppression. The current component ablations do not isolate this pathway. A controlled comparison among the weighted proxy without access to raw modality features, an attention model with uniform reliability weights, and the complete architecture is needed to identify the effect of the proposed weighting mechanism.

  2. The evidence does not establish the claimed rectification of optimization-induced uncertainty bias as distinctly from ordinary unimodal representation improvement. DSR supervises each modality with the task label and prevents fusion gradients from updating its encoder. These operations can improve unimodal learning, but they do not explicitly identify whether a representation is weak because of inadequate optimization or because of input corruption. Appendix A.3.3 helpfully demonstrates that these two degradation sources can elicit similar uncertainty changes, and Table 8 shows that full gradient separation improves prediction over alternative optimization strategies. Neither result directly shows that DSR corrects the relative reliability assigned to a genuinely under-optimized modality while preserving the suppression of a genuinely corrupted one under matched conditions. In Table 3, DSR added to the single-Gaussian model actually slightly increases both reported calibration errors, so its calibration benefit should not be inferred from predictive performance alone. The paper needs a more direct before-and-after analysis of the estimated reliability and final predictions for the two degradation sources.

  3. The distributional motivation and the calibrated-reliability claim are not fully aligned with what the mixture model estimates. Figure 1 uses a better Gaussian-mixture fit to the distribution of projected modality features to motivate multi-peak uncertainty modeling, whereas Equation (5) fits a mixture to the conditional distribution of the scalar sentiment label given a modality representation. Multimodality in feature space does not by itself imply multimodality of the conditional label distribution for an individual input. The improved calibration errors in Table 3 are relevant positive evidence, but the paper does not make sufficiently clear how the stated modality-wise expected calibration error is computed from the conditional mixture or whether it reflects the final reliability weights used by the fusion model. This leaves uncertainty about whether MPUM's gains arise from better-calibrated reliability estimation or simply a more flexible prediction and regularization module.

  4. The bibliographic information for Sun et al. (2024), "Learning Specific and Shared Representations for Multimodal Sentiment Analysis," could not be independently verified using the reported title and DOI (10.1109/TMM.2024.3353556). The authors should verify this reference and provide a valid source. In addition, Appendix A.3.1 (line 804) contains an unresolved figure reference ("Fig. ??"). These issues raise concerns about the care taken in preparing the manuscript and should be corrected.

Questions for authors
  1. In Equation (2), what prevents the attention layer from bypassing low reliability weights through its direct access to the original modality representations? Can the authors report a controlled comparison that separates the contribution of reliability weighting from that of the attention block?
  2. In the controlled under-optimization and corruption conditions of Appendix A.3.3, how do the estimated uncertainty, fusion weights, and downstream predictions change with and without DSR? In particular, does DSR restore the weight of an informative but insufficiently optimized modality without retaining a high weight for a corrupted modality?
  3. How is the modality-wise expected calibration error in Table 3 computed from the Gaussian-mixture predictions, and what empirical evidence connects multimodality in the feature visualizations with multimodality of the conditional sentiment distribution in Equation (5)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.