acceptodds
Withdrawn from ICLR 2027

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Sentiment Analysis (MSA) aims to infer human emotions from multiple modalities such as text, audio, and vision. In practice, inputs are often corrupted by noise and missing modalities, which degrades performance. Existing methods typically address these challenges in isolation, limiting their effectiveness in realistic settings. To address this limitation, we propose a Reliability-aware Cross-sample Enhancement (RCE) framework. Specifically, RCE first introduces an adaptive variational information bottleneck to model modality-wise uncertainty and perform quality-aware information compression, thereby suppressing redundant noise in unreliable modalities. Furthermore, we design a reliability-aware cross-sample enhancement strategy that retrieves high-confidence, semantically consistent neighbors from a large candidate pool to enrich and calibrate current representations, effectively alleviating information deficiency caused by missing modalities. Building upon this, RCE integrates cross-modal interactions with a multilevel reliability-aware fusion mechanism to adaptively aggregate information across modalities and enhancement stages, leading to more robust multimodal representations. Extensive experiments demonstrate that RCE consistently outperforms state-of-the-art methods across full, noisy, and missing-modality settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Representation Learning, Uncertainty-aware, Cross-sample Knowledge Enhancement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BJLEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Reliability-aware Cross-sample Enhancement (RCE) for multimodal sentiment analysis under complete, noisy, and missing-modality conditions. RCE combines a vMF-based variational information bottleneck for modality-wise uncertainty estimation, a memory bank for reliability- and label-aware cross-sample enhancement during training, and multilevel fusion of original and enhanced representations. The authors evaluate the framework on sentiment analysis, humor detection, and sarcasm detection benchmarks, report competitive performance under complete and individual degradation settings, and provide ablation and additional analyses in the appendix.

Critical strengths
  1. A practically relevant and coherent framework. Modality corruption and missingness are important obstacles to multimodal sentiment analysis. Connecting modality-wise uncertainty estimation, cross-sample enhancement, and adaptive fusion provides a plausible approach to learning useful representations under imperfect observations. This practical motivation is a substantive strength of the work.

  2. Promising empirical performance with useful supporting analyses. The reported comparisons cover multiple benchmarks and both complete and individual degradation settings. Component ablations and additional stability and efficiency analyses help assess the framework's practical benefits. These results make the approach worth considering, even though the central robustness and mechanism claims require stronger validation.

Critical weaknesses
  1. The correctness and reproducibility of the central mechanism are not sufficiently established. In Equations (10) and (11), the key/value representations appear to be individual pooled vectors. Under standard token-wise attention, a single key produces an attention weight of one, preventing the query from influencing the output through the attention weights. This raises a substantive question about the claimed prompt-dependent and cross-modal interactions. The vMF sampling procedure and the inference pathway after disabling RME are also insufficiently specified. These issues are consequential because the proposed interactions and reliability estimates are central to the paper's contribution. The implementation clarifications in Questions 1-3 could materially change my assessment.

  2. The evidence does not sufficiently establish the claimed contribution beyond closely related methods or robustness under simultaneous missingness and corruption. HME already combines cross-sample enhancement with uncertainty-aware fusion, and PML uses vMF concentration for reliability estimation and fusion. Targeted comparisons against HME with a memory bank and label-aware weighting, or a Gaussian VIB with a matched architecture, would help isolate RCE's contribution. TMDC directly addresses missing and noisy modalities and is cited in the manuscript, but no comparison or explanation of incompatible assumptions is provided. Moreover, the central motivation concerns coexistence of missingness and noise, whereas the reported evaluations primarily examine them separately. The fixed-missing evaluations also cover only combinations containing text. These gaps limit the evidence for the specific novelty and robustness claims that determine my recommendation.

Questions for authors

The following are clarification requests about the current method and reported results, rather than suggestions for future work.

  1. Attention implementation: What are the exact query, key, and value tensor shapes in Equations (10) and (11)? If the key/value input is a single pooled token, how does the query affect the output? Please specify any omitted residual connections or retained token sequences and clarify which operations implement the claimed cross-modal interactions.

  2. vMF sampling and concentration: What is the distribution and sampling procedure for the scalar ww in Equation (4), and how are its gradients computed? Does the sampler produce the same posterior distribution whose KL is evaluated in Equation (5)? Table 5 uses d=256d=256 for CMU-MOSI, while Table 9 reports mean concentrations around 0.7-1.6. For illustration, a genuine vMF distribution with d=256d=256 and κ=1.573\kappa=1.573 has mean directional alignment approximately 0.00614 and KL to the uniform distribution approximately 0.00483 nats. Is any additional concentration scaling used before sampling, and how should these values be interpreted in the implemented model?

  3. Inference and missing modalities: When RME is disabled at inference, what replaces the neighbor-enhanced representation used during training? How are unavailable modalities handled in the token construction, shared representation, auxiliary losses, and multilevel fusion? Please specify the relevant masking or substitution rules.

  4. Reported F1 metric: Equation (31) and its TP/FP/FN definitions describe ordinary binary positive-class F1, although Appendix D.5 calls the metric weighted F1. Which averaging convention is actually used for the table results? Please confirm whether the implementation computes the support-weighted mean of class-specific F1 scores and correct the mathematical definition accordingly.

Reviewer_EgE7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Reliability-aware Cross-sample Enhancement (RCE) for robust multimodal sentiment analysis. It combines a von Mises–Fisher-based variational information bottleneck for modality uncertainty estimation, memory-based cross-sample enhancement, and hyper-modality generation with reliability-aware multilevel fusion. Experiments on CMU-MOSI, CMU-MOSEI, UR-FUNNY, and MUStARD report competitive performance under full, noisy, and missing-modality settings, supported by ablation studies and supplementary analyses.

Critical strengths
  1. The paper addresses a practically relevant problem by considering modality uncertainty, missing information, and noisy observations within a unified multimodal sentiment analysis framework. The combination of uncertainty-aware compression, cross-sample enhancement, and adaptive fusion is reasonably motivated.

  2. The evaluation covers multiple benchmarks and degradation settings, with generally competitive results. The paper is easy to follow.

Critical weaknesses
  1. In Eq. (10), the modality-enhanced representation is a single vector serving as the key and value for cross-attention. Under the stated formulation, the softmax attention weights are constant when only one key is available, making the attention output independent of the query. A similar issue arises in Eq. (11), where cross-modal interactions appear to operate on single modality-level vectors. It is therefore unclear whether these operations perform meaningful query-dependent attention or primarily act as learned feature transformations. Although Table 2 demonstrates the contribution of the overall hyper-modality module, it does not isolate the effectiveness of the proposed attention operations. Clarifying the actual token structure and comparing against simpler projection-based alternatives would help substantiate the claimed cross-modal interaction mechanism.

  2. Relevant methods such as P-RMF (ACL 2025) [1], TF-Mamba (EMNLP 2025) [2], TMDC (AAAI 2026) [3], and SMCIR (AAAI 2026) [4] are absent from the experimental comparisons. These methods directly address incomplete modalities, modality uncertainty, or quality degradation, making them relevant to assessing RCE's contribution. Furthermore, although the paper emphasizes robustness when noise and missing modalities coexist, the evaluations primarily consider these conditions separately, without directly testing simultaneous corruption and missingness. The ablations also provide limited evidence of how the reliability-related components behave under increasing degradation severity. Comparisons under matched protocols and targeted robustness ablations would better establish the claimed advantages of the unified framework.

Refs:

[1] Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data (Zhu et al., ACL 2025)

[2] TF-Mamba: Text-enhanced Fusion Mamba with Missing Modalities for Robust Multimodal Sentiment Analysis (Li et al., EMNLP Findings 2025)

[3] TMDC: A Two-Stage Modality Denoising and Complementation Framework for Multimodal Sentiment Analysis with Missing and Noisy Modalities (Zhuang et al., AAAI 2026)

[4] Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations (Chen et al., AAAI 2026)

Questions for authors
  1. In Eqs. (10)–(11), do the attention operations use only the single modality-level vectors described in the paper, or are additional token sequences involved in the implementation? If only one key-value token is used, how do these operations achieve query-dependent information selection?

  2. Section 3.3 interprets the vMF concentration parameter κm\kappa_m as modality reliability and uses it to guide neighbor weighting and multilevel fusion. However, posterior concentration does not necessarily correspond to input quality or prediction correctness. Appendix E.3 mainly examines correlations between average concentration and aggregate performance across degradation levels, which does not establish whether κm\kappa_m can distinguish reliable from unreliable samples under the same corruption condition. Moreover, the reliability-aware fusion in Eqs. (13)–(14) modulates learned attention logits using κm\kappa_m, but the resulting weights are not directly validated against actual modality quality. While the existing ablations demonstrate downstream performance effects, they do not sufficiently establish that the proposed reliability estimates and fusion weights behave as intended. Does the concentration parameter κm\kappa_m correlate with sample-level prediction correctness or actual modality quality under a fixed corruption condition? Is there evidence that the resulting reliability-aware fusion weights appropriately reflect differences in modality quality?

  3. Do the existing analyses show how the contributions of AVIB, confidence-based neighbor weighting, and reliability-aware fusion change with increasing noise or missingness? In particular, do these components provide larger benefits when modality reliability deteriorates?

  4. Please expand the baseline comparison to include recent relevant methods such as P-RMF [1], TF-Mamba [2], TMDC [3], and SMCIR [4], where evaluation protocols are comparable. Evaluating simultaneous noise and modality missingness would also help substantiate the claimed robustness of the unified framework.

My main concern is the insufficient coverage of relevant recent baselines. I would be happy to raise my score if the authors can adequately address the concerns raised above, particularly by providing more comprehensive and fair comparisons with closely related methods.

Refs:

[1] Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data (Zhu et al., ACL 2025)

[2] TF-Mamba: Text-enhanced Fusion Mamba with Missing Modalities for Robust Multimodal Sentiment Analysis (Li et al., EMNLP Findings 2025)

[3] TMDC: A Two-Stage Modality Denoising and Complementation Framework for Multimodal Sentiment Analysis with Missing and Noisy Modalities (Zhuang et al., AAAI 2026)

[4] Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations (Chen et al., AAAI 2026)

Reviewer_zbuyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RCE for robust multimodal sentiment analysis under full, noisy, and missing-modality settings. It combines (1) an adaptive variational information bottleneck with von Mises-Fisher (vMF) distributions for modality-wise uncertainty, (2) reliability-aware cross-sample enhancement using a memory bank during training, and (3) hyper-modality generation with reliability-aware multilevel fusion. Experiments cover CMU-MOSI and CMU-MOSEI under full, missing, and noisy conditions, plus UR-FUNNY, MUStARD, and CH-SIMS v2, with ablations and further analyses in the appendix.

Critical strengths
  1. RCE is state-of-the-art or competitive on CMU-MOSI and CMU-MOSEI under full-modality and most missing and noisy settings (Tables 1, 6-8).

  2. The paper includes ablations, sensitivity analysis, validation of kappa as a reliability proxy, retrieval-strategy comparisons, and five-seed stability results.

  3. The reservoir bank outperforms intra-batch and EMA-prototype retrieval (Table 16) and reduces negative enhancement relative to HME (Tables 3, 12, 13), although this reduction is partly by design since neighbor weights use ground-truth labels (Eq. 8).

  4. The memory bank adds little memory and training time, and is not used at inference (Tables 17-18), although RCE has the largest parameter count.

Critical weaknesses
  1. RCE largely combines existing ideas: vMF reliability modeling (PML, IJCAI 2025), cross-sample enhancement with uncertainty-aware fusion (HME, NeurIPS 2025), and hyper-modality representations (ALMT). The paper does not clearly establish a new methodological insight beyond their integration.

  2. The paper is motivated by co-occurring noise and missingness, but evaluates them only separately (Appendices E.1, E.2). In addition, the fixed-missing protocol always keeps text, and all ablations use full modalities, so the contributions of AVIB and RME under degraded inputs are not isolated.

  3. TMDC (AAAI 2026), which targets both missing and noisy modalities, is cited but not compared. RCE is also not consistently best: at Salt-pepper noise 5.0 on CMU-MOSEI it has the lowest Acc2, F1, and Corr among all methods (Table 8), and at Gaussian noise 5.0 on CMU-MOSI several baselines achieve higher Acc7 (Table 7).

  4. In Eqs. 10 and 11, the key/value is a single vector. Softmax over one key always gives weight 1, so the output is independent of the query and reduces to a linear projection. The claimed cross-modal interaction does not occur in these steps.

  5. Reliability and retrieval concerns: (a) Reported mean kappa values are about 0.6-1.6 with d = 256 (Table 9); a vMF at this concentration is nearly uniform, so the sampled z may carry little information. (b) kappa_T drops the most under noise that is stated to affect only audio and vision, which questions whether kappa reflects per-modality reliability. (c) The kappa validation relies on only 4-5 aggregate points. (d) Eq. 6 does not exclude sample i from memory-bank candidates, so with N_b = 1024 and about 1.3k CMU-MOSI training samples, self-retrieval is likely. (e) RME uses ground-truth labels and is training-only, so it is unclear how it helps missing modalities at test time.

Questions for authors
  1. Can you report results with noise and missingness applied jointly, including text-absent settings, and ablate AVIB and RME under degraded inputs?
  2. Why was TMDC excluded, and can you compare against it under matched protocols?
  3. Do Eqs. 10-11 actually attend over multiple tokens? If not, what do they provide beyond a linear projection?
  4. How is w sampled in Eq. 4, how informative is z at the reported kappa values, and was text noise applied in Tables 7-9?
  5. Is sample i excluded from its own memory-bank candidates, and what representation replaces the RME output at inference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.