acceptodds
Under review as a conference paper at ICLR 2027

Fusing Evidence, Learning Across Views: Multimodal Sentiment Analysis with Incomplete Data

Abstract

Multimodal sentiment analysis (MSA) is vulnerable to fine-grained missingness, where different corruption patterns retain different subsets of evidence from the same underlying sample. We propose Cross-view Evidence Learning and Alignment (CELA), a same-sample multi-corruption framework that aggregates evidence within each incomplete view and coordinates representations across independently corrupted views. CELA integrates modality-specific and common evidence through slot-based fusion, anchors incomplete-view representations to a task-supervised full-view representation, and explicitly promotes consistency among corrupted-view representations of the same utterance. At inference, CELA requires only a single incomplete observation, without explicitly reconstructing missing content. Experiments on MOSI, MOSEI, and SIMS show that CELA achieves the best missing-rate-averaged performance among the evaluated methods, with clear advantages under low-to-moderate missingness. These advantages also persist under fixed- and mixed-missing-rate training protocols, CIM-oriented nonidealities, and component-level physical ReRAM-based CIM execution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Analysis, Incomplete Multimodal Learning, Cross-view Consistency, Evidence Fusion, Computing-in-Memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_omgNOpenReview ↗
Rating
1(−1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Cross-view Evidence Learning and Alignment (CELA) for multimodal sentiment analysis with incomplete observations. The framework combines Common Evidence Slot Fusion (CESF), full-view task supervision and representation alignment (FSA), and consistency between incomplete views (CMC). Inference requires only a single incomplete observation. Experiments on MOSI, MOSEI, and SIMS report improvements over nine evaluated baselines when averaged across matched training and testing missing rates. Additional experiments investigate alternative training protocols, structured missingness, component ablations, and the number of corrupted views.

Critical strengths
  1. The training–inference separation is practical and clearly defined. CELA exploits complete observations and additional corrupted views during training while requiring only one incomplete view at inference. This avoids the need for multiple observations or explicit reconstruction at deployment.
  2. The main improvements extend beyond the proposed aggregate metric. CELA improves rate-averaged BGSS over the strongest evaluated baseline by 0.0255 on MOSI, 0.0149 on MOSEI, and 0.0206 on SIMS.
Critical weaknesses
  1. The related-work discussion does not sufficiently address relevant studies on multimodal understanding with uncertain or missing modalities, including Yang et al. [1] and Chi et al. [2]. The authors should discuss these works and clarify how CELA differs from them in problem setting and training methodology.
  2. The distinct contribution of the proposed mechanisms remains insufficiently isolated. The framework combines shared projections, learned-slot cross-attention, supervised complete-view training, and representation consistency. The paper’s own Proposition 2 shows that FSA alignment already includes a dispersion term, while CMC increases its relative weight. These observations do not invalidate the method, but they make it important to establish what CELA contributes beyond a well-controlled combination of conventional fusion and consistency regularization.
  3. The deployment-oriented evaluation is narrower than the main benchmark coverage. The primary results average over separately trained models whose training missing rate matches the testing missing rate. The paper appropriately adds single-checkpoint fixed-rate and mixed-rate evaluations, but these are restricted to MOSI and four selected baselines. In particular, Self-MM (the strongest rate-averaged MOSI baseline in Table 1) is not included in these comparisons.

[1] Qu Yang, Qinghongya Shi, Tongxin Wang, and Mang Ye. Uncertain Multimodal Intention and Emotion Understanding in the Wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24700–24709, 2025. [2] Haozhe Chi, Minghua Yang, Junhao Zhu, Guanhong Wang, and Gaoang Wang. Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 121–130, 2022.

Questions for authors
  1. Can you disentangle architecture and objective contributions?
  2. What explains the discrepancies between Tables 4/18 and Tables 16/17?
  3. How much improvement remains over a comparably trained text-only model?
  4. Can you clarify the hardware measurement and robustness protocol?
  5. Do the single-checkpoint robustness gains generalize beyond MOSI?
2 comments
AuthorsOpenReview ↗

Related Work, Controlled Experiments, and Robustness Evaluation

W1: We sincerely thank the reviewer for highlighting Yang et al. (CVPR 2025) and Chi et al. (AACL 2022). We agree they clarify CELA's positioning. Yang et al. study uncertain modality availability and joint emotion-intention understanding in social media; CELA targets fine-grained missingness robustness via evidence fusion and cross-view learning. M3S combines random missingness augmentation with meta-learning. CELA instead coordinates same-sample complete and independently corrupted views via FSA, CMC, and CESF. We appreciate this guidance and will add both to Related Work.

W2: We sincerely thank the reviewer for examining each mechanism's role. CESF's common-evidence branch constructs common/modality-specific evidence for co-designed slot-based fusion. Removing either alters their interaction; w/o CESF tests the joint design. We acknowledge uncertainty in Table 3 (10 MRs × 3 seeds). Appendix Figure 7 shows CESF mainly helps at low-to-moderate MRs, while CMC has larger gains at some moderate-to-high MRs. Proposition 2 shows FSA-align already constrains incomplete-view dispersion; separately weighted CMC adjusts this without stronger full-view anchoring. Table 3, Figure 7, and missing-view tests support overall robustness.

W3: Please see Q4 and Q5.

Q1: We sincerely thank the reviewer for requesting clearer architecture/objective separation. Unlike Concat+MLP, CESF uses Learned-slot Cross-attention to flexibly integrate retained common and modality-specific evidence. Appendix A.6 Tables 16–17 compare CELA with conventional fusion under identical full-view supervision, two corrupted views, optimization, and Test masks; Fusion+Prediction parameters differ by just 0.0502%. Results: Anonymous repository – Anonymous GitHub. CELA gains 0.0238/0.0250 BGSS at MR=0.5/0.9, beyond extra views, supervision, or parameter count. FSA offers sentiment guidance through full-view supervision/alignment; CMC constrains corrupted-view differences. Ablations and view-count tests provide further evidence.

Q2: We sincerely thank the reviewer for identifying discrepancies between Tables 4/18 and 16/17. We agree that clarification is essential for reproducibility. Different training runs, Validation-selected checkpoints, and Test-mask protocols explain the numerical differences. In the revision, we will re-evaluate shared configurations using checkpoints from the same runs and identical Test masks, then update the results accordingly.

Q3: We sincerely thank the reviewer for asking about gains over a strong Text-only model. Under Matched-MR on MOSI: Anonymous repository – Anonymous GitHub. CELA gains slightly at moderate-to-high MRs but trails Text-only at low MR; three-rate averages nearly match. As missingness rises, text's relative advantage may weaken, allowing retained audio/visual cues to help. Under Mixed-MR, one model spans MRs: Anonymous repository – Anonymous GitHub. CELA gains on average, especially at MR=0.5, but nearly ties at high MR. Thus, gains depend on MR and protocol and may be clearer with one model across MRs.

Q4: We sincerely thank the reviewer for raising concerns about hardware scope and robustness. HE-100 has 576×128 signed logical weights per array, consistent with physical-chip array scales reported by Wang et al. (Nature Electronics, 2025) and Yu et al. (Nature Communications, 2025). This scale balances capacity, analog precision, and feasibility: larger arrays may increase voltage drops and readout costs, offsetting energy gains. We mapped the prediction head's two linear layers onto three hardware tiles and assessed variability through three independent programming cycles. Conductance drift up to one year was simulated, not physically measured. Details: Appendix A.9 and Anonymous repository – Anonymous GitHub.

We chose Matched-MR for controlled evaluation, following TFR-Net and related work with matched train/test MRs; MCT-HFR (ACM TOMM 2024) calls this one-to-one training. It enables comparisons at matched missingness levels. Section 4.6 adds Fixed/Mixed-MR to assess train-test MR mismatch; CELA leads in mean BGSS under both.

Q5: We sincerely thank the reviewer for noting that Section 4.6 covered only MOSI and omitted strong Self-MM. We added SIMS Fixed/Mixed-MR experiments and Self-MM in both protocols.

MOSI results: Anonymous repository – Anonymous GitHub.

SIMS results: Anonymous repository – Anonymous GitHub.

CELA leads in cross-MR mean BGSS on both datasets and protocols, supporting robustness to train-test MR mismatch.

We noticed that you lowered your rating to 1 (confidence 4) without providing any explanation, despite our detailed rebuttal. Could you please clarify which concerns remain unresolved and what specifically justifies this substantial change in your assessment?

Without such feedback, we have no meaningful opportunity to address your concerns.

Reviewer_SFLGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CELA , a framework for multimodal sentiment analysis under fine-grained incomplete data , during training, generates multiple independently corrupted views of the same utterance and couples three objectives within-view evidence fusion via slot-based cross-attention plus common-token agreement, anchoring of each incomplete view to a task-supervised full view, and consistency among corrupted views while inference requires only a single incomplete observation with no reconstruction. On MOSI, MOSEI, and SIMS, CELA achieves the best missing-rate-averaged performance among nine baselines with gains concentrated in low-to-moderate missingness and persisting under fixed/mixed-missing-rate training and physical ReRAM-based compute-in-memory execution, showing that learning across multiple corrupted views is an effective, inference-cost-free training principle for incomplete MSA.

Critical strengths
  1. The method incurs no additional inference cost. At inference it only needs a single incomplete observation, with no reconstruction, no full view, and no extra corrupted views.

  2. The experimental reporting is rigorous and detailed, and it also faithfully reports unfavorable results.

Critical weaknesses
  1. Some baselines report results substantially lower than those in their original papers (e.g., LNLN). Can the authors explain this discrepancy, and confirm whether the baselines were reproduced using the original papers' settings and code?

  2. The methodological novelty is limited. Multi-perturbation view training and MISA-style shared/private decomposition have already been adopted by prior works, so the authors need to further clarify their own contribution. The results of removing CMC and removing CESF are almost within the standard deviation of the full model, making it difficult to demonstrate the contribution of these components.

Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Baseline Reproducibility, Methodological Novelty, and Ablation Evidence

W1: Baseline Reproduction and Experimental Fairness

We sincerely thank the reviewer for carefully examining the quality of our baseline reproductions and the fairness of our experimental comparisons. We appreciate the opportunity to clarify these important concerns.

First, Appendix Table 10 lists the implementation sources and key model configurations of all nine baselines. All nine baselines were reproduced or adapted from publicly available implementations. For P-RMF, we additionally applied numerical-stability fixes to the available code, guided by the original paper, while preserving its core methodology. We would like to clarify that our main experiments did not strictly retain every training hyperparameter from the original papers. Instead, we standardized certain training and evaluation settings to ensure a consistent experimental protocol, as detailed in Appendix A.4.

Second, we observed that some publicly released implementations select checkpoints using the Test set or different evaluation metrics. To improve fairness, we consistently evaluated each method with three random seeds and selected checkpoints according to Validation BGSS. Test evaluation was conducted only after the model configuration and weights had been finalized, preventing Test information from influencing model selection.

To further address the reviewer's concern, we conducted additional comparisons using the original methods' publicly available configurations while maintaining a unified checkpoint-selection criterion based on Validation BGSS. The results are available at Anonymous repository – Anonymous GitHub.

After reevaluation, CELA still achieves the highest average BGSS on both MOSI and SIMS, although the performance gaps with some baselines become smaller. We acknowledge that this demonstrates the substantial influence of baseline training configurations and further underscores the importance of fair comparisons. We would also like to note that the original baseline configurations were typically optimized specifically for their respective architectures, whereas CELA has not undergone a hyperparameter search of comparable scale. Therefore, these results are more appropriately interpreted as comparisons of methods under their respective configurations; small performance gaps should not, by themselves, be taken to imply an absence of methodological differences.

W2: Methodological Novelty and Ablation Evidence

We sincerely thank the reviewer for the careful assessment of CELA's methodological novelty and the contributions of its components. We appreciate these observations and would like to respond from two perspectives.

(1) Methodological novelty. We acknowledge that multi-perturbation view training and shared/private representation modeling have already been explored in prior work. CELA does not claim these basic operations as its novelty. Rather, its contribution lies in jointly designing within-view evidence fusion and cross-view collaborative learning for fine-grained missingness. Specifically, while MISA explicitly decomposes modality representations into invariant and specific subspaces and fuses them through self-attention, CESF retains modality-specific evidence, constructs common evidence using a shared availability-conditioned projector, and integrates both through Learned-slot Cross-attention to support fusion under incomplete inputs. CELA also explicitly distinguishes full-view semantic anchoring (FSA) from consistency among incomplete views (CMC): the former uses the complete view to provide task supervision and a representational reference, whereas the latter further regulates dispersion across different incomplete-view representations. Together with CESF, these mechanisms form a unified framework for fine-grained missingness.

(2) CESF/CMC ablations and statistical uncertainty. We sincerely appreciate the reviewer's careful examination of Table 3. The table reports ablation results averaged across 10 missing rates, with three random seeds for each rate. Removing CESF or CMC decreases mean BGSS by 0.0112 and 0.0084, respectively, accompanied by an increase in MAE and a decrease in the correlation between predictions and ground-truth labels. These results indicate positive average contributions from both components under the evaluated conditions.

Furthermore, Appendix Figure 7 presents the BGSS differences between the full model and its ablated variants at individual missing rates. CESF mainly contributes under low-to-moderate missingness, while CMC, as a separately adjustable cross-view regularization objective, shows more pronounced average gains under some moderate-to-high missing rates. Taken together, these results show that both components improve overall average performance, with larger benefits under certain missingness conditions, supporting their inclusion in the complete framework.

Reviewer_zXwzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CELA to address fine-grained missingness of language tokens and audio/visual frames in MSA. Specifically, CESF integrates modality-specific and common evidence within each incomplete observation. FSA uses a task-supervised full view to guide incomplete-view representation learning, while CMC reduces representation differences between independently corrupted views of the same utterance. Experiments on MOSI, MOSEI, and CH-SIMS demonstrate the effectiveness of the proposed method. Additional missingness protocols experiments further evaluate its robustness.

Critical strengths
  1. This paper evaluates CELA across multiple missing rates on MOSI, MOSEI, and CH-SIMS, with improvements across classification and regression metrics. Additional MOSI experiments cover fixed and mixed missing rate training and structured missingness. These experiments provide complementary evidence of the CELA's effectiveness under different conditions.
  2. This paper provides substantial details about the architecture, data processing, training, and evaluation procedures.
Critical weaknesses
  1. The distinct contribution of the core designs and the reasons for the performance improvements remain insufficiently established. Complete-to-incomplete representation alignment and same-sample cross-view learning have already been studied. Proposition 2 also shows that FSA-align already penalizes dispersion among incomplete-view representations, while CMC further increases this penalty. The current ablation therefore shows that CMC improves performance under the chosen configuration, but does not establish whether it provides a consistent additional benefit over a sufficiently tuned FSA-only alternative. The fusion analysis has a similar limitation. Removing CESF simultaneously replaces the fusion architecture, removes the common-evidence branch, and disables common-token agreement, making it difficult to identify which design choices contribute to the gains. Increasing the number of corrupted views from one to two also enables CMC.

  2. Comparisons with missing-data methods remain incomplete, and their adaptation and training settings need further clarification. A substantial portion of the main comparison, including MISA, Self-MM, MMIM, and TETFN, consists of methods primarily developed for MSA with complete inputs. These are useful references, but outperforming them is not sufficient to establish an advantage over existing missing-data approaches. The inclusion of LNLN, P-RMF, and TF-Mamba is valuable, but directly relevant methods are absent, as are distillation-based missing-modality methods such as UMDF and CorrKD. The latter also use complete-data information to guide incomplete representation learning and are particularly relevant to the paper’s central approach.

Questions for authors
  1. What additional benefit does CMC provide beyond the existing alignment constraint? If the FSA-only model is allowed to tune its alignment weight on the validation set, does adding CMC still provide a consistent improvement?

  2. Which components account for the performance gains? Are there results that separately examine the common-evidence branch, slot-based fusion, and cross-view consistency? Beyond aggregate performance, is there evidence that the model makes better use of retained sentiment information, rather than simply making representations from different views more similar?

  3. How is a fair comparison ensured under the adopted missingness protocol? Please clarify how each baseline is adapted to fine-grained missingness, whether it has access to full-view supervision during training, and how the reimplementation and parameter configurations are validated. What adjustments are made for methods originally developed for complete inputs or whole-modality absence?

1 comment
AuthorsOpenReview ↗

Clarifications on Baseline Fairness and Component Contributions

W1: Please see Q1 and Q2.

W2 & Q3: We sincerely thank the reviewer for the questions on baseline coverage, fine-grained adaptation, and fairness. We agree these details are essential for reliable comparisons.

(1) Fine-grained adaptation. Appendix A.4 specifies shared data splits, features, and metrics. Valid text tokens are replaced by [UNK] and audio/visual frames are zeroed; special tokens and padding are excluded. We retain core architectures/objectives when adapting complete-input methods and preserve missing-data mechanisms, e.g., LNLN's completeness labels and TFR-Net's visibility masks.

(2) Full-view supervision. We acknowledge different uses of complete-view information: some baselines employ reconstruction/auxiliary objectives; CELA uses FSA task supervision and alignment. The main comparisons use unified evaluation, not identical supervision. Appendix Tables 16–17 control for this: CELA and a conventional fusion baseline share full-view supervision, two corrupted views, optimization and evaluation settings, with Fusion+Prediction parameters differing by only 0.0502%. CELA gains 0.0238/0.0250 BGSS at MR=0.5/0.9, supporting the full design.

(3) Implementation and settings. Appendix Table 10 lists sources/settings for nine baselines. Except P-RMF, whose original open-source implementation has numerical risks, the others were reproduced from open-source implementations. We standardized some training settings instead of retaining all original hyperparameters, while retaining core architectures and objectives (Appendix A.4).

(4) Model selection and fairness. Some public robust-MSA implementations use Test for checkpoint selection or select by different metrics. We therefore use three seeds, select checkpoints by Validation BGSS, and evaluate on Test only after fixing configurations and weights. We distinguish Matched-MR training from Fixed/Mixed-MR single-checkpoint evaluation.

(5) UMDF and CorrKD. We thank the reviewer for highlighting these related works and recognize their shared aims with CELA. UMDF uses MMD for heterogeneous-view feature-distribution matching and JS divergence/MSE for bidirectional prediction self-distillation. CMC instead constrains same-sample fused representations across independently corrupted views via symmetric stop-gradient distance, without distribution matching or prediction distillation. CorrKD uses a separately pretrained full-modality teacher for cross-sample, category-prototype, and response distillation. CELA jointly optimizes complete and incomplete views with shared parameters, without a separate pretrained teacher. Without verifiable official open-source implementations, we could not ensure faithful comparisons. We will discuss both in Related Work and note this limitation.

Q1: We sincerely appreciate the reviewer's insightful analysis of FSA-align and CMC. We agree FSA-align already constrains incomplete-view dispersion, so CMC's incremental benefit warrants testing. We tuned FSA-only's alignment weight on Validation to λfa=0.05. Keeping other settings fixed, we compared with/without CMC on MOSEI at MR=0.1, 0.5, 0.9. Results: Anonymous repository – Anonymous GitHub. CMC yields positive additional gains, consistent with constraining corrupted-view differences without stronger full-view anchoring.

Q2: We sincerely thank the reviewer for asking about component contributions and retained sentiment information.

(1) Performance contributions. Table 3 shows that removing CESF, CMC, FSA-align, and FSA reduces mean BGSS by 0.0112, 0.0084, 0.0152, and 0.0374, respectively, while MAE and Corr also worsen, indicating positive average contributions.

(2) CESF subcomponents. The common-evidence branch and Slot-based Fusion in CESF are jointly designed: the former constructs common and modality-specific evidence, while the latter integrates them. Removing either alters their interaction; thus, w/o CESF tests the joint fusion design. Table 3 averages 10 missing rates with three seeds each. Removing CESF or disabling CMC reduces mean BGSS by 0.0112 and 0.0084, respectively. Appendix Figure 7 shows CESF mainly benefits low-to-moderate missingness, while CMC's contribution varies by MR, with larger gains at some moderate-to-high rates, such as MR=0.6 and 0.8. Missing-view tests further support overall robustness.

(3) Retained sentiment. We fully agree that consistency and aggregate gains alone cannot prove better use of specific sentiment cues. Appendix A.8 tests whole-modality removal, asymmetric missingness, and temporal missingness patterns, providing indirect evidence of using remaining information. Like TFR-Net, LNLN, P-RMF, and TF-Mamba, which assess prediction, ablations, and robustness under missingness, CELA targets robust MSA with incomplete inputs rather than fine-grained sentiment evidence attribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.