acceptodds
Under review as a conference paper at ICLR 2027

Beyond Summaries: Learning to Read Local Evidence for Incomplete Multimodal Sentiment Analysis

11 views

Abstract

Incomplete multimodal sentiment analysis predicts sentiment from partially observed language, audio and visual inputs. Existing approaches for incomplete multimodal sentiment analysis predominantly rely on global representations, yet overlook the heterogeneous performance degradation induced by the loss of distinct local regions. Even under identical missing ratios, the absence of decisive local cues may completely flip the predicted sentiment polarity. To mitigate such drawbacks, this paper presents DREAM, a damage-guided framework for discovering crossmodal local evidence. Through contrastive analysis of prediction risks triggered by paired regional erasures, DREAM quantifies the task-level relevance of multi-scale local evidence while preserving global summary information. Equipped with a summary-referenced reader module, each modality adaptively balances a source modality’s global summary with its complementary local evidence, conditioned on the receiving modality’s available content. Extensive evaluations on MOSI, MOSEI and CH-SIMS demonstrate consistent improvements in both sentiment classification and regression. Performance gains are especially substantial on CH-SIMS, where missing fragments exert highly uneven impacts on sentiment polarity. Without full-view teachers or supervision from fully observed counterparts, DREAM improves binary classification accuracy and F1 over MIDAS by 4.11 and 5.05 percentage points, respectively, while reducing MAE by 12.72%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Analysis, Incomplete Multimodal Learning, Damage-Guided Learning, Multiscale Local Evidence, Cross-Modal Evidence Selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_d39YOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DREAM, a framework for incomplete multimodal sentiment analysis that addresses the limitation of existing methods, which primarily rely on global representations and struggle to distinguish the predictive damage caused by the absence of different local information. The core idea is to compare changes in prediction risk after removing the same amount of content from different locations, thereby learning the task relevance of local evidence at multiple scales. In addition, the paper introduces a cross-modal reading module that uses global summaries as a reference, allowing the model to select between global summaries and local evidence based on the specific content of the receiving modality. Experiments on three benchmarks, MOSI, MOSEI, and CH-SIMS, demonstrate that the proposed method achieves promising performance under both random and structured missingness. Overall, the paper combines learning the importance of local information with cross-modal evidence utilization, offering a valuable approach to incomplete multimodal sentiment analysis.

Critical strengths
  1. The research problem is practically meaningful. The paper considers not only the amount of missing modality information but also how the specific missing content affects prediction outcomes differently. This provides a reasonable research motivation and helps advance incomplete multimodal learning beyond modality-level reliability estimation toward the identification and utilization of local evidence.

  2. The experimental evaluation is relatively comprehensive. The paper evaluates classification and regression performance on three datasets and further investigates scenarios involving different missing rates, contiguous segment removal, and entire-modality missingness. The experiments also include multiple component ablations and qualitative examples, helping analyze the sources of performance improvements.

  3. The experimental results are reasonably convincing. According to the reported results, DREAM achieves strong average performance across all three datasets. On CH-SIMS, compared with MIDAS, it improves binary classification accuracy and F1 by 4.11 and 5.05 percentage points, respectively, while reducing MAE by 12.72%. The ablation studies also show that removing the region-ranking supervision, damage-weighted allocation, or local evidence reading component negatively affects model performance.

Critical weaknesses
  1. Combinatorial novelty. The paper proposes DREAM, whose core contribution is to learn the task relevance of multi-scale local evidence through paired deletion under an equal budget, and to combine this with cross-modal compatibility for receiver-modality-conditioned local evidence reading. However, its core components are related to existing ideas in deletion-based perturbation importance assessment, multi-scale evidence modeling, and adaptive cross-modal fusion. Thus, the overall novelty lies more in combining existing techniques than in introducing an entirely new methodological paradigm.

  2. The independent contribution of the core method needs further clarification. The paper uses prediction damage caused by deleting local regions to learn evidence value, but this supervision signal itself depends on model prediction risk and ground-truth labels. Although the ablation studies validate the effectiveness of the relevant components, the current experiments do not fully clarify how much of the performance gain comes from damage-guided local evidence learning, as opposed to additional training supervision and architectural design.

  3. Matching deletion lengths may not fully control for differences between regions. The paper controls the amount of content removed from different regions by matching deletion lengths, but regions of equal length may differ in semantic completeness, temporal span, and information density. For example, deleting a complete emotional phrase may naturally have a different impact from deleting the same number of ordinary words. Therefore, although the experiments demonstrate that deletion at different positions has different effects, further analysis is needed to distinguish whether the gains arise from learning local task relevance or from distributional differences introduced by the particular deletion strategy.

Questions for authors
  1. How would the performance change if other local importance estimation methods were used in place of deletion-damage-based region-ranking supervision, such as gradient-based or perturbation-based importance scores? This would help further establish the independent contribution of the core design.

  2. Compared with major baselines such as MIDAS, how much additional training time, computational cost, and inference latency does DREAM incur? Does computing deletion-induced damage require multiple additional forward passes for each training sample?

  3. Have the authors conducted statistical significance tests for metrics with relatively small improvements? Are the performance advantages observed across three random seeds statistically reliable?

Reviewer_7mMVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses incomplete multimodal sentiment analysis by highlighting that missing different local regions can cause highly uneven performance degradation, which global-representation methods often overlook. To address this issue, the authors propose DREAM, a damage-guided framework for cross-modal local evidence learning. The paper presents a coherent pipeline from motivation and method design to experimental validation, with evaluations across multiple datasets, missing scenarios, and metrics. The appendix also provides strong support for reproducibility.

Critical strengths

The paper has several strengths. Its main contribution is a fresh perspective that emphasizes task-value differences across local regions rather than relying on conventional feature completion or global fusion. The paired equal-budget erasure strategy provides a practical self-supervised signal without requiring complete-view teachers, while the multiscale decomposition, damage-guided value learning, and cross-modal reading form a coherent technical pipeline. Extensive experiments across three Chinese and English benchmarks, multiple missing settings, and both classification and regression tasks, together with ablations and visualizations, provide strong empirical support for the proposed framework.

Critical weaknesses

A major weakness is the limited evaluation under realistic missing patterns. The current experiments focus on relatively idealized settings, such as random erasure, equal-length consecutive missing spans, and complete modality dropout, leaving the robustness of DREAM under more complex and correlated real-world failures insufficiently characterized. This directly affects confidence in the practical generalization of the proposed framework. A second weakness is the incomplete comparison with recent approaches, particularly methods such as DNRNet (CVPR 2026) and LLM-based multimodal solutions, which makes it difficult to fully assess the competitiveness of DREAM against the latest developments in incomplete multimodal learning.

Questions for authors

Clarification requests that could materially affect my assessment:

  1. How does DREAM compare conceptually and empirically with very recent incomplete multimodal methods such as DNRNet (CVPR 2026) and LLM-based multimodal approaches? Please clarify the main relative advantages and whether the conclusions remain competitive against these newer methods.

  2. How is DREAM expected to behave under more realistic missing patterns, such as long consecutive sensor failures or correlated missing events across multiple modalities? Please clarify the likely performance boundaries and which components are expected to remain effective or become less reliable.

  3. What is the rationale for the key structural hyperparameters, particularly the number of multiscale levels and the erasure-region size? Please clarify whether the method is sensitive to these choices and how much tuning would be required when transferring DREAM to new settings.

Suggestions for future work:

Further qualitative analysis of whether high-value regions align with human-interpretable emotional cues, and extension of DREAM to other incomplete multimodal tasks beyond sentiment analysis, would improve the broader impact of the framework.

Reviewer_b73HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new framework for incomplete multimodal sentiment analysis, called DREAM. In particular, the proposed method consists of two main components: damage-guided multiscale evidence learning and summary-referenced cross-modal reading. The first component estimates the importance of local regions by comparing the prediction errors caused by different erasures, while the second component combines global summaries with local evidence for cross-modal interaction. Finally, experiments on 3 popular MSA datasets, including MOSI, MOSEI, and CH-SIMS, are conducted to demonstrate the effectiveness of the proposed method. In general, I believe this work is interesting and addresses an important problem in incomplete multimodal learning. But I still have some questions for the authors for clarification.

Critical strengths
  1. The motivation is intuitive and interesting. This work considers the different impacts of missing local information instead of relying only on global representations, enriching the design space for incomplete multimodal sentiment analysis.

  2. The proposed method combines local evidence learning with cross-modal interaction. In particular, using prediction damage to estimate the importance of local regions is a reasonable design for preserving sentiment-related information.

  3. The main results seem promising, especially on CH-SIMS, where the proposed method outperforms several popular baselines. Besides, the authors provide relatively comprehensive ablation studies and evaluations under different missing conditions.

Critical weaknesses
  1. I have some questions about the novelty of the proposed method. Local attention, multiscale representations, and cross-modal interaction have been widely explored in MSA. Although this work introduces damage-guided evidence learning, is the main improvement mainly from identifying important regions or from the additional model complexity? I think a comparison with simpler importance-based methods would help clarify the unique contribution of this work.

  2. Based on Tables 1–3, the improvements are not equally significant across different datasets. The results on CH-SIMS are impressive, but the improvements on MOSEI are relatively small. For example, F1+ only increases from 77.84 to 78.47, and Corr increases from 0.606 to 0.609. Are these improvements statistically significant? Besides, why does the proposed method achieve much larger improvements on CH-SIMS than on MOSEI? I think a more detailed analysis would strengthen the conclusions.

  3. This work uses prediction damage under paired regional erasures to estimate local evidence importance. However, the damage is calculated based on the model's own prediction errors. I wonder whether this estimation is reliable when the model itself makes incorrect predictions. Besides, the experiments mainly consider artificially generated missing patterns. Can the learned importance generalize to natural missingness or noisy audio-visual inputs? I think this is important for the practical application of incomplete MSA.

  4. The computational overhead of the proposed framework needs further discussion. Based on Table 13 in the Appendix, DREAM introduces considerably higher inference latency than RECAP, especially on MOSEI and CH-SIMS. Considering that the improvements on some datasets are relatively small, is the additional computational cost justified? Besides, can the authors provide a simpler version of the framework with a better performance-efficiency trade-off?

Questions for authors
  1. What is the main technical difference between damage-guided evidence learning and existing local attention or importance estimation methods? In particular, based on the ICLR submission deadlines, does this work compare or discuss those works in ACL/AAAI/CVPR/ICML 2025 or 2026, given the same benchmarks?

  2. Why are the improvements much larger on CH-SIMS than on MOSEI? Are the smaller improvements on MOSEI statistically significant?

  3. How does the proposed damage estimation avoid being affected by incorrect predictions from the current model?

  4. Can DREAM generalize to natural missingness or noisy audio-visual inputs beyond synthetic erasures?

  5. Can the authors further explain the trade-off between predictive performance and the additional inference latency introduced by DREAM?

Comments on the paper

4 comments

Dear reviewers,

As the authors have posted their responses, I would encourage you to actively parcitipate in the dicussion.

Thank you.

Best,

AC

AuthorsOpenReview ↗

Manuscript revisions in response to reviewer feedback

We sincerely thank all reviewers for their constructive feedback. Taking all reviewers' comments into account, we have incorporated the key clarifications and supporting results into the main text, with additional analyses and experimental details in the appendix. Substantive additions are highlighted in blue for ease of review. We hope these revisions clarify our responses to the concerns raised, and we welcome further feedback from the reviewers and Area Chair.

Dear AC,

Thank you sincerely for your kind reminder and for the attention you have given to our paper. We truly appreciate your efforts in encouraging an active discussion.

With the DDL approaching, we are sincerely looking forward to discussing with the remaining reviewers and would greatly value their feedback on whether our responses have addressed their concerns. We remain happy to provide any further clarification or results before the DDL.

Thank you again for your time and support.

Best regards, The Authors

AuthorsOpenReview ↗

General_Response

Dear AC and Reviewers,

We sincerely thank you for the time and patience you devoted to the discussion, and for your recognition of and support for this work. Your thoughtful and constructive comments helped us examine and improve DREAM more thoroughly.

Highlights of DREAM

The reviewers recognized these main highlights:

  • A fresh perspective: DREAM models task-value differences across local regions instead of relying on feature completion or global fusion.
  • A coherent, teacher-free design: paired equal-budget erasure provides a practical self-supervised signal without complete-view teachers, within a coherent three-module pipeline.
  • Strong results: on CH-SIMS, DREAM improves Acc-2 and F1 over MIDAS by 4.11 and 5.05 points and reduces MAE by 12.72%. In the rebuttal, it stayed best under real ASR noise and raw-signal missingness, and against DNRNet/SemMSA.
  • Comprehensive evaluation: three Chinese and English benchmarks, classification and regression, multiple missing settings, ablations and visualizations.

Key Comments

Without changing the experimental protocol or the model architecture, we added extensive experiments. Most verify the current method's effectiveness; several core comments matter greatly for this work's future, so we have incorporated them and the supporting evidence into the paper (substantive revisions in blue):

  • b73H: Evaluation using only the mean over ten missing rates. A cross-rate mean can hide weaknesses at individual rates, and readers need to see performance at each rate. We added a discussion of per-rate performance to Section 4.2 and per-rate results for all metrics in App. A.1–A.2.
  • 7mMV: Performance under real noise. Existing protocols simulate missingness on features, whereas real missingness occurs in raw signals. We added experiments on real ASR transcription noise (App. A.6, Table 16) and on missingness introduced directly into raw video, audio and text, re-extracting features with the official pipeline (App. A.6.2, Tables 17–18).
  • d39Y: Whether high-value words correspond to sentiment words. Whether high-value regions match human-interpretable sentiment cues needs a statistical test. We checked 667 fully observed MOSI test utterances against the VADER lexicon. At the finest scale, the highest-valued word appears in the lexicon 16.6% of the time, versus 11.7% for random within-utterance selection (permutation test, p < 0.001). With scores averaged across scales, the hit rate is 11.8%, close to chance (App. C.1).

Overview of All Comments

Reviewer b73H

  1. Gains from identifying important regions vs. added complexity: comparison with four simple importance-based methods (Table 4(c), App. B.1).
  2. Why gains differ between CH-SIMS and MOSEI: dataset-characteristics analysis (App. B.5, Table 23).
  3. Significance of the smaller MOSEI gains: multi-seed tests (App. A.7, Table 20).
  4. Reliability of damage estimation when the model's predictions are wrong: training-error and correction-trajectory analysis (App. B.6, Table 24).
  5. Inference cost and performance–efficiency trade-off: DREAM-Lite (App. E, Tables 27–30).
  6. Follow-up: comparison with classic and recent methods under complete inputs, i.e., missing rate 0 (App. F, Tables 31–34); positioning and comparison settings in the main text (Sections 2 and 4.2).

Reviewer 7mMV

  1. More realistic missing patterns (long contiguous, correlated cross-modal and complete modality loss) and applicability boundaries: Section 4.3, Figure 4, App. A.3 (Tables 8–11).
  2. Comparison with DNRNet and LLM-based SemMSA: Section 2.1, Figure 5(d), App. A.5 (Tables 13–15).
  3. Rationale for and sensitivity to key hyperparameters (number of scales, deletion-window size): Figure 5, App. B.4 (Tables 21–22).
  4. Future work: extension to incomplete multimodal tasks beyond sentiment analysis (Figure 7, App. A.6, Table 19).
  5. Follow-up: raw-signal missingness (missing video frames, corrupted audio, incomplete text): App. A.6.2 (Tables 17–18); real ASR noise in App. A.6 (Table 16).

Reviewer d39Y

  1. Whether novelty mainly comes from combining existing techniques: clarified contributions and related-work comparison (Sections 1–3, 2.2).
  2. Independent contribution of damage-guided learning vs. extra supervision and architecture: Table 4(c)(d), App. B.1.
  3. Whether equal-length deletion controls for regional differences: deletion-strategy controls (Table 4(e), App. B.1).
  4. Replacing damage supervision with gradient or perturbation importance: Table 4(d), App. B.1.
  5. High-value words vs. sentiment words: VADER hit rate vs. random selection (App. C.1).
  6. Training time, computational cost, inference latency and extra forward passes for damage: App. D.3 and E (Tables 27–30).

Thank you again for your valuable comments and support.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.