acceptodds
Withdrawn from ICLR 2027

Dynamic Modality-Aware Dual-Noise Training for Robust Multimodal Sentiment Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Sentiment Analysis (MSA) aims to infer human emotions by integrating text, audio, and visual modalities. However, real-world deployment remains challenging due to heterogeneous perturbations, such as random data loss and transmission-induced feature distortion in distributed sensing environments. Existing methods frequently presuppose static modality dominance and focus on mitigating isolated forms of noise. To address the diverse and complex noise perturbations encountered in real-world scenarios, we propose a Dynamic Modality-Aware Dual-Noise Training framework (DMDN). Our framework first introduces the Dynamic Information-guided Modality Correction Module (DIMC), which estimates modality degradation via KL divergence and dynamically identifies the most reliable modality to enhance corrupted features. In this framework, modality features are encoded into sparse spike trains by a spiking neuron-based sender, transmitted through simulated noisy channels, and decoded by a State Space Model (SSM)-based receiver. Our framework jointly models random data missing and channel noise, better reflecting the complex perturbation patterns in real-world distributed systems. Empirically, joint training under such multi-noise conditions encourages the model to suppress irrelevant perturbations while preserving task-relevant information. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art baselines across multiple MSA benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Analysis, Robust Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UBAgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses robust multimodal sentiment analysis under two perturbations: random data missing at the input and bit-level channel noise (BSC/BEC) during feature transmission in a distributed setting. The proposed DMDN consists of a Dynamic Information-guided Modality Correction module (DIMC), which estimates per-modality degradation via KL divergence, selects the least-degraded modality as dominant and corrects it with an adversarially generated proxy, and a joint source-channel coding pipeline in which features are encoded into spike trains by a Conv-BN-LIF sender with temporal-sequence attention, passed through a simulated binary channel, and decoded by a Mamba-based receiver before ALMT-style fusion. Experiments on MOSI, MOSEI and SIMS at a 0.5 missing rate report gains over LNLN and other baselines, with robustness curves, ablations and a sensitivity analysis.

Critical strengths
  1. Jointly considering input-level missing data and transmission-level corruption is a reasonable extension of the robust-MSA setting, and replacing a fixed language-dominant assumption with a per-sample choice is well motivated.
  2. The paper provides robustness curves over noise levels, a module ablation, a dominance-selection ablation, a loss-weight sensitivity study and a missing-rate sweep.
  3. The paper is concise and the overall pipeline is easy to follow.
Critical weaknesses
  1. The novelty is limited: the method is an assembly of existing components. DIMC reuses the dominant-modality correction of LNLN [3] (generator, effectiveness discriminator, completeness-weight loss, and even the same loss weights 0.9/0.8/0.1/1.0); fusion is ALMT's hypermodality learning; the spiking sender/channel/receiver follows the authors' cited spiking transmission frameworks [1, 2]; the TSA module is TCJA-SNN. The new element, KL-based dynamic dominance, gives marginal gains (Table 7: fixed-vision dominance matches the full model on Acc-7; MAE 1.084 vs. 1.067).
  2. The channel-noise setting is artificial and the comparison is not fair. Noise is injected into the model's own intermediate spike representation, and LNLN is evaluated through a 3-bit serialization interface built by the authors that it was never designed for. No conventional alternative (quantization + channel coding, DeepJSCC-style autoencoders, or plain repetition over the 8 timesteps) is compared, so the benefit of spikes + SSM is not established; Table 8 even shows higher inference time, and the FLOPs reduction is unexplained given the shared ALMT backbone.
  3. Several claims are contradicted by the paper's own results. The abstract says DMDN "consistently outperforms" baselines, yet Table 6 shows LNLN is better at missing rates ≤ 0.4 (e.g., 40.23 vs. 37.03 Acc-7 at 0.1), while the main tables report only the 0.5 operating point where DMDN starts to win. Training with channel noise, claimed to improve robustness, lowers MOSEI Acc-7 from 50.83 to 46.00/45.12 and MAE from 0.633 to 0.684/0.688 (Table 1).
  4. No standard deviations, seeds or significance tests are reported anywhere. Many margins (MOSI Acc-2 74.24 vs. 73.56, Corr 0.519 vs. 0.518, ~1-point differences in Table 3) are within typical seed noise on MOSI. Whether baseline numbers are re-runs or copied is not stated, and some entries look erroneous (MFMB-Net MOSEI Acc-2 47.64/58.11; MRUF MOSEI MAE 0.975).
  5. Technical and presentation issues: the inference-time proxy KL(R(X̃)‖X̃) measures how much the reconstructor changes its input rather than degradation, and the train/test mismatch is never validated; Section 3.3 says the sender emits binary spikes while Appendix A.1 quantizes "channel symbols" to 3-bit integers; Eq. (17) refers to 5×5/3×3/1×1 branches while Eq. (12) uses 1D kernels and Fig. 1 labels o^(1..3); H*_m and the supervision target for w (Eqs. 11, 19) are undefined; SIMS is described as "dialogues and interviews" although it is collected from movies and TV; robust-MSA baselines such as MMIN, GCNet, DiCMoR/IMDer and the semantic-communication literature are not discussed; no code is released.

The combined setting is worth studying, but in its current form the contribution is an assembly of prior modules, the evaluation is selectively reported without variance, and key claims are not supported. I would reconsider if the authors report mean±std over seeds across the full missing-rate range, add a fair channel baseline, and reconcile the contradictions above.

[1] Neuron-based Spiking Transmission and Reasoning Network for Robust Image-Text Retrieval (TCSVT 2023) [2] Multi-scale Spiking Pyramid Wireless Communication Framework for Food Recognition (TMM 2024) [3] Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024)

Questions for authors

see above

Reviewer_DR8UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Dynamic Modality-Aware Dual-Noise Training (DMDN) for robust multimodal sentiment analysis under random data missingness and channel transmission noise. It introduces a Dynamic Information-guided Modality Correction (DIMC) module to estimate modality degradation and adaptively select a dominant modality for feature enhancement. The corrected features are subsequently encoded into spike trains using an SNN-based sender, transmitted through simulated noisy channels, and reconstructed by an SSM-based receiver. Experiments on MOSI, MOSEI, and SIMS report competitive results under different corruption settings, supported by component ablations and robustness analyses.

Critical strengths
  1. The paper addresses a practically relevant robustness problem by considering both source-side data missingness and channel-side transmission noise. Integrating dynamic modality correction with spike-based transmission and SSM-based reconstruction provides a coherent framework for studying multimodal sentiment analysis under heterogeneous perturbations.

  2. The paper includes several useful experimental analyses. These experiments provide useful evidence of the proposed framework's empirical behavior.

Critical weaknesses
  1. The validity of the degradation-based modality selection mechanism is not sufficiently established. In Section 3.2, DIMC estimates modality degradation using the KL divergence between clean and corrupted features during training. However, at inference time, the clean features are unavailable, and the degradation score is instead computed between reconstructed and corrupted features. This introduces a discrepancy between the training and inference criteria, as reconstruction discrepancy does not necessarily reflect the actual degradation severity. Moreover, a smaller feature-level KL divergence does not necessarily indicate that a modality retains more task-relevant sentiment information. Although Table 7 compares dynamic selection against fixed modality dominance, it does not directly validate the accuracy of the estimated degradation scores or the resulting modality rankings. I would like to see more targeted experiments, such as comparing reconstruction-based rankings with oracle degradation rankings, evaluating their relationship with unimodal prediction quality, or comparing against simpler missing-ratio-based selection strategies.

  2. Although the paper includes MFMB-Net (AAAI 2025) and MRUF (2026), several closely related methods are absent, particularly TMDC (AAAI 2026) [1], which explicitly addresses both missing and noisy modalities, P-RMF (ACL 2025) [2], which models modality uncertainty under incomplete observations, and TF-Mamba (EMNLP 2025) [3], which employs Mamba-based modeling for robust multimodal sentiment analysis. More importantly, the central claim of joint source-channel robustness requires clearer controlled comparisons. While the appendices include mixed-noise evaluations and matched quantization interfaces, the existing results do not sufficiently isolate the benefits of the proposed SNN–SSM transmission architecture from those of conventional noise augmentation or differences in communication representations. More comprehensive comparisons under matched missingness and channel conditions, together with controlled evaluations of communication budgets and simpler noise-training alternatives, would provide stronger evidence for the claimed advantages.

Refs:

[1] Yan Zhuang et al. TMDC: A Two-Stage Modality Denoising and Complementation Framework for Multimodal Sentiment Analysis with Missing and Noisy Modalities. AAAI 2026.

[2] Aoqiang Zhu et al. Proxy-Driven Robust Multimodal Sentiment Analysis with Incomplete Data. ACL 2025.

[3] Xiang Li et al. TF-Mamba: Text-enhanced Fusion Mamba with Missing Modalities for Robust Multimodal Sentiment Analysis. Findings of EMNLP 2025.

Questions for authors
  1. In Section 3.2, modality degradation is estimated using clean–corrupted feature divergence during training but reconstructed–corrupted feature divergence during inference. How do the authors ensure that the two criteria produce consistent modality rankings? Have they evaluated the ranking agreement between these two estimates?
  2. Does a lower KL-based degradation score reliably indicate higher sentiment-predictive utility? In particular, how does the proposed selection mechanism behave when a modality is only mildly corrupted but contains limited sentiment-relevant information, while another modality is more heavily corrupted but retains stronger emotional evidence?
  3. Appendix A.1 describes matched 3-bit channel interfaces for DMDN and LNLN. Could the authors clarify whether the effective communication rate, transmitted bit budget, and transmission latency are also matched, particularly given the multiple SNN simulation timesteps?
  4. Please consider expanding the baseline comparison to include recent relevant approaches such as TMDC [1], P-RMF [2], and TF-Mamba [3], where evaluation protocols are comparable. In particular, TMDC appears especially relevant because it explicitly studies both missing and noisy modalities.
  5. I would like to see more targeted experiments that isolate the benefits of the proposed dual-noise training and SNN–SSM communication design, such as comparisons with conventional feature-noise augmentation, alternative low-bit feature encoders, or different training and testing channel conditions under matched communication budgets.

I would be willing to raise my score if the authors can adequately address these concerns, especially through more comprehensive baseline comparisons and targeted validation of the modality selection and communication mechanisms.

Reviewer_JsAXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DMDN to address random data missing and channel transmission noise in MSA. Specifically, the framework introduces a Dynamic Information-guided Modality Correction Module, which selects the dominant modality based on estimated modality degradation and uses cross-modal information to compensate for incomplete features. It also employs a spiking-neuron-based sender and a State Space Model-based receiver to improve robustness to transmission noise during feature encoding and recovery. Experiments on MOSI, MOSEI, and SIMS demonstrate improvements under certain evaluation settings and metrics, demonstrating the effectiveness of the proposed framework.

Critical strengths

This paper jointly considers random data missing and channel transmission noise, proposes a complete framework for robust MSA, and evaluates it on three widely used benchmarks. This setting has potential practical value.

Critical weaknesses
  1. This paper does not clearly specify the sources of baseline results, feature extraction configurations, or the complete training and evaluation noise protocols, making it difficult to assess fairness and reproducibility. The main comparisons are restricted to a 50% missing rate without sufficient justification. Although the appendix evaluates additional missing rates, it only compares LNLN and DMDN BSC on MOSI, which is insufficient to establish consistent advantages across models and datasets. Additionally, table caption placement is inconsistent between the main text and the appendix.
  2. DIMC closely resembles the Dominant Modality Correction module in LNLN in its proxy dominant feature generation, completeness-weighted correction, and adversarial learning structure. The newly introduced KL-divergence-based dynamic dominance selection lacks direct validation of its reliability. Furthermore, channel transmission noise is primarily implemented through simulated bit-level perturbations. This paper does not sufficiently justify this choice relative to source-side feature corruption or conventional noise augmentation, nor does it clearly establish the edge–cloud computation partition or matched communication budgets.
Questions for authors
  1. Are the baseline results in Tables 1 and 2 obtained through retraining, public checkpoints, or previously published results? Please specify the feature extractors for each modality, dataset versions, checkpoint selection protocol, random seeds, and whether corrected public evaluation implementations were used.
  2. Why was a 50% missing rate selected for the main comparisons? What missing rates and sampling strategies are used during training, validation, and testing? Could the authors provide comparisons against more competitive baselines across multiple missing rates on all three datasets?
  3. Beyond dynamic dominance selection, what substantive changes does DIMC introduce relative to the DMC module in LNLN? What evidence shows that the modality with the lowest KL divergence retains more sentiment-relevant information and that this criterion outperforms alternatives such as missing-rate-based selection, prediction confidence, or learned gating?
  4. Why prioritize channel transmission noise over acquisition noise or source-side feature corruption? Could the authors clarify the edge–cloud computation partition and compare against conventional feature noise augmentation and non-spiking encoders under matched transmitted-bit budgets, to distinguish the benefits of the communication mechanisms from those of general regularization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.