acceptodds
Under review as a conference paper at ICLR 2027

Reason, Decompose, and Deliberate: Partial-Information-Guided Multimodal Sentiment Analysis

Abstract

Multimodal Sentiment Analysis (MSA) is challenging because textual, acoustic, and visual cues relate to sentiment differently across instances. They may jointly convey a meaning absent from any single modality (e.g., sarcasm), or leave the decisive cue to one modality. To handle this variability, we propose the Reason–Decompose–Deliberate (RDD) framework, which organizes multimodal evidence into three cross-modal roles of redundancy, synergy, and uniqueness. This role-level view turns fusion into an instance-specific decision about how evidence should be combined, so that shared, interaction-dependent, and modality-specific cues are modeled separately rather than entangled in a single representation. To Reason, Multimodal Information-Role Reasoning and Routing (MIRR) prompts an MLLM with chain-of-thought to estimate an instance-specific cross-modal role vector and maps it to routing gates. To Decompose, Information-Role Guided Expert Decomposition (IGED) extracts redundant, synergistic, and unique features with dedicated experts and fuses them via gate-weighted summation across modalities. To Deliberate, Uncertainty-Aware Latent Deliberation (UALD) identifies uncertain samples by predictive entropy and iteratively refines their representations with a latent think token before prediction. Consequently, RDD unifies role reasoning, role-specific decomposition, and uncertainty-driven refinement into a single process that adapts to the evidence configuration of each instance. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show that RDD achieves the best results on all reported metrics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Analysis,Expert Routing,Chain-of-Thought,Partial Information Decomposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UsLfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Reason–Decompose–Deliberate (RDD) framework for multimodal sentiment analysis, integrating cross-modal information-role reasoning, expert-based decomposition, and uncertainty-aware latent deliberation. The framework is well structured and achieves competitive performance on three benchmark datasets.

Critical strengths
  1. The introduction of redundancy, synergy, and uniqueness into adaptive multimodal fusion is interesting and potentially valuable.
  2. The experimental evaluation includes comparisons with recent methods, ablation studies, and visualization analyses.
Critical weaknesses
  1. Insufficient validation of information decomposition. Although inspired by Partial Information Decomposition (PID), the framework lacks explicit information-theoretic constraints to ensure that the experts capture redundancy, synergy, and uniqueness as intended. The t-SNE visualizations alone are insufficient to demonstrate effective disentanglement.

  2. Questionable effectiveness of latent deliberation. UALD primarily relies on confidence-guided stochastic perturbations. However, higher prediction confidence does not necessarily imply better prediction accuracy. Further analysis is needed to justify the effectiveness and stability of this mechanism.

Questions for authors
  1. Limited novelty. The three expert modules mainly rely on existing techniques, including VAE, Cross-Attention, KAN, and TCN. The overall framework appears to be an integration of established components rather than a fundamentally novel methodological contribution.

  2. Insufficient validation of information decomposition. Although inspired by Partial Information Decomposition (PID), the framework lacks explicit information-theoretic constraints to ensure that the experts capture redundancy, synergy, and uniqueness as intended. The t-SNE visualizations alone are insufficient to demonstrate effective disentanglement.

  3. Questionable effectiveness of latent deliberation. UALD primarily relies on confidence-guided stochastic perturbations. However, higher prediction confidence does not necessarily imply better prediction accuracy. Further analysis is needed to justify the effectiveness and stability of this mechanism.

  4. The computational overhead introduced by MLLM-based reasoning and multiple experts is not adequately evaluated.

6 comments
AuthorsOpenReview ↗

Response to Question 1: Limited novelty.

We thank the reviewer for the constructive comments and for recognizing the potential value of information-role-guided fusion and the breadth of our experimental evaluation.

We acknowledge that VAE, cross-attention, KAN, and TCN are established building blocks. Our methodological contribution lies in how role-specific expert modeling, uncertainty-adaptive latent refinement, and feedback-based prior revision are formulated and coupled for instance-specific multimodal sentiment prediction. We will clarify these contributions and distinguish them from the underlying architectural components.

1. Role-specific expert architecture and coordination

The experts have different computational structures and training constraints, rather than serving as interchangeable transformations. The R-expert combines variational compression with cross-modal contrastive alignment to encourage compact, consistent representations. The S-expert performs cross-modal attention followed by nonlinear projection to model interaction-dependent patterns. The U-expert explicitly conditions its suppression gate on the R-expert's representation and reconstructs the original input jointly with the redundancy representation, encouraging complementary modality-specific features. In particular, the U branch is structurally coupled to the R branch rather than being an independently learned private encoder.

MIRR further translates MLLM-derived semantic assessments of redundancy, synergy, and uniqueness into modality-specific expert weights through a learnable router. Thus, the design combines role-specific computational biases with sample-dependent semantic guidance. Relative to statistical decomposition methods such as DMIL, our distinction is the use of MLLM-derived semantic role priors to guide these specialized operators, together with subsequent latent refinement and prior re-evaluation. We do not claim that the individual building blocks are new or that the expert outputs constitute an exact PID decomposition.

2. Uncertainty-adaptive latent refinement

UALD introduces a sample-dependent refinement process after expert fusion. At each iteration, the current latent refinement state is added residually to the fused representation. The model evaluates predictions under Gaussian perturbations, uses their dispersion to estimate predictive uncertainty, and updates the state through a confidence-weighted combination of the perturbations. Refinement terminates when uncertainty falls below a threshold or the iteration budget is exhausted. This design couples uncertainty-based computation allocation with a prediction-guided update direction: uncertainty determines whether further refinement is needed, while candidate confidence determines how the latent state changes. The resulting state supplies an adaptive residual correction to the expert-fused representation.

3. Feedback-based revision of role priors

The feedback mechanism complements latent refinement by revisiting the semantic guidance itself. During training, samples with large prediction errors and high self-reported MLLM confidence are queued for offline re-evaluation. The MLLM receives the training sample's ground-truth sentiment and network prediction as feedback and reconsiders its earlier role assessment. Revised role vectors are cached for subsequent training epochs, allowing the routing guidance to be updated rather than remaining fixed throughout training.

Importantly, this selection criterion triggers re-examination; it does not establish that every selected sample has an incorrect role prior, since downstream processing can also cause prediction errors. The mechanism therefore links downstream training outcomes to the reassessment of upstream semantic guidance, while UALD separately refines the fused latent representation. This feedback is restricted to training and does not use test labels.

AuthorsOpenReview ↗

Response to Weakness 1 and Question 2: Insufficient validation of information decomposition.

We thank the reviewer for raising this question. Our decomposition is guided by both architectural dependencies and training constraints, rather than by t-SNE visualization alone. As described in Section 3.2, the R branch uses KL regularization and cross-modal contrastive alignment, while the U branch employs redundancy-conditioned gating and joint reconstruction. These designs encourage compact shared representations and complementary modality-specific features. Appendix A.1 further positions the experts as PID-inspired functional inductive biases rather than an exact recovery of PID information atoms.

1. Controlled comparisons with additional constraints

To examine whether additional explicit constraints improve this formulation, we conducted the controlled comparisons below. C0 retains the original RDD objective. C1 adds auxiliary sentiment supervision to the R branch to encourage target-information preservation. C2 introduces an explicit mutual-information penalty between the R and U representations. Unlike auxiliary prediction supervision, this objective directly targets statistical dependence between the learned representations and therefore constitutes an additional information-theoretic regularizer. C3 adds auxiliary sentiment supervision to the S branch, and C4 combines all three objectives. All settings use the same data splits, input features, backbone, and training protocol, with auxiliary weights selected on the validation set.

For each expert group and modality, C2 pairs the R-expert's decoded output with the corresponding U-expert output after gating, temporal convolution, and pooling. Each pair has its own conditional-density estimator; representations from different modalities are not concatenated into a single estimation problem. Before estimation, both vectors undergo LayerNorm without learnable affine parameters and receive independent Gaussian noise with a default standard deviation of 0.05. The penalty therefore operates on normalized, noise-perturbed expert representations.

SettingR-task lossR/U MI lossS-task lossAcc2 ↑F1 ↑Acc7 ↑Corr ↑MAE ↓
C0. Original RDD———87.8287.7949.540.8250.673
C1. R supervisionYes——86.2686.2543.210.8050.682
C2. R/U MI penalty—Yes—87.1787.1944.810.8090.680
C3. S supervision——Yes85.5185.5843.940.7950.740
C4. CombinedYesYesYes86.4886.4646.860.8020.679

2. Results and interpretation

C2 provides the most direct test of adding an information-theoretic separation objective. Relative to C0, it reduces Acc2 from 87.82% to 87.17% and Acc7 from 49.54% to 44.81%, while increasing MAE from 0.673 to 0.680. Thus, explicitly penalizing R/U dependence does not improve sentiment prediction under the evaluated setting. Importantly, statistical independence between two learned representations is not equivalent to PID separation of their information about the target. Both representations can retain target-related dependencies, and suppressing these dependencies may also remove useful predictive structure. This is a possible explanation for the observed degradation, rather than evidence that stronger independence necessarily produces better role specialization.

The auxiliary-supervision comparisons provide complementary evidence. C1 and C3 encourage each branch to predict sentiment independently, but this does not directly enforce redundancy or synergy: an auxiliary predictor can exploit any available target-relevant cue. Their performance reductions therefore suggest that independently strengthening branch-level prediction is not sufficient to improve the intended role-guided processing. Combining all objectives in C4 also remains below C0 across all five metrics, indicating that the additional objectives do not provide complementary gains in this configuration.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 3: Questionable effectiveness of latent deliberation.

We thank the reviewer for this constructive question.

We evaluate latent deliberation through its effect on prediction performance, rather than assume that greater confidence necessarily implies greater accuracy, as detailed in Appendix A.12. Specifically, we compare disabling iteration, uniform-weight updates with a fixed iteration count, confidence-weighted updates with the same fixed iteration count, and adaptive iteration with an entropy threshold of 0.5 against the full RDD configuration.

SettingAcc2 ↑F1 ↑Acc7 ↑Corr ↑MAE ↓
U0. No iteration86.2586.1747.850.7700.715
U1. Fixed, uniform86.7386.7545.210.8030.716
U2. Fixed, confidence87.3287.3344.080.8140.684
U3. Adaptive, threshold 0.586.4186.4046.270.8030.689
U4. RDD87.8287.7949.540.8250.673

U1 and U2 use the same fixed iteration count with uniform and confidence-based perturbation weights, respectively. U3 uses entropy-based adaptive stopping with a threshold of 0.5. U4 uses the full RDD configuration with the default threshold of 0.3.

1. Effectiveness of latent deliberation

Compared with disabling iteration, full RDD improves Acc2 by 1.57 percentage points, F1 by 1.62 percentage points, and Acc7 by 1.69 percentage points. Correlation increases from 0.770 to 0.825, while MAE decreases from 0.715 to 0.673. These consistent improvements across classification and regression metrics demonstrate the empirical benefit of refining the fused latent representation before prediction.

2. Why unguided perturbation is insufficient

U1 improves Acc2 and F1 over U0, but lowers Acc7 and leaves MAE essentially unchanged. Uniform weighting gives all candidate perturbations equal influence, regardless of their predictive quality. Consequently, it does not preferentially retain directions supported by the predictor. The mixed results indicate that additional stochastic computation alone is insufficient to deliver consistent gains.

3. Why confidence weighting also needs adaptive stopping

With the iteration count held fixed, U2 improves Acc2 by 0.59 percentage points and reduces MAE by 0.032 relative to U1, supporting the usefulness of confidence-guided update directions. However, Acc7 decreases from 45.21% to 44.08%. A fixed iteration budget does not account for differences in sample uncertainty: it may continue updating already-resolved samples or stop before difficult samples are sufficiently refined. This offers a plausible explanation for the uneven gains, without assuming that every confidence-increasing update improves accuracy.

4. Why the full configuration performs better

RDD combines confidence-weighted updates with sample-dependent stopping. Relative to U2, it improves Acc7 by 5.46 percentage points and reduces MAE from 0.684 to 0.673, indicating that controlling when refinement terminates matters alongside the update direction. U3 further shows that adaptive stopping requires an appropriate threshold: the larger threshold of 0.5 permits termination at higher uncertainty and may stop refinement prematurely. With the default threshold of 0.3, RDD improves Acc2 by 1.41 percentage points and reduces MAE by 0.016 relative to U3. Together, these comparisons support the complementary roles of guided exploration and uncertainty-dependent stopping in achieving balanced performance across all five metrics.

Reviewer_XwgdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Reason–Decompose–Deliberate (RDD) framework for multimodal sentiment analysis. Inspired by Partial Information Decomposition (PID), it divides multimodal sentiment information into three roles: redundancy, synergy, and modality-specific uniqueness. First, a multimodal large language model (MLLM) analyzes the relationships between modalities and produces a per-sample "prior" over these roles, which guides dynamic feature fusion. Then, three specialized expert networks extract and integrate the information for each role. Finally, an iterative mechanism guided by prediction confidence refines the latent representations of uncertain samples, while feedback from misclassified samples is used during training to correct the initial role priors. The authors report competitive results on MOSI, MOSEI, and CH-SIMS.

Critical strengths

The main contribution is applying PID to multimodal sentiment analysis, which gives a fresh view of how modalities contribute to sentiment through redundancy, synergy, and uniqueness. In complex expressions such as sarcasm, irony, or jest, different modalities may convey contradictory cues, and the true sentiment often depends on how they interact. PID offers a meaningful framework for describing such cross-modal interactions. Although the paper does not fully show that the proposed modules actually achieve the intended decomposition, the motivation and the problem modeling are valuable for future research.

Critical weaknesses
  1. The contribution of the MLLM is not separated from that of the rest of the architecture. The authors use Qwen2.5-Omni to analyze inter-modal relationships and guide feature fusion. However, they do not compare against a baseline that uses Qwen2.5-Omni directly for sentiment prediction, nor do they test the same MLLM with a simpler fusion architecture. In addition, the existing MLLM-based baseline, MMSLF, uses GPT-4o-mini (with Gemini-2.0-Flash in an additional experiment), which differs from Qwen2.5-Omni. As a result, the experiments cannot tell how much of the gain comes from the MLLM itself and how much from the additional complex architecture.

  2. The synergy expert lacks a clear mechanism to separate synergistic from redundant information. In the paper, redundancy refers to sentiment information that multiple modalities can provide individually, while synergy refers to information that can only be obtained by combining modalities. Cross-attention enables information exchange across modalities, but it can also extract information already available from individual modalities. For example, when text, audio, and vision all express positive sentiment, cross-attention may capture their shared positive cues, which remain redundant despite cross-modal processing. The subsequent KAN provides nonlinear transformations but does not address this distinction.

  3. The confidence-based feedback mechanism is not well justified. During training, the authors flag samples with large prediction errors but high self-reported MLLM confidence, and ask the MLLM to re-evaluate its earlier role assessment. However, such errors may also come from the downstream feature extraction or fusion modules, and do not necessarily mean the MLLM's role assessment was wrong. The paper provides no analysis to verify that this error attribution, and the resulting correction, are reasonable.

Questions for authors
  1. Necessity of the proposed architecture beyond the MLLM: Could you add a baseline where the same MLLM directly predicts sentiment? This would help clarify the additional contribution of the proposed architecture.

  2. Validity of the decomposition: Could you give a more convincing argument, or evidence, that cross-attention plus KAN extracts synergistic rather than redundant information? Also, why does the paper use a KAN with sine basis functions instead of the classic B-spline-based KAN?

  3. UALD and the feedback mechanism: Could you use human annotations or case studies to check whether the MLLM's role assessments are actually flawed for the error samples selected by the feedback mechanism? In addition, why is the UALD mechanism called a "Latent Think Token" (page 6, second-to-last line)? It does not seem to involve what is usually meant by "thought tokens" in the LLM literature.

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1: The contribution of the MLLM is not separated from that of the rest of the architecture. And necessity of the proposed architecture beyond the MLLM.

We thank the reviewer for the careful assessment and constructive suggestions, and for recognizing the motivation and information-role perspective of RDD. Below, we jointly address Weakness 1 and Question 1 concerning the contributions of the MLLM and the downstream architecture.

1. MLLM selection

GPT-4o-mini and Gemini-2.0-Flash, used by MMSLF, are proprietary models without publicly available weights. We chose Qwen2.5-Omni-3B to support local deployment and direct control over the checkpoint, multimodal preprocessing, and token-score extraction required for role-prior generation. Nevertheless, we agree that comparisons with MMSLF alone cannot isolate architectural benefits because the MLLM backbones differ.

2. Controlled comparisons

We added a baseline in which the same Qwen2.5-Omni-3B [3] checkpoint directly outputs a continuous sentiment regression value, as detailed in Appendix A.11. The comparison uses the same multimodal inputs, preprocessing, and comparable prompting conditions, with identical evaluation metrics. To make the evaluation more comprehensive, we additionally evaluated two other open-weight MLLMs, gemma-3n-E2B-it [1] and Megrez-3B-Omni [2], for both direct sentiment regression and RSU-prior generation.

SettingAcc2 ↑F1 ↑Acc7 ↑Corr ↑MAE ↓
gemma-3n-E2B-it [1] (direct prediction)85.1685.1845.500.7920.753
Megrez-3B-Omni [2] (direct prediction)85.7585.7744.130.7840.766
Qwen2.5-Omni-3B [3] (direct prediction)85.6485.7243.210.7910.785
gemma-3n-E2B-it [1] (RSU priors)86.4686.4548.350.8060.689
Megrez-3B-Omni [2] (RSU priors)87.5087.4846.770.7950.692
RDD (Qwen2.5-Omni-3B [3] RSU priors)87.8287.7949.540.8250.673

With Qwen2.5-Omni-3B held fixed, RDD improves Acc2 from 85.64 to 87.82, F1 from 85.72 to 87.79, and Acc7 from 43.21 to 49.54. Correlation increases from 0.791 to 0.825, while MAE decreases from 0.785 to 0.673. These gains show that the complete framework provides benefits beyond direct sentiment prediction by the same MLLM.

The improvements also extend to the alternative backbones. For gemma-3n-E2B-it, RSU-guided prediction improves Acc2 by 1.30 percentage points and Acc7 by 2.85 percentage points, while reducing MAE from 0.753 to 0.689. For Megrez-3B-Omni, the corresponding improvements are 1.75 and 2.64 percentage points, with MAE decreasing from 0.766 to 0.692. Both backbones also improve F1 and correlation, supporting the framework's utility across different MLLMs.

The alternatives exhibit complementary strengths: Megrez-3B-Omni achieves higher Acc2 and F1, whereas gemma-3n-E2B-it achieves higher Acc7 and correlation and lower MAE. Although Qwen2.5-Omni-3B is not uniformly strongest at direct prediction, its RSU priors yield the best downstream performance across all five metrics. This comprehensive performance supports its suitability as our role-prior generator.

References

[1] Gemma Team. Gemma 3n. 2025. https://ai.google.dev/gemma/docs/gemma-3n.

[2] Boxun Li et al. Megrez-Omni Technical Report. arXiv:2502.15803, 2025.

[3] Jin Xu et al. Qwen2.5-Omni Technical Report. arXiv:2503.20215, 2025.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2: The synergy expert lacks a clear mechanism to separate synergistic from redundant information. And validity of the decomposition.

We thank the reviewer for the opportunity to clarify how the S-expert functions within the complete role-guided architecture.

1. Role-guided modeling across diverse configurations

The example in which all three modalities express positive sentiment represents an agreement-dominant configuration, but does not by itself establish that the S-expert is ineffective across diverse multimodal interactions. Cross-attention may retain shared positive cues in this setting; however, RDD does not require every sample to rely equally on every expert. MIRR assesses the evidence configuration, and the learnable router adjusts the contributions of the specialized branches accordingly. When modalities independently provide consistent evidence, the redundancy branch is intended to receive greater emphasis. The S-expert provides additional modeling capacity for more challenging configurations, where the relationship among modalities is important for interpreting sentiment. Its cross-attention captures cross-modal dependencies, and the subsequent nonlinear projection models context-dependent responses to the exchanged information. Thus, the rationale for the S-expert is its complementary function across heterogeneous samples, rather than the exclusion of every shared cue in an agreement-dominant example that conventional fusion can also handle.

2. Representative cases and expert ablation

The two real samples illustrate this sample-dependent coordination. In 1iG0909rllw.25, the transcript "MY FAVORITE SCENE WOULD DEFINATELY BE THE VOLTURI" and the speaker's smiling expression both convey positive affect. MIRR assigns an RSU prior of (0.62, 0.15, 0.23), favoring redundancy rather than placing the greatest emphasis on the S branch. This case directly illustrates how our framework handles the consistent-evidence configuration raised by the reviewer.

In wMbj6ajWbic.4, the transcript "BUT I WASNT BIG ON IT UM" expresses a negative evaluation, whereas the visual expression conveys a different affective tendency. The MLLM identifies this cross-modal discrepancy and assigns an RSU prior of (0.28, 0.54, 0.18), providing stronger guidance toward the S branch. Cross-attention and nonlinear projection then provide the capacity to model the relationship between these inconsistent cues, rather than simply aggregate their individual polarities.

This division of responsibilities connects semantic evidence assessment in MIRR with interaction modeling in the S-expert. Its predictive contribution is also supported by the existing expert ablation: removing the S-expert reduces Acc2 from 87.82% to 87.20% and Acc7 from 49.54% to 48.55%. We will add the two-case visualization and its analysis to the main text to make this adaptive coordination explicit.

3. Choice of sine-basis parameterization

We use a sine-basis parameterization because it provides a nonlinear mapping without requiring spline knots or grid adaptation, as detailed in Appendix A.3. Its global basis functions allow the learned mapping to be adjusted across the latent feature range, which may be useful for modeling cross-modal relationships. Here, frequency refers to variation over latent feature values, rather than the physical frequency of audio or video signals.

To evaluate this choice empirically, we compared a two-layer MLP, two cubic B-spline configurations, and the sine-basis projection while retaining the same encoder, cross-attention module, and training protocol. As shown below, the sine-basis configuration achieves the best results across all five metrics. Compared with the MLP, it improves Acc2 by 0.56 percentage points and reduces MAE by 0.012. Compared with the five-interval B-spline configuration, it improves Acc2 by 2.55 percentage points and reduces MAE by 0.035. These results demonstrate the effectiveness of the sine-basis projection for cross-modal feature transformation and provide empirical justification for its use in RDD.

SettingAcc2 ↑F1 ↑Acc7 ↑Corr ↑MAE ↓
K1. MLP87.2687.2148.670.7810.685
K2. B-spline (1)84.6184.6542.980.7820.761
K3. B-spline (5)85.2785.2044.530.8020.708
K4. Sine (RDD)87.8287.7949.540.8250.673

K1 uses a two-layer MLP. K2 and K3 use cubic B-splines with 4 basis functions over 1 interval and 8 basis functions over 5 intervals, respectively. K4 uses the sine-basis projection adopted in RDD.

AuthorsOpenReview ↗

Response to Weakness 3 and Question 3: The confidence-based feedback mechanism is not well justified. And UALD and the feedback mechanism.

We thank the reviewer for raising this question about the rationale and effectiveness of the feedback mechanism.

1. Feedback criterion and downstream architecture comparisons

Our criterion identifies samples worth re-examining, rather than attributing every prediction error exclusively to the MLLM. Errors can arise at different processing stages, and the downstream predictor is not assumed to be error-free. We examine whether MIRR provides useful guidance across architectures and whether revisiting that guidance improves selected predictions.

Under a unified protocol, we compared classical architectures with and without MIRR, including RDD without MIRR as its corresponding baseline.

MethodAcc2: BaselineAcc2: +MIRRAcc7: BaselineAcc7: +MIRRMAE: BaselineMAE: +MIRR
TFN80.1283.4137.1440.670.8540.815
LMF82.3385.2536.9740.160.8260.796
MulT84.5486.5240.6245.370.8370.759
RDD86.4387.8248.1749.540.6970.673

For RDD, Baseline denotes the model without MIRR. Higher Acc2/Acc7 and lower MAE are better.

MIRR improves all three metrics across the four architectures. For TFN and LMF, Acc2 increases by 3.29 and 2.92 percentage points, while MAE decreases by 0.039 and 0.030. For MulT, Acc2 and Acc7 improve by 1.98 and 4.75 percentage points, with MAE decreasing from 0.837 to 0.759. For RDD, Acc2 and Acc7 improve by 1.39 and 1.37 percentage points, while MAE decreases from 0.697 to 0.673. These gains show that MIRR supplies useful guidance across fusion mechanisms, rather than benefiting only one design. They do not imply that downstream errors disappear; they establish that the guidance is meaningful and its re-evaluation can be investigated separately.

2. Training cases before and after feedback

We analyzed training samples selected by the feedback mechanism to examine whether revised priors help correct predictions, as detailed in Appendix A.7. RSU components below are ordered as redundancy, synergy, and uniqueness.

SampleRSU: before → afterPredictionLabelAbsolute error
A(0.48,0.16,0.36) → (0.36,0.12,0.52)2.8 → 1.81.41.4 → 0.4
B(0.41,0.19,0.40) → (0.49,0.14,0.37)2.0 → 1.10.41.6 → 0.7
C(0.28,0.50,0.22) → (0.31,0.24,0.45)-0.2 → -0.8-1.41.2 → 0.6
D(0.40,0.15,0.45) → (0.48,0.18,0.34)-2.7 → -1.4-1.21.5 → 0.2

A: 0h-zjBukYpk_14

WHICH IS A SMART MOVE SINCE YOU KNOW YOU HAVE TO BE REAL COMIC BOOK FAN ......

B: 1DmNV9C1hbY_12

SURE THERE WERE CUTE MONEY PARTS IN THE MIXTING OF THE FILM

C: 1iG0909rllw_29

THE REST OF THE FILM WAS JUST WATCHING KRISTEN JUST MOB AROUND

D: BXuRRbG0Ugk_9

BUT THIS PROBABLY HAD THE UNFORTUNATE SIDE EFFECT OF RAISING BAD EXPECTATIONS TO ......

In A, uniqueness becomes dominant and the prediction decreases from 2.8 to 1.8, closer to the label of 1.4. In B, redundancy increases as the prediction decreases from 2.0 to 1.1, reducing overestimation of mildly positive sentiment. In C, synergy decreases and uniqueness increases; the prediction moves from -0.2 to -0.8, halving the absolute error. In D, increased redundancy accompanies a correction of excessive negativity, reducing the error from 1.5 to 0.2.

The revisions do not uniformly increase one role or strengthen sentiment: uniqueness increases in A/C, redundancy increases in B/D, and sentiment magnitude changes in different directions. Mean absolute error across these four illustrative cases decreases from 1.425 to 0.475. These cases support the practical value of reconsidering role assessments and updating cached priors, while allowing that downstream processing may also contribute to the original errors. We will include these cases and clarify that feedback performs targeted re-examination rather than presupposing an incorrect role assignment for every flagged sample.

3. Clarifying "Latent Think Token"

The original name describes a continuous state supporting additional latent computation. It is not a discrete LLM vocabulary token and does not involve thought-token generation in the usual sense. To avoid ambiguity, we will consistently rename it "Recursive Deliberation State" and explicitly describe its initialization, perturbation-weighted updates, and residual integration with the fused representation.

AuthorsOpenReview ↗

Follow-up on our responses and manuscript revisions

Dear Reviewer Xwgd,

Thank you again for your thoughtful review and for recognizing the motivation and information-role perspective of RDD. We have provided detailed responses to the three weaknesses and the three corresponding questions, and revised the manuscript with additional experiments and clarifications. Following the same order as our responses, we summarize the main updates below:

  1. MLLM selection and the contribution of the architecture beyond direct MLLM prediction (Weakness 1 and Question 1). We clarified our choice of Qwen2.5-Omni-3B and acknowledged that comparisons with MMSLF alone cannot isolate architectural benefits because the MLLM backbones differ. We therefore added a direct sentiment-prediction baseline using the same Qwen2.5-Omni-3B checkpoint, multimodal inputs, and preprocessing, under comparable prompting conditions. RDD improves Acc2 from 85.64 to 87.82 and Acc7 from 43.21 to 49.54, while reducing MAE from 0.785 to 0.673. Additional comparisons with gemma-3n-E2B-it and Megrez-3B-Omni show improvements over their respective direct predictions across all five metrics (Appendix A.11, Table 9).

  2. The S-expert’s function and the choice of sine-basis parameterization (Weakness 2 and Question 2). Our response clarifies how MIRR assesses the evidence configuration and the router coordinates the specialized experts, while cross-attention and nonlinear projection model cross-modal dependencies. Two representative cases illustrate this coordination under consistent and conflicting evidence. The existing expert ablation supports the S-expert’s predictive contribution: removing it reduces Acc2 from 87.82 to 87.20 and Acc7 from 49.54 to 48.55 (Section 3.2.2; Appendix A.6, Table 6). To justify the sine-basis choice, we added comparisons with a two-layer MLP and two cubic B-spline configurations under the same encoder, cross-attention module, and training protocol. The sine configuration achieves the best results across all five metrics; compared with the five-interval B-spline, Acc2 improves by 2.55 percentage points and MAE decreases by 0.035 (Appendix A.3, Table 5).

  3. The feedback criterion, supporting cases, and UALD terminology (Weakness 3 and Question 3). We clarified that the feedback criterion identifies samples for re-examination, while acknowledging that downstream processing can also cause prediction errors. Following our detailed response, we first provided paired comparisons with and without MIRR across TFN, LMF, MulT, and RDD, showing improvements in all three reported metrics across the four architectures (Appendix A.15, Table 15). We then analyzed four selected training cases before and after feedback. Their sample-dependent prior revisions accompany a reduction in mean absolute error from 1.425 to 0.475, illustrating the practical utility of re-evaluation without establishing that every flagged sample has an incorrect role assessment (Appendix A.7, Table 7). Finally, we replaced “Latent Think Token” with “Recursive Deliberation State” (RDS) in the method description and framework diagram, clarifying its continuous-state interpretation, initialization, perturbation-weighted updates, and residual integration (Sections 3.3–3.4; Figure 2; Appendix A.4).

We would greatly appreciate your feedback on whether these responses and revisions address your concerns, and whether any points still require clarification. If these additions satisfactorily address your concerns, we would be grateful if you could reconsider your assessment and score in light of the new evidence. We remain happy to provide further clarification.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_MJpiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RDD, a multimodal Sentiment Analysis Framework. The authors first used an MLLM to assess the redundant, synergistic, and unique information in each sample and used these estimates to route features to three types of experts for fusion. Predictions are then refined through uncertainty-driven latent-space iterations, while feedback from erroneous predictions during training updates the Information role priors. However, I find the motivation insufficiently differentiated from prior work. Several studies have already explored PID in multimodal settings and provided explicit mathematical formulations and estimators for its synergy, redundancy, and unique information components. Against this background, the decision to use an MLLM to assign heuristic scores to these roles is insufficiently justified. The paper neither establishes how these scores relate to the underlying information-theoretic quantities nor demonstrates their advantages over existing PID-based approaches, leaving the necessity and novelty of this design unclear. Also, expert-based routing is widely used to fuse different modalities, which further weakens the authors' contribution.

Critical strengths
  1. The experiments cover three widely used benchmarks and achieve the best results among the reported baselines. However, MOSI and CH-SIMS are relatively small, and performance gains on these datasets may be sensitive to random seeds. The modest improvements are insufficient to establish a reliable advantage. Although the component ablations and replacement studies provide supporting evidence, they do not resolve this concern.

  2. As a standalone module, UALD offers a somewhat novel design by combining perturbation-based predictive uncertainty estimation with confidence-weighted latent updates for adaptive sentiment prediction refinement.

Critical weaknesses
  1. Insufficient discussion and comparison with directly relevant PID methods. The paper omits DMIL (CVPR 2026) and related multimodal PID work from NeurIPS 2023 and 2025. In particular, DMIL already studies the decomposition and learning of sample-specific redundant, unique, and synergistic interactions. The paper instead constructs role priors through MLLM judgments, without sufficiently establishing their advantages over existing decomposition methods or validating their correspondence to the claimed information roles. This leaves its novelty and methodological justification insufficiently established.

  2. Lack of direct empirical validation of uncertainty estimation. Removing the entire UALD module establishes its overall utility, but does not show whether estimated uncertainty correlates with prediction error.

  3. The experiments do not sufficiently isolate the benefit of Information role modeling from that of additional MLLM-derived information. Existing comparisons replace the role priors with static weights, generic attention, or uniform fusion, without retaining equivalent MLLM access.

  4. The MLLM is central to generating instance-specific role priors, yet its configuration is insufficiently documented. Even with the appendix, the exact checkpoint, multimodal input preprocessing, complete prompts, and decoding settings remain unspecified, limiting reproducibility of both the role-prior generation process and the overall results.

  5. The paper uses an MLLM to interpret multimodal inputs and generate role priors, but does not compare against a straightforward baseline that uses the same MLLM to predict sentiment directly. Under identical inputs and comparable prompting conditions, this comparison would clarify whether role reasoning, expert decomposition, and latent refinement provide benefits beyond the MLLM’s own sentiment prediction capabilities. Without it, the advantage of the proposed pipeline over direct MLLM inference remains unclear.

Questions for authors

Please refer to the weakness.

6 comments
AuthorsOpenReview ↗

Response to Weakness1: Insufficient discussion and comparison with directly relevant PID methods.

We thank the reviewer for the constructive suggestions and recognition of our benchmark evaluation and UALD design.

1. Related work, methodological differences, and comparisons

We have expanded Section 2.1 (page 3) to discuss directly relevant PID methods and their connections to and differences from RDD. RDD couples MLLM-derived semantic role guidance with specialized experts, uncertainty-adaptive latent refinement, and training-time feedback.

CVX [2] (NeurIPS 2023) estimates PID components through convex optimization for discrete-support distributions. Flow-PID [3] (NeurIPS 2025) uses information-preserving normalizing flows and latent Gaussian PID. LSMI [4] (ICML 2025) quantifies sample-level interactions using pointwise information measures and lightweight entropy estimation. In contrast, MIRR interprets multimodal evidence to construct instance-specific semantic role priors for routing, rather than directly estimating PID information amounts.

DMIL [1] (CVPR 2026) performs hierarchical variational decomposition, separates shared and unique predictive information, constructs synergy from residual components, and uses three-stage training with component-specific gating. I²MoE [5] (ICML 2025) trains interaction experts with weak supervision using intact and modality-perturbed inputs, then learns prediction reweighting. In contrast, RDD uses MIRR to guide differentiated IGED experts, UALD to refine uncertain fused representations, and training-time feedback to reconsider cached priors using downstream predictions.

We supplemented comparisons under the same data splits, input features, and metrics. We will separately document adaptation procedures for estimation methods and learning frameworks, including safeguards against test-label leakage.

SettingAcc2 ↑F1 ↑Acc7 ↑Corr ↑MAE ↓
P1. DMIL (CVPR26)87.4887.5045.400.8090.679
P2. CVX (NeurIPS23)86.5486.5445.370.8030.685
P3. Flow-PID (NeurIPS25)87.4787.4844.820.8230.690
P4. LSMI (ICML25)87.3987.3743.650.8120.679
P5. I²MoE (ICML25)86.7186.6944.960.8040.680
P6. RDD87.8287.7949.540.8250.673

RDD outperforms CVX-, Flow-PID-, and LSMI-based alternatives on all five metrics. Acc2 improves by 1.28, 0.35, and 0.43 percentage points; Acc7 improves by 4.17, 4.72, and 5.89 points; MAE decreases by 0.012, 0.017, and 0.006, respectively. These gains are consistent with MIRR guiding IGED's differentiated processing of shared, interaction-dependent, and modality-specific cues.

Compared with DMIL and I²MoE, Acc2 improves by 0.34 and 1.11 percentage points, and Acc7 by 4.14 and 4.58 points, respectively. MAE decreases from 0.679 and 0.680 to 0.673. MIRR supplies semantic guidance and UALD revisits uncertain representations through confidence-weighted updates. The sentiment-prior control and UALD update-strategy ablations further support these complementary functions; this table reports their combined effect.

2. Correspondence between role priors and information roles

We examined 200 MOSI samples using individual modalities and their combinations. Representative cases are shown below; T, A, and V denote text, audio, and vision.

SampleTAVT,AT,VA,VT,A,VTruth
High Rpospospospospospospospos
High Sposnegnegnegnegnegnegneg
High Uposposnegposnegnegnegneg

For high-R samples, at least two modalities independently yield correct, consistent judgments. Here, all modalities and combinations correctly predict positive sentiment, supporting the redundancy interpretation.

For high-S samples, we examine whether combinations improve prediction relative to their constituent unimodal inputs. Here, adding audio or vision corrects the erroneous positive text-only prediction.

For high-U samples, one modality supplies decisive evidence and remains correct when combined with others. Here, vision alone correctly predicts negative sentiment, while text and audio predict positive sentiment. Every combination containing vision remains negative, whereas the text–audio combination remains incorrect.

References

[1] Zequn Yang et al. Information-Theoretic Decomposition for Multimodal Interaction Learning. arXiv:2606.11614, 2026.

[2] Paul Pu Liang et al. Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework. NeurIPS, 2023.

[3] Wenyuan Zhao et al. Partial Information Decomposition via Normalizing Flows in Latent Gaussian Distributions. NeurIPS, 2025.

[4] Zequn Yang, Hongfa Wang, and Di Hu. Efficient Quantification of Multimodal Interaction at Sample Level. ICML, 2025.

[5] Jiayi Xin et al. I²MoE: Interpretable Multimodal Interaction-Aware Mixture-of-Experts. ICML, 2025.

AuthorsOpenReview ↗

Response to Weakness 2: Lack of direct empirical validation of uncertainty estimation.

We thank the reviewer for raising this concern.

Removing the entire UALD module demonstrates its overall utility, but does not directly establish whether the estimated uncertainty is associated with prediction error. We therefore supplement the ablation with an analysis of prediction risk and grouping consistency, as detailed in Appendix A.12.

1. Relationship between uncertainty and prediction error

We fixed the trained model and recorded sample-level uncertainty and absolute prediction error before any latent-state updates. We then sorted the 686 test samples by uncertainty in ascending order and divided them into five approximately equal-sized groups.

Group (Low → High)SamplesMAE ↓
11370.646
21370.667
31370.669
41370.685
51380.713

MAE increases monotonically across the five groups. Lower-uncertainty samples exhibit smaller prediction errors, whereas errors become larger toward the higher-uncertainty groups. This progression provides direct empirical evidence that the estimated uncertainty distinguishes different levels of prediction risk, complementing the original module ablation.

2. Consistency across repeated inference runs

Across repeated inference runs, we observed that a subset of samples was consistently assigned to the same uncertainty group when ranked from low to high. This recurring membership suggests that the grouping captures reproducible differences among samples, rather than being entirely driven by stochastic variation. Together, the increasing error trend and the observed consistency of group membership support the usefulness and repeatability of the uncertainty-based sample ranking.

AuthorsOpenReview ↗

Response to Weakness 3: The experiments do not sufficiently isolate the benefit of Information role modeling from that of additional MLLM-derived information.

We thank the reviewer for highlighting the need to separate information-role modeling from the benefit of additional MLLM-derived information.

1. Controlled comparison with equivalent MLLM access

We added a controlled comparison with equivalent MLLM access and the same downstream backbone, as detailed in Appendix A.12. Both settings use the same MLLM checkpoint, multimodal inputs, and preprocessing pipeline, with prompts tailored to the respective prediction targets. In the control setting, the MLLM produces negative, neutral, and positive sentiment probabilities instead of R/S/U role priors. Both three-dimensional vectors are mapped to expert weights through learnable routers with the same architecture and capacity.

We retain the same feature-extraction backbone, downstream experts, UALD module, and training protocol, changing the type of MLLM-derived guidance. Sentiment classes are therefore not directly assigned to individual experts; the control router can learn how to use sentiment probabilities for expert weighting.

SettingAcc2 ↑F1 ↑Acc7 ↑Corr ↑MAE ↓
Sentiment-prior-guided framework86.7586.7547.440.8020.735
R/S/U-prior-guided framework87.8287.7949.540.8250.673

2. Results and interpretation

The R/S/U-prior-guided framework improves Acc2 by 1.07 percentage points, F1 by 1.04 percentage points, and Acc7 by 2.10 percentage points. Correlation increases from 0.802 to 0.825, while MAE decreases from 0.735 to 0.673. The improvements span binary classification, fine-grained sentiment discrimination, and continuous estimation, indicating that the advantage is not restricted to a single evaluation criterion.

The two priors provide different guidance despite sharing the same dimensionality and MLLM source. Sentiment probabilities summarize the likely polarity, whereas R/S/U priors describe whether sentiment evidence is shared across modalities, depends on joint interpretation, or is concentrated in one modality. Because MLLM access, the downstream backbone, and router capacity are controlled, these results support the additional value of information-role semantics beyond simply introducing MLLM-derived sentiment information.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript revisions in response to reviewer feedback

We sincerely thank all reviewers for their thoughtful and constructive feedback. We have carefully considered the comments, provided point-by-point responses, and revised the manuscript accordingly. Key clarifications and supporting results have been incorporated into the main text, with additional analyses and experimental details in the appendix. Substantive additions are highlighted in blue for ease of review. We hope these revisions address the concerns raised and would greatly appreciate further feedback from the reviewers and Area Chair. We remain happy to provide any additional clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.