- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Reason–Decompose–Deliberate (RDD) framework for multimodal sentiment analysis, integrating cross-modal information-role reasoning, expert-based decomposition, and uncertainty-aware latent deliberation. The framework is well structured and achieves competitive performance on three benchmark datasets.
Critical strengths
- The introduction of redundancy, synergy, and uniqueness into adaptive multimodal fusion is interesting and potentially valuable.
- The experimental evaluation includes comparisons with recent methods, ablation studies, and visualization analyses.
Critical weaknesses
-
Insufficient validation of information decomposition. Although inspired by Partial Information Decomposition (PID), the framework lacks explicit information-theoretic constraints to ensure that the experts capture redundancy, synergy, and uniqueness as intended. The t-SNE visualizations alone are insufficient to demonstrate effective disentanglement.
-
Questionable effectiveness of latent deliberation. UALD primarily relies on confidence-guided stochastic perturbations. However, higher prediction confidence does not necessarily imply better prediction accuracy. Further analysis is needed to justify the effectiveness and stability of this mechanism.
Questions for authors
-
Limited novelty. The three expert modules mainly rely on existing techniques, including VAE, Cross-Attention, KAN, and TCN. The overall framework appears to be an integration of established components rather than a fundamentally novel methodological contribution.
-
Insufficient validation of information decomposition. Although inspired by Partial Information Decomposition (PID), the framework lacks explicit information-theoretic constraints to ensure that the experts capture redundancy, synergy, and uniqueness as intended. The t-SNE visualizations alone are insufficient to demonstrate effective disentanglement.
-
Questionable effectiveness of latent deliberation. UALD primarily relies on confidence-guided stochastic perturbations. However, higher prediction confidence does not necessarily imply better prediction accuracy. Further analysis is needed to justify the effectiveness and stability of this mechanism.
-
The computational overhead introduced by MLLM-based reasoning and multiple experts is not adequately evaluated.
6 comments
Response to Weakness 1 and Question 2: Insufficient validation of information decomposition.
We thank the reviewer for raising this question. Our decomposition is guided by both architectural dependencies and training constraints, rather than by t-SNE visualization alone. As described in Section 3.2, the R branch uses KL regularization and cross-modal contrastive alignment, while the U branch employs redundancy-conditioned gating and joint reconstruction. These designs encourage compact shared representations and complementary modality-specific features. Appendix A.1 further positions the experts as PID-inspired functional inductive biases rather than an exact recovery of PID information atoms.
1. Controlled comparisons with additional constraints
To examine whether additional explicit constraints improve this formulation, we conducted the controlled comparisons below. C0 retains the original RDD objective. C1 adds auxiliary sentiment supervision to the R branch to encourage target-information preservation. C2 introduces an explicit mutual-information penalty between the R and U representations. Unlike auxiliary prediction supervision, this objective directly targets statistical dependence between the learned representations and therefore constitutes an additional information-theoretic regularizer. C3 adds auxiliary sentiment supervision to the S branch, and C4 combines all three objectives. All settings use the same data splits, input features, backbone, and training protocol, with auxiliary weights selected on the validation set.
For each expert group and modality, C2 pairs the R-expert's decoded output with the corresponding U-expert output after gating, temporal convolution, and pooling. Each pair has its own conditional-density estimator; representations from different modalities are not concatenated into a single estimation problem. Before estimation, both vectors undergo LayerNorm without learnable affine parameters and receive independent Gaussian noise with a default standard deviation of 0.05. The penalty therefore operates on normalized, noise-perturbed expert representations.
| Setting | R-task loss | R/U MI loss | S-task loss | Acc2 ↑ | F1 ↑ | Acc7 ↑ | Corr ↑ | MAE ↓ |
|---|---|---|---|---|---|---|---|---|
| C0. Original RDD | — | — | — | 87.82 | 87.79 | 49.54 | 0.825 | 0.673 |
| C1. R supervision | Yes | — | — | 86.26 | 86.25 | 43.21 | 0.805 | 0.682 |
| C2. R/U MI penalty | — | Yes | — | 87.17 | 87.19 | 44.81 | 0.809 | 0.680 |
| C3. S supervision | — | — | Yes | 85.51 | 85.58 | 43.94 | 0.795 | 0.740 |
| C4. Combined | Yes | Yes | Yes | 86.48 | 86.46 | 46.86 | 0.802 | 0.679 |
2. Results and interpretation
C2 provides the most direct test of adding an information-theoretic separation objective. Relative to C0, it reduces Acc2 from 87.82% to 87.17% and Acc7 from 49.54% to 44.81%, while increasing MAE from 0.673 to 0.680. Thus, explicitly penalizing R/U dependence does not improve sentiment prediction under the evaluated setting. Importantly, statistical independence between two learned representations is not equivalent to PID separation of their information about the target. Both representations can retain target-related dependencies, and suppressing these dependencies may also remove useful predictive structure. This is a possible explanation for the observed degradation, rather than evidence that stronger independence necessarily produces better role specialization.
The auxiliary-supervision comparisons provide complementary evidence. C1 and C3 encourage each branch to predict sentiment independently, but this does not directly enforce redundancy or synergy: an auxiliary predictor can exploit any available target-relevant cue. Their performance reductions therefore suggest that independently strengthening branch-level prediction is not sufficient to improve the intended role-guided processing. Combining all objectives in C4 also remains below C0 across all five metrics, indicating that the additional objectives do not provide complementary gains in this configuration.
Response to Weakness 2 and Question 3: Questionable effectiveness of latent deliberation.
We thank the reviewer for this constructive question.
We evaluate latent deliberation through its effect on prediction performance, rather than assume that greater confidence necessarily implies greater accuracy, as detailed in Appendix A.12. Specifically, we compare disabling iteration, uniform-weight updates with a fixed iteration count, confidence-weighted updates with the same fixed iteration count, and adaptive iteration with an entropy threshold of 0.5 against the full RDD configuration.
| Setting | Acc2 ↑ | F1 ↑ | Acc7 ↑ | Corr ↑ | MAE ↓ |
|---|---|---|---|---|---|
| U0. No iteration | 86.25 | 86.17 | 47.85 | 0.770 | 0.715 |
| U1. Fixed, uniform | 86.73 | 86.75 | 45.21 | 0.803 | 0.716 |
| U2. Fixed, confidence | 87.32 | 87.33 | 44.08 | 0.814 | 0.684 |
| U3. Adaptive, threshold 0.5 | 86.41 | 86.40 | 46.27 | 0.803 | 0.689 |
| U4. RDD | 87.82 | 87.79 | 49.54 | 0.825 | 0.673 |
U1 and U2 use the same fixed iteration count with uniform and confidence-based perturbation weights, respectively. U3 uses entropy-based adaptive stopping with a threshold of 0.5. U4 uses the full RDD configuration with the default threshold of 0.3.
1. Effectiveness of latent deliberation
Compared with disabling iteration, full RDD improves Acc2 by 1.57 percentage points, F1 by 1.62 percentage points, and Acc7 by 1.69 percentage points. Correlation increases from 0.770 to 0.825, while MAE decreases from 0.715 to 0.673. These consistent improvements across classification and regression metrics demonstrate the empirical benefit of refining the fused latent representation before prediction.
2. Why unguided perturbation is insufficient
U1 improves Acc2 and F1 over U0, but lowers Acc7 and leaves MAE essentially unchanged. Uniform weighting gives all candidate perturbations equal influence, regardless of their predictive quality. Consequently, it does not preferentially retain directions supported by the predictor. The mixed results indicate that additional stochastic computation alone is insufficient to deliver consistent gains.
3. Why confidence weighting also needs adaptive stopping
With the iteration count held fixed, U2 improves Acc2 by 0.59 percentage points and reduces MAE by 0.032 relative to U1, supporting the usefulness of confidence-guided update directions. However, Acc7 decreases from 45.21% to 44.08%. A fixed iteration budget does not account for differences in sample uncertainty: it may continue updating already-resolved samples or stop before difficult samples are sufficiently refined. This offers a plausible explanation for the uneven gains, without assuming that every confidence-increasing update improves accuracy.
4. Why the full configuration performs better
RDD combines confidence-weighted updates with sample-dependent stopping. Relative to U2, it improves Acc7 by 5.46 percentage points and reduces MAE from 0.684 to 0.673, indicating that controlling when refinement terminates matters alongside the update direction. U3 further shows that adaptive stopping requires an appropriate threshold: the larger threshold of 0.5 permits termination at higher uncertainty and may stop refinement prematurely. With the default threshold of 0.3, RDD improves Acc2 by 1.41 percentage points and reduces MAE by 0.016 relative to U3. Together, these comparisons support the complementary roles of guided exploration and uncertainty-dependent stopping in achieving balanced performance across all five metrics.
Response to Question 1: Limited novelty.
We thank the reviewer for the constructive comments and for recognizing the potential value of information-role-guided fusion and the breadth of our experimental evaluation.
We acknowledge that VAE, cross-attention, KAN, and TCN are established building blocks. Our methodological contribution lies in how role-specific expert modeling, uncertainty-adaptive latent refinement, and feedback-based prior revision are formulated and coupled for instance-specific multimodal sentiment prediction. We will clarify these contributions and distinguish them from the underlying architectural components.
1. Role-specific expert architecture and coordination
The experts have different computational structures and training constraints, rather than serving as interchangeable transformations. The R-expert combines variational compression with cross-modal contrastive alignment to encourage compact, consistent representations. The S-expert performs cross-modal attention followed by nonlinear projection to model interaction-dependent patterns. The U-expert explicitly conditions its suppression gate on the R-expert's representation and reconstructs the original input jointly with the redundancy representation, encouraging complementary modality-specific features. In particular, the U branch is structurally coupled to the R branch rather than being an independently learned private encoder.
MIRR further translates MLLM-derived semantic assessments of redundancy, synergy, and uniqueness into modality-specific expert weights through a learnable router. Thus, the design combines role-specific computational biases with sample-dependent semantic guidance. Relative to statistical decomposition methods such as DMIL, our distinction is the use of MLLM-derived semantic role priors to guide these specialized operators, together with subsequent latent refinement and prior re-evaluation. We do not claim that the individual building blocks are new or that the expert outputs constitute an exact PID decomposition.
2. Uncertainty-adaptive latent refinement
UALD introduces a sample-dependent refinement process after expert fusion. At each iteration, the current latent refinement state is added residually to the fused representation. The model evaluates predictions under Gaussian perturbations, uses their dispersion to estimate predictive uncertainty, and updates the state through a confidence-weighted combination of the perturbations. Refinement terminates when uncertainty falls below a threshold or the iteration budget is exhausted. This design couples uncertainty-based computation allocation with a prediction-guided update direction: uncertainty determines whether further refinement is needed, while candidate confidence determines how the latent state changes. The resulting state supplies an adaptive residual correction to the expert-fused representation.
3. Feedback-based revision of role priors
The feedback mechanism complements latent refinement by revisiting the semantic guidance itself. During training, samples with large prediction errors and high self-reported MLLM confidence are queued for offline re-evaluation. The MLLM receives the training sample's ground-truth sentiment and network prediction as feedback and reconsiders its earlier role assessment. Revised role vectors are cached for subsequent training epochs, allowing the routing guidance to be updated rather than remaining fixed throughout training.
Importantly, this selection criterion triggers re-examination; it does not establish that every selected sample has an incorrect role prior, since downstream processing can also cause prediction errors. The mechanism therefore links downstream training outcomes to the reassessment of upstream semantic guidance, while UALD separately refines the fused latent representation. This feedback is restricted to training and does not use test labels.