- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes EmoTrust, a framework for multimodal emotion reasoning under cross-modal emotional conflict. It combines multi-view supervised fine-tuning, cue-grounded consistency estimation, and conflict-conditioned policy optimization. Modality contributions are estimated from changes in ground-truth label confidence under full and modality-ablated inputs, and are used to reweight counterfactual policy updates. Experiments on MOSI, MOSEI, MELD, and OV-MERD report competitive results, with additional analyses of conflict severity and agreement with human modality-preference judgments.
Critical strengths
-
The paper addresses a meaningful problem in multimodal emotion reasoning by explicitly considering which modality should guide prediction when visual and acoustic evidence conflicts. The combination of multi-view SFT and conflict-conditioned policy optimization is reasonably motivated and provides a coherent framework for learning sample-dependent modality preferences.
-
The empirical analysis provides useful support for the proposed training strategy. In addition to improvements across four benchmarks, the ablations include uniform and reversed modality preferences, quality gating, and progressive training variants. The human preference annotations and consistency-stratified evaluations also provide evidence beyond aggregate task performance.
Critical weaknesses
-
Modality contributions are estimated by comparing the ground-truth label confidence under full and modality-ablated inputs in Eqs. (7)–(9). However, the compared confidence scores are also conditioned on separately generated reasoning traces. Consequently, the observed confidence reduction may reflect changes in reasoning content or quality, rather than the contribution of the removed modality alone. Moreover, since these estimates are produced by the model itself, they may reinforce existing modality biases rather than identify genuinely reliable evidence. Although the RevPref and UniPref ablations demonstrate the effectiveness of preference-based weighting, they do not establish the validity of the underlying contribution estimates. I would like to see more targeted experiments, such as controlling for variations in reasoning traces when estimating modality contributions, or evaluating whether the estimated contributions align with independently assessed modality reliability.
-
The consistency score in Section 3.3 is derived from cues extracted from ground-truth reasoning annotations, rather than directly from the observed modalities. Furthermore, Appendix D.2 constructs OV-MERDhuman by selecting samples from the lowest consistency-score bins, meaning that the proposed consistency estimator also participates in defining the evaluation subset. Although the human annotation analyses provide useful supporting evidence, the reported improvements on this subset do not independently establish effectiveness on human-identified conflicts. Appendix D.3 also leaves the test-time computation of model-side modality preferences insufficiently specified, particularly regarding the use of ground-truth labels in the confidence comparisons. Can you evaluate on an independently annotated conflict subset? I think this would make the evidence for learning reliable modality preferences more convincing.
Questions for authors
-
Please clarify the relationship with closely related multimodal emotion reasoning optimization methods, particularly OPPO (ICML 2026) [1]. Where evaluation protocols permit, a matched comparison would help isolate the benefit of contribution-weighted counterfactual optimization over existing modality-aware perception optimization.
-
In Eq. (7), the confidence scores under full and modality-ablated inputs are conditioned on separately generated reasoning traces. How do the authors distinguish the effect of removing a modality from changes in the reasoning trajectories themselves? Are there controlled comparisons that isolate these two factors?
-
How are the model-side modality preference weights computed for the human-alignment evaluation in Appendix D.3? In particular, are ground-truth emotion labels used to calculate the confidence differences at test time, and are preference estimates obtained for all evaluation samples under the same protocol?
-
How does the method distinguish genuine modality reliability from the model's existing modality preference or bias? Is there evidence that confidence-based contribution estimates identify the more informative modality even when the original model relies more heavily on a misleading modality?
-
Please consider evaluating EmoTrust on a conflict subset identified independently by human annotators, rather than selected using the proposed consistency score. This would help establish whether the improvements extend to externally defined cross-modal emotional conflicts.
The paper addresses a meaningful problem and presents a reasonably motivated framework with encouraging experimental results. The ablations and human preference analyses provide useful evidence for the proposed training strategy. However, the counterfactual confidence estimates do not sufficiently isolate modality contributions from changes in reasoning trajectories, and the conflict-specific evaluation is not fully independent of the proposed consistency measure. These concerns currently limit confidence in the central claim that EmoTrust learns to identify genuinely reliable modalities. I would be willing to raise my score if the authors can adequately address these issues, particularly by clarifying the contribution estimation procedure and providing stronger independent validation of modality preference learning.
Refs:
[1] Omni-Perception Policy Optimization for Multimodal Emotion Reasoning, Zhiyuan Han et al., ICML 2026.
5 comments
Q4: How does the method distinguish genuine modality reliability from the model's existing modality preference or bias? Is there evidence that confidence-based contribution estimates identify the more informative modality even when the original model relies more heavily on a misleading modality?
A4: Thank you for raising this point. We address this question through a bias-correction analysis. (i) Design. We use MSFT+GRPO as the control because it shares EmoTrust’s backbone and multi-view SFT but lacks counterfactual reweighting. Thus, its decisions could reflect the model’s existing modality preference. On OV-MERD (N=330), we identify cases where this preference conflicts with human judgments: 20 acoustic-dominant clips classified as visual-dominant and 24 visual-dominant clips classified as acoustic-dominant.
(ii) Bias-correction rate. We define the bias-correction rate as the proportion of these samples in which EmoTrust switches to the human-indicated modality. EmoTrust corrects 11/20 acoustic-dominant cases (55.0%) and 17/24 visual-dominant cases (70.8%), i.e. 28/44 (63.6%) overall.
Corrections occur in both directions and exceed the 33% chance level for a three-way decision. Moreover, visual-dominant cases account for only 14.8% of the human preference labels yet have a higher correction rate, arguing against a fixed modality bias or majority-class effect. These results indicate that confidence-based contribution estimates can identify the more informative modality even when the original model favors a misleading one.
We will add this analysis to the revised manuscript.
Q5: Please consider evaluating EmoTrust on a conflict subset identified independently by human annotators, rather than selected using the proposed consistency score. This would help establish whether the improvements extend to externally defined cross-modal emotional conflicts.
A5: Thank you for this suggestion. We evaluate EmoTrust on a conflict subset identified independently by human annotators, without using the proposed consistency score for sample selection.
(i) Independent conflict annotation. Three additional annotators labeled all OV-MERD clips as conflict, non-conflict, or unclear, judging from the emotions conveyed by the visual and audio modalities. They did not see or any model output. Majority voting yielded 78 conflict, 240 non-conflict, and 14 unclear clips (Krippendorff's α = 0.523; mean pairwise agreement = 0.807). The 78 human-identified conflict clips constitute OV-MERD_conflict.
(ii) Evaluation on the human-confirmed conflict set. On OV-MERD_conflict (N = 78), EmoTrust reaches an EW score of 68.76, outperforming the strongest baseline OPPO (66.59) by +2.17 and ranking first among all compared methods (Table below). This result indicates the improvements extend to externally defined cross-modal emotional conflicts.
Table: EW score on OV-MERD_conflict.
| Model | EW score | Model | EW score |
|---|---|---|---|
| PandaGPT | 50.37 | R1-Omni | 51.89 |
| LLaMA-VID | 60.18 | AffectGPT | 64.36 |
| Emotion-LLaMA | 49.25 | MoSEAR | 65.98 |
| Qwen2.5-Omni-3B | 63.78 | OPPO | 66.59 |
| Qwen2.5-Omni-7B | 65.02 | EmoTrust (ours) | 68.76 |
We will include these results in the revised manuscript.
Thank you for the detailed response and additional experiments, which have addressed most of my concerns. In particular, the controlled reasoning-trace analysis and the evaluation on an independently annotated conflict subset provide helpful supporting evidence. One point that still requires clarification is whether the backbone, training data, and optimization budget are matched in the comparison with OPPO. I also encourage the authors to incorporate the key additional experiments, together with their protocols and relevant clarifications, into the main text. This would make the supporting evidence clearer and easier for readers to assess. I would be willing to raise my score if the authors can adequately address these issues.
We sincerely thank the reviewer for the careful reading and constructive comments. We especially appreciate the questions regarding confidence-based contribution estimation and conflict-set evaluation. These comments prompted us to conduct additional controlled analyses and human-annotated conflict evaluations, which further validate our method and clarify its design. Our point-by-point responses are provided below.
Q1: Please clarify the relationship with closely related multimodal emotion reasoning optimization methods, particularly OPPO (ICML 2026). Where evaluation protocols permit, a matched comparison would help isolate the benefit of contribution-weighted counterfactual optimization over existing modality-aware perception optimization.
A1: Thank you for the question. EmoTrust builds on OPPO’s perception-oriented optimization by adopting the perception reward from Han et al. (2026). However, the two methods have distinct objectives: OPPO improves perception quality, whereas EmoTrust further optimizes modality reliance under cross-modal conflict. We will clarify this relationship in the revised Related Work section.
To isolate this benefit, we evaluate both methods on a conflict subset independently identified by human annotators (see A5). EmoTrust achieves an EW score of 68.76, outperforming OPPO (66.59) under the unified evaluation pipeline.
Q2: In Eq. (7), the confidence scores under full and modality-ablated inputs are conditioned on separately generated reasoning traces. How do the authors distinguish the effect of removing a modality from changes in the reasoning trajectories themselves? Are there controlled comparisons that isolate these two factors?
A2: We design controlled comparisons to isolate the effects of modality removal and changes in reasoning trajectories.
Let Ci(X,r) denote the confidence in the ground-truth label under input X and reasoning trace r. We decompose the total change Ci(XF,rF)−Ci(X−m,r−m) into a trace term Ci(XF,rF)−Ci(XF,r−m) and an input term Ci(XF,r−m)−Ci(X−m,r−m). The former varies only the trace, while the latter varies only the input. They equal the total change. Following Eqs. (8–9) in the paper, we recompute modality-preference weights under the total, input-only, and trace-only settings. This analysis yields two key findings.
(i) The confidence changes are driven by the modality input, not by trace fluctuation. On OV-MERD, the Spearman correlation between the total and the input-only preference is higher than that between the total and the trace-only preference (Δρ = 0.227, 95% CI [0.069, 0.377]).
(ii) The human-aligned signal resides in the input component. The input-only component is computed with the trace held fixed, so its agreement with human modality preferences cannot be attributed to trajectory variation.
Agreement between model and human modality preferences on OV-MERD (N=330):
Trace-only preference is essentially uninformative about human judgment (κ = 0.08), whereas input-only retains almost all of the total agreement (κ = 0.54 vs. 0.56). EmoTrust's confidence differences therefore mainly reflect modality removal, not trace changes.
Q3: How are the model-side modality preference weights computed for the human-alignment evaluation in Appendix D.3? In particular, are ground-truth emotion labels used to calculate the confidence differences at test time, and are preference estimates obtained for all evaluation samples under the same protocol?
A3: Sorry for not making it clear. The use of ground-truth emotion labels is intentional. In D.1, annotators were shown these labels and asked to assess each modality’s contribution, producing three categories: visual-dominant, acoustic-dominant, and tied. To ensure task consistency, we compute wV and wA for each evaluation sample using Eqs. (7)–(9), with Eq. (7) measuring confidence against the ground-truth labels. This is an offline diagnostic of whether the learned weights align with human judgments. During inference, EmoTrust neither computes w nor requires ground-truth labels.
All models are evaluated on the same OV-MERD samples using the same protocol, making the preference estimates directly comparable.