acceptodds
Under review as a conference paper at ICLR 2027

EmoTrust: Learning Which Modality to Trust for Multimodal Emotion Reasoning under Cross-Modal Conflict

Abstract

Multimodal emotion reasoning aims to infer human affect from visual, acoustic, and linguistic signals, but remains challenging when modalities convey conflicting emotional evidence. Existing methods improve perception and reasoning quality, yet they typically fuse multimodal cues without explicitly modeling which modality should be trusted more when cross-modal emotional conflict arises. To address this challenge, we propose EmoTrust, a framework for multimodal emotion reasoning that explicitly resolves such conflicts through adaptive modality preference learning. EmoTrust begins with multi-view supervised fine-tuning on full and modality-ablated inputs, enabling robust reasoning under varying modality availability. It then estimates cross-modal consistency through cue-grounded modality-level emotion support distributions. To resolve modality disagreement, EmoTrust further introduces conflict-conditioned policy optimization, which compares confidence under full and ablated inputs to estimate modality-specific contributions and reinforce reasoning grounded in the more informative modality. Experiments on several benchmarks show that EmoTrust consistently outperforms strong baselines, especially on low-consistency samples, while learning modality preferences that align well with human judgments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Modality Preference Learning, Cross-modal Emotional Conflict, Multimodal Emotion Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vjPxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EmoTrust, a framework for multimodal emotion reasoning under cross-modal emotional conflict. It combines multi-view supervised fine-tuning, cue-grounded consistency estimation, and conflict-conditioned policy optimization. Modality contributions are estimated from changes in ground-truth label confidence under full and modality-ablated inputs, and are used to reweight counterfactual policy updates. Experiments on MOSI, MOSEI, MELD, and OV-MERD report competitive results, with additional analyses of conflict severity and agreement with human modality-preference judgments.

Critical strengths
  1. The paper addresses a meaningful problem in multimodal emotion reasoning by explicitly considering which modality should guide prediction when visual and acoustic evidence conflicts. The combination of multi-view SFT and conflict-conditioned policy optimization is reasonably motivated and provides a coherent framework for learning sample-dependent modality preferences.

  2. The empirical analysis provides useful support for the proposed training strategy. In addition to improvements across four benchmarks, the ablations include uniform and reversed modality preferences, quality gating, and progressive training variants. The human preference annotations and consistency-stratified evaluations also provide evidence beyond aggregate task performance.

Critical weaknesses
  1. Modality contributions are estimated by comparing the ground-truth label confidence under full and modality-ablated inputs in Eqs. (7)–(9). However, the compared confidence scores are also conditioned on separately generated reasoning traces. Consequently, the observed confidence reduction may reflect changes in reasoning content or quality, rather than the contribution of the removed modality alone. Moreover, since these estimates are produced by the model itself, they may reinforce existing modality biases rather than identify genuinely reliable evidence. Although the RevPref and UniPref ablations demonstrate the effectiveness of preference-based weighting, they do not establish the validity of the underlying contribution estimates. I would like to see more targeted experiments, such as controlling for variations in reasoning traces when estimating modality contributions, or evaluating whether the estimated contributions align with independently assessed modality reliability.

  2. The consistency score in Section 3.3 is derived from cues extracted from ground-truth reasoning annotations, rather than directly from the observed modalities. Furthermore, Appendix D.2 constructs OV-MERDhuman by selecting samples from the lowest consistency-score bins, meaning that the proposed consistency estimator also participates in defining the evaluation subset. Although the human annotation analyses provide useful supporting evidence, the reported improvements on this subset do not independently establish effectiveness on human-identified conflicts. Appendix D.3 also leaves the test-time computation of model-side modality preferences insufficiently specified, particularly regarding the use of ground-truth labels in the confidence comparisons. Can you evaluate on an independently annotated conflict subset? I think this would make the evidence for learning reliable modality preferences more convincing.

Questions for authors
  1. Please clarify the relationship with closely related multimodal emotion reasoning optimization methods, particularly OPPO (ICML 2026) [1]. Where evaluation protocols permit, a matched comparison would help isolate the benefit of contribution-weighted counterfactual optimization over existing modality-aware perception optimization.

  2. In Eq. (7), the confidence scores under full and modality-ablated inputs are conditioned on separately generated reasoning traces. How do the authors distinguish the effect of removing a modality from changes in the reasoning trajectories themselves? Are there controlled comparisons that isolate these two factors?

  3. How are the model-side modality preference weights computed for the human-alignment evaluation in Appendix D.3? In particular, are ground-truth emotion labels used to calculate the confidence differences at test time, and are preference estimates obtained for all evaluation samples under the same protocol?

  4. How does the method distinguish genuine modality reliability from the model's existing modality preference or bias? Is there evidence that confidence-based contribution estimates identify the more informative modality even when the original model relies more heavily on a misleading modality?

  5. Please consider evaluating EmoTrust on a conflict subset identified independently by human annotators, rather than selected using the proposed consistency score. This would help establish whether the improvements extend to externally defined cross-modal emotional conflicts.

The paper addresses a meaningful problem and presents a reasonably motivated framework with encouraging experimental results. The ablations and human preference analyses provide useful evidence for the proposed training strategy. However, the counterfactual confidence estimates do not sufficiently isolate modality contributions from changes in reasoning trajectories, and the conflict-specific evaluation is not fully independent of the proposed consistency measure. These concerns currently limit confidence in the central claim that EmoTrust learns to identify genuinely reliable modalities. I would be willing to raise my score if the authors can adequately address these issues, particularly by clarifying the contribution estimation procedure and providing stronger independent validation of modality preference learning.

Refs:

[1] Omni-Perception Policy Optimization for Multimodal Emotion Reasoning, Zhiyuan Han et al., ICML 2026.

5 comments

We sincerely thank the reviewer for the careful reading and constructive comments. We especially appreciate the questions regarding confidence-based contribution estimation and conflict-set evaluation. These comments prompted us to conduct additional controlled analyses and human-annotated conflict evaluations, which further validate our method and clarify its design. Our point-by-point responses are provided below.

Q1: Please clarify the relationship with closely related multimodal emotion reasoning optimization methods, particularly OPPO (ICML 2026). Where evaluation protocols permit, a matched comparison would help isolate the benefit of contribution-weighted counterfactual optimization over existing modality-aware perception optimization.

A1: Thank you for the question. EmoTrust builds on OPPO’s perception-oriented optimization by adopting the perception reward from Han et al. (2026). However, the two methods have distinct objectives: OPPO improves perception quality, whereas EmoTrust further optimizes modality reliance under cross-modal conflict. We will clarify this relationship in the revised Related Work section.

To isolate this benefit, we evaluate both methods on a conflict subset independently identified by human annotators (see A5). EmoTrust achieves an EW score of 68.76, outperforming OPPO (66.59) under the unified evaluation pipeline.

Q2: In Eq. (7), the confidence scores under full and modality-ablated inputs are conditioned on separately generated reasoning traces. How do the authors distinguish the effect of removing a modality from changes in the reasoning trajectories themselves? Are there controlled comparisons that isolate these two factors?

A2: We design controlled comparisons to isolate the effects of modality removal and changes in reasoning trajectories.

Let Ci(X,r)C_i(X,r) denote the confidence in the ground-truth label under input XX and reasoning trace rr. We decompose the total change Ci(XF,rF)−Ci(X−m,r−m)C_i(X^{F},r^{F})-C_i(X^{-m},r^{-m}) into a trace term Ci(XF,rF)−Ci(XF,r−m)C_i(X^{F},r^{F})-C_i(X^{F},r^{-m}) and an input term Ci(XF,r−m)−Ci(X−m,r−m)C_i(X^{F},r^{-m})-C_i(X^{-m},r^{-m}). The former varies only the trace, while the latter varies only the input. They equal the total change. Following Eqs. (8–9) in the paper, we recompute modality-preference weights under the total, input-only, and trace-only settings. This analysis yields two key findings.

(i) The confidence changes are driven by the modality input, not by trace fluctuation. On OV-MERD, the Spearman correlation between the total and the input-only preference is higher than that between the total and the trace-only preference (Δρ = 0.227, 95% CI [0.069, 0.377]).

(ii) The human-aligned signal resides in the input component. The input-only component is computed with the trace held fixed, so its agreement with human modality preferences cannot be attributed to trajectory variation.

Agreement between model and human modality preferences on OV-MERD (N=330):

Modality preferenceBalanced Acc.Macro-F1Cohen's κWAF
Total0.70550.70530.56190.7823
Input-only0.69030.66200.54050.7661
Trace-only0.40910.38250.08350.5182

Trace-only preference is essentially uninformative about human judgment (κ = 0.08), whereas input-only retains almost all of the total agreement (κ = 0.54 vs. 0.56). EmoTrust's confidence differences therefore mainly reflect modality removal, not trace changes.

Q3: How are the model-side modality preference weights computed for the human-alignment evaluation in Appendix D.3? In particular, are ground-truth emotion labels used to calculate the confidence differences at test time, and are preference estimates obtained for all evaluation samples under the same protocol?

A3: Sorry for not making it clear. The use of ground-truth emotion labels is intentional. In D.1, annotators were shown these labels and asked to assess each modality’s contribution, producing three categories: visual-dominant, acoustic-dominant, and tied. To ensure task consistency, we compute wVw_{V} and wAw_{A} for each evaluation sample using Eqs. (7)–(9), with Eq. (7) measuring confidence against the ground-truth labels. This is an offline diagnostic of whether the learned weights align with human judgments. During inference, EmoTrust neither computes ww nor requires ground-truth labels.

All models are evaluated on the same OV-MERD samples using the same protocol, making the preference estimates directly comparable.

Q4: How does the method distinguish genuine modality reliability from the model's existing modality preference or bias? Is there evidence that confidence-based contribution estimates identify the more informative modality even when the original model relies more heavily on a misleading modality?

A4: Thank you for raising this point. We address this question through a bias-correction analysis. (i) Design. We use MSFT+GRPO as the control because it shares EmoTrust’s backbone and multi-view SFT but lacks counterfactual reweighting. Thus, its decisions could reflect the model’s existing modality preference. On OV-MERD (N=330), we identify cases where this preference conflicts with human judgments: 20 acoustic-dominant clips classified as visual-dominant and 24 visual-dominant clips classified as acoustic-dominant.

(ii) Bias-correction rate. We define the bias-correction rate as the proportion of these samples in which EmoTrust switches to the human-indicated modality. EmoTrust corrects 11/20 acoustic-dominant cases (55.0%) and 17/24 visual-dominant cases (70.8%), i.e. 28/44 (63.6%) overall.

Corrections occur in both directions and exceed the 33% chance level for a three-way decision. Moreover, visual-dominant cases account for only 14.8% of the human preference labels yet have a higher correction rate, arguing against a fixed modality bias or majority-class effect. These results indicate that confidence-based contribution estimates can identify the more informative modality even when the original model favors a misleading one.

We will add this analysis to the revised manuscript.

Q5: Please consider evaluating EmoTrust on a conflict subset identified independently by human annotators, rather than selected using the proposed consistency score. This would help establish whether the improvements extend to externally defined cross-modal emotional conflicts.

A5: Thank you for this suggestion. We evaluate EmoTrust on a conflict subset identified independently by human annotators, without using the proposed consistency score for sample selection.

(i) Independent conflict annotation. Three additional annotators labeled all OV-MERD clips as conflict, non-conflict, or unclear, judging from the emotions conveyed by the visual and audio modalities. They did not see SsimS_{sim} or any model output. Majority voting yielded 78 conflict, 240 non-conflict, and 14 unclear clips (Krippendorff's α = 0.523; mean pairwise agreement = 0.807). The 78 human-identified conflict clips constitute OV-MERD_conflict.

(ii) Evaluation on the human-confirmed conflict set. On OV-MERD_conflict (N = 78), EmoTrust reaches an EW score of 68.76, outperforming the strongest baseline OPPO (66.59) by +2.17 and ranking first among all compared methods (Table below). This result indicates the improvements extend to externally defined cross-modal emotional conflicts.

Table: EW score on OV-MERD_conflict.

ModelEW scoreModelEW score
PandaGPT50.37R1-Omni51.89
LLaMA-VID60.18AffectGPT64.36
Emotion-LLaMA49.25MoSEAR65.98
Qwen2.5-Omni-3B63.78OPPO66.59
Qwen2.5-Omni-7B65.02EmoTrust (ours)68.76

We will include these results in the revised manuscript.

Thank you for the detailed response and additional experiments, which have addressed most of my concerns. In particular, the controlled reasoning-trace analysis and the evaluation on an independently annotated conflict subset provide helpful supporting evidence. One point that still requires clarification is whether the backbone, training data, and optimization budget are matched in the comparison with OPPO. I also encourage the authors to incorporate the key additional experiments, together with their protocols and relevant clarifications, into the main text. This would make the supporting evidence clearer and easier for readers to assess. I would be willing to raise my score if the authors can adequately address these issues.

Reviewer_GiZnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EmoTrust, a framework for multimodal emotion reasoning under cross-modal conflict, combining multi-view supervised fine-tuning, cue-grounded consistency estimation, and conflict-conditioned policy optimization. Experiments on four benchmarks show the effectiveness of the proposed method.

Critical strengths
  • The paper addresses an important challenge in multimodal emotion understanding.
  • The proposed framework is reasonably motivated.
  • The ablation studies provide some evidence supporting the effectiveness of modality-preference learning.
Critical weaknesses
  • The proposed framework largely integrates existing techniques, including modality ablation, semantic consistency estimation, and GRPO-based optimization. The main novelty lies in counterfactual confidence-based advantage reweighting, but its conceptual and technical distinction from existing modality-aware optimization methods is not sufficiently established.
  • Several published methods report stronger performance than EmoTrust. For example, OPPO (ICML 2026) achieves 86.50/84.63 WAF on MOSI/MOSEI and 64.06 HIT on MELD, exceeding EmoTrust's 84.26/83.16 and 62.36, respectively. AlignMamba (CVPR 2025) also reports higher accuracy on MOSI and MOSEI.
    • OPPO: Omni-Perception Policy Optimization for Multimodal Emotion Reasoning, ICML 2026
    • AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment, CVPR 2025
  • The proposed counterfactual contribution scores are computed from ground-truth label confidence under different modality inputs and independently generated reasoning traces. Such confidence differences may reflect changes in reasoning trajectories rather than the intrinsic reliability of individual modalities. The causal interpretation of these scores remains questionable.
  • Modality dominance does not necessarily imply conflicting emotional evidence. The proposed conflict evaluation relies heavily on the authors' own consistency measure rather than independent conflict annotations.
Questions for authors
  • OV-MERDhuman is constructed using the proposed consistency score, while human annotations measure relative modality contributions rather than explicit emotional contradictions. How do the authors establish that the selected samples genuinely exhibit cross-modal conflict rather than simply unequal modality informativeness?
  • Modality contributions are estimated using ground-truth label confidence under independently generated reasoning traces. How can the authors distinguish the effect of removing a modality from changes induced by different reasoning trajectories? What evidence supports interpreting these confidence differences as genuine modality reliability?
6 comments

Thank you for the insightful and constructive feedback. The comments on our distinctions from prior modality-aware methods, conflict identification, and confidence decomposition motivated us to provide clearer comparisons and conduct additional controlled analyses with independently annotated conflict samples. We respond to each point below.

Q1: The conceptual and technical distinction from existing modality-aware optimization methods (e.g. OPPO, AlignMamba) is not sufficiently established.

A1: Thank you for raising this point.

(a) conceptual distinction. AlignMamba aligns feature representations, not decisions. OPPO improves perceptual quality but does not explicitly model modality trust. In contrast, EmoTrust learns which modality to trust under cross-modal conflict.

(b) Technical distinction. AlignMamba is trained in-domain on MOSI/MOSEI labels for binary classification, requiring target-domain supervision and precluding open-vocabulary emotion outputs. Its higher accuracy is therefore not directly comparable to our zero-shot transfer results from MER-Caption in Table 1. OPPO is also evaluated zero-shot, but applies its perception objectives uniformly across samples. EmoTrust instead uses counterfactual confidence drops to weight auxiliary branches, activated only for low-consistency samples. Moreover, OPPO's official release uses Qwen2.5-Omni-7B, whereas EmoTrust uses a 3B backbone with 2.3× fewer parameters.

Using newly collected human conflict annotations, we construct OV-MERD_conflict, on which EmoTrust achieves an EW score of 68.76, outperforming the strongest baseline, OPPO (66.59). This result supports its ability to resolve modality conflicts (see A2).

Q2: The conflict evaluation relies on the proposed consistency measure rather than independent annotations. How do the authors verify that selected samples reflect cross-modal conflict rather than unequal modality informativeness?

A2: Thank you for this suggestion. We address it through independent human annotation, validation of the consistency measure SsimS_{sim}, and re-evaluation on the human-confirmed conflict subset.

(i) Independent conflict annotation. Three additional annotators labeled all OV-MERD clips as conflict, non-conflict, or unclear based on the emotions conveyed by the visual and audio modalities. Majority voting yielded 78 conflict, 240 non-conflict, and 14 unclear clips (Krippendorff's α = 0.523; mean pairwise agreement =0.807).

(ii) SsimS_{sim} agrees with human conflict judgments. Conflict clips have lower SsimS_{sim} than non-conflict clips (mean difference −0.174, Welch t=-7.90, p<0.001).

(iii) Evaluation on the human-confirmed conflict set. On the OV-MERD_conflict (N=78) , EmoTrust reaches 68.76 EW, outperforming the strongest baseline OPPO by +2.17.

Table R1: EW score on OV-MERD_conflict.

ModelEW scoreModelEW score
PandaGPT50.37R1-Omni51.89
LLaMA-VID60.18AffectGPT64.36
Emotion-LLaMA49.25MoSEAR65.98
Qwen2.5-Omni-3B63.78OPPO66.59
Qwen2.5-Omni-7B65.02EmoTrust (ours)68.76

We will include these results in the revised manuscript.

Q3: How do the authors separate modality-removal effects from reasoning-trajectory changes and show that confidence differences reflect modality reliability?

A3: Thank you for this question. We decompose each confidence change into its modality and trajectory components, and find that these confidence differences could reflect modality reliability.

Let Ci(X,r)C_i(X,r) denote the confidence in the ground-truth label under input XX and reasoning trace rr. We decompose the total change Ci(XF,rF)−Ci(X−m,r−m)C_i(X^{F},r^{F})-C_i(X^{-m},r^{-m}) into a trace term Ci(XF,rF)−Ci(XF,r−m)C_i(X^{F},r^{F})-C_i(X^{F},r^{-m}) and an input term Ci(XF,r−m)−Ci(X−m,r−m)C_i(X^{F},r^{-m})-C_i(X^{-m},r^{-m}). These terms isolate trace and modality changes, respectively, and sum exactly to the total change. Following Eq. (8)–(9) in the paper, we compute the modality preference weights under the three settings: total, input-only, and trace-only.

(i) The confidence changes are driven by the modality input, not by trace fluctuation. On OV-MERD, the Spearman correlation between the total and the input-only preference is higher than that between the total and the trace-only preference (Δρ = 0.227, 95% CI [0.069, 0.377]).

(ii) The human-aligned signal resides in the input component (Table R2). The input-only component is computed with the trace held fixed, so it cannot be attributed to trajectory variation; its agreement with human modality preferences shows that it tracks human-perceived modality reliability, not trace artifacts.

Table R2: Agreement between model-side and human modality preferences on OV-MERD.

Modality preferenceBalanced Acc.Macro-F1Cohen's κWAF
Total0.70550.70530.56190.7823
Input-only0.69030.66200.54050.7661
Trace-only0.40910.38250.08350.5182

Trace-only preference barely agrees with human judgment (κ = 0.08), while input-only retains almost all of the total agreement (κ = 0.54 vs. 0.56). Thus, EmoTrust’s confidence differences mainly reflect modality removal, not trace changes.

Thank you for your time and constructive suggestions. We have incorporated the additional experiments and clarifications into the manuscript and uploaded a revised PDF, with all changes highlighted in blue. Specifically:

  • Distinction from existing methods: Clarified the conceptual and technical distinctions between EmoTrust and related methods, particularly OPPO and AlignMamba (Related Work: p. 2, lines 93–95 and 100–102; pp. 7–8, lines 377–394).
  • Independently annotated cross-modal conflicts: Added independent human conflict annotations, validated the proposed consistency measure against these annotations, and reported results on the human-confirmed conflict subset (last columns of Tables 1–3; p. 7, lines 368–373; p. 9, lines 469–473; Appendix D.2).
  • Confidence differences and modality reliability: Added controlled analyses separating modality-removal effects from reasoning-trajectory changes and showing that the confidence differences reflect modality reliability rather than trajectory artifacts (Figure 5 and Table 7; p. 10, lines 507–519; Appendix E).
Reviewer_FeNbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EmoTrust learns modality preferences when visual and acoustic emotion cues disagree. It combines multi-view supervised fine-tuning, cue-based consistency estimation, and policy optimization guided by label-confidence changes after modality removal. Results across four benchmarks and the ablations are encouraging, although the distinction between unequal modality contributions and emotional conflict, and the scope of the human evaluation, need clarification.

Critical strengths

1、 The paper addresses a meaningful problem: deciding which modality to rely on when emotional cues disagree.

2、The improvements are consistent across tasks. Uniform and reversed weighting ablations provide more convincing evidence than comparisons against the base model alone.

3、Human preference annotations and inter-annotator agreement offer useful additional insight into model behavior.

Critical weaknesses

I find the direction worthwhile, and the ablations suggest that the method deserves further investigation. However, the evidence does not yet fully support the central interpretation that the model learns to resolve modality conflicts, and the human-evaluation sample discrepancy needs clarification.

Questions for authors

1、The human annotations measure relative contribution rather than conflict directly. Both modalities could express the same emotion with different strengths. Could independent conflict annotations verify that the low-consistency subset actually contains more conflicting samples?

2、The human-evaluation sample count needs clarification. The dataset contains 332 samples, and the appendix mentions only one exclusion, yet the confusion matrices in Figures 4D/6A total 226. What accounts for the remaining exclusions, and were all models evaluated on the same samples?

3、Confidence changes reflect both modality removal and regenerated reasoning. How do the authors establish that these changes indicate modality reliability? Also, a good full-input response does not guarantee correct auxiliary reasoning; could positive advantages reinforce erroneous auxiliary traces?

3 comments

We sincerely thank the reviewer for the careful assessment and constructive questions. The comments on conflict identification, evaluation coverage, and confidence-based modality estimation prompted us to clarify the sampling protocol and conduct additional controlled analyses with independent human annotations. These results further support EmoTrust’s ability to identify and resolve cross-modal conflicts. Our point-by-point responses are provided below.

Q1: Could independent conflict annotations verify that the low-consistency subset actually contains more conflicting samples? The evidence does not yet show that the model learns to resolve modality conflicts.

A1: Yes. We asked three additional annotators to assess whether the emotions conveyed by the visual and audio modalities conflict in each OV-MERD clip. Majority voting identified 78 conflict, 240 non-conflict, and 14 unclear clips. After ranking the samples by the proposed consistency score SsimS_{sim} and dividing them into five equal-sized bins, the human-annotated conflict rate decreases monotonically from bin 1 to bin 5 (58.7 → 28.9 → 15.6 → 2.2 → 2.2%). The two lowest-consistency bins defining OV-MERD_human have the highest human conflict rates. This confirms that this subset actually contains more conflicting samples.

Using these independent conflict annotations, we construct OV-MERD_conflict (N=78). EmoTrust achieves 68.76 EW, outperforming the strongest baseline, OPPO (ICML 2026; 66.59), supporting its ability to resolve modality conflicts.

Q2: The dataset has 332 samples, but Figures 4D/6A include only 226 despite one reported exclusion. Why were the others excluded, and were all models evaluated on the same samples?

A2: We apologize for the ambiguity. (a) Why the figures total 226. The human-behavior analyses in Appendix D are stratified by SsimS_{sim}, which can be computed only when the ground-truth reasoning provides explicit cues for both modalities (Section 3.3; Appendix A.1). This retains 227 of the 332 clips. For one of these clips, the three annotators assigned three different labels, so no majority-vote label could be determined, leaving 226 clips for analysis. All models in Figure 6 were evaluated on this identical set. Moreover, Appendix D.1 incorrectly reported only one no-majority clip in the full dataset. There are two, one inside and one outside the 227-clip set. We have corrected this.

(b) Analysis on the full benchmark. We recomputed the human–model modality-preference agreement on all clips. Excluding two clips without majority labels leaves 330. The conclusion is unchanged: EmoTrust reaches κ=0.562 against 0.365 for the strongest variant MSFT+GRPO.

Q3.1: Confidence changes reflect both modality removal and regenerated reasoning. How do the authors establish that these changes indicate modality reliability?

A3.1: Thank you for this question. We isolate the two factors directly. Let Ci(X,r)C_i(X,r) denote the confidence in the ground-truth label under input XX and reasoning trace rr (Eq. 7). The confidence change (total:Ci(XF,rF)−Ci(X−m,r−mC_i(X^{F},r^{F})-C_i(X^{-m},r^{-m})) splits exactly into two terms: a trace term, computed by keeping the full input and swapping only the reasoning trace (trace-only: Ci(XF,rF)−Ci(XF,r−mC_i(X^{F},r^{F})-C_i(X^{F},r^{-m})), and an input term, computed by keeping the ablated-branch trace and swapping only the modality (input-only: Ci(XF,r−m)−Ci(X−m,r−mC_i(X^{F},r^{-m})-C_i(X^{-m},r^{-m})). The two terms sum to the original change. Following Eq. (8)–(9) in the paper, we compute the modality preference weights under the three settings (i.e., total, input-only, and trace-only) and find the confidence changes could indicate human-aligned modality reliability:

(i) The confidence changes are driven mainly by the modality input, not by trace fluctuation. On OV-MERD, the Spearman correlation between the total preference and the input-only preference is higher than that between the total preference and the trace-only preference (Δρ = 0.227, 95% CI [0.069, 0.377]).

(ii) The human-aligned signal resides in the input component. Agreement between model-side and human modality preferences on OV-MERD all samples:

Modality preferenceBalanced Acc.Macro-F1Cohen's κWAF
Total0.70550.70530.56190.7823
Input-only0.69030.66200.54050.7661
Trace-only0.40910.38250.08350.5182

Trace-only preference is essentially uninformative about human judgment (κ = 0.08), whereas input-only retains almost all of the total agreement (κ = 0.54 vs. 0.56). The alignment reported in the paper is therefore not an artifact of regenerated reasoning trajectories.

Q3.2: Could positive advantages reinforce erroneous auxiliary traces?

A3.2: To assess this, we measure how often positive advantage lands on auxiliary branches whose traces hallucinate evidence from the removed modality. An auxiliary trace is flagged if any clause matches the removed-modality cues more strongly than the retained-modality cues. Because a single clause triggers the flag, the resulting 14% is an upper bound. Yet this minority does not impact the gains: Auxiliary branches are activated only for low-consistency samples druing training, where harmful reinforcement should have the greatest impact. Instead, this is where they yield the largest improvements over the same model without auxiliary branches: 66.68→69.74 on OV-MERD_human and 65.86→68.76 on the independently annotated OV-MERD_conflict.

Thank you for your time and constructive suggestions. We have incorporated the additional experiments and clarifications into the manuscript and uploaded a revised PDF, with all changes highlighted in blue. Specifically:

  • Independent validation of the low-consistency subset: Added independent human conflict annotations and showed that the human-annotated conflict rate decreases monotonically as the consistency score increases, confirming that the low-consistency subset contains more cross-modal conflicts (p. 9, lines 469–473; p. 20, lines 1045–1052).
  • Clarification of evaluation sample sizes: Explicitly reported the sample size used in each analysis, explained the exclusion criteria, and confirmed that all models were evaluated on identical sample sets (p. 8, line 431; p. 19, lines 1020–1022; p. 20, lines 1046–1049; Table 6; Figure 6; p. 21, lines 1110–1112).
  • Confidence differences and modality reliability: Added controlled analyses separating modality-removal effects from reasoning-trajectory changes and showing that the confidence differences capture human-aligned modality reliability rather than trajectory artifacts (Figure 5 and Table 7; p. 10, lines 507–519; Appendix E).

We would greatly appreciate it if you could review these revisions at your convenience. Thank you again for your careful assessment and valuable feedback, which have substantially improved the clarity and rigor of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.