acceptodds
Under review as a conference paper at ICLR 2027

MACR: Multimodal Affective Contradiction Reasoning for Emotion Understanding

Abstract

Multimodal emotion models can recognize individual affective cues yet misinterpret their combination when fusion suppresses the disagreement that carries meaning. We treat cross-source affective disagreement as a control signal for inference, determining whether evidence should be aggregated directly or should undergo arbitration before fusion. We introduce MACR, the Multimodal Affective Contradiction Reasoner, a recognize-then-arbitrate framework that preserves source-specific evidence and evaluates disagreement before fusion. MACR separates cross-source disagreement from source uncertainty: the former determines when to invoke cross-source reasoning, while the latter informs how much each source contributes. Strong disagreement activates cross-source attention before weighted fusion; low-conflict inputs follow a direct aggregation path. The resulting policy adapts how sources interact, rather than merely adjusting their contributions within a fixed fusion procedure. Across four multimodal benchmarks, MACR outperforms fine-tuned baselines, achieving F1 scores of 89.2 on HFM and 80.9 on MVSA-M. On a 480-sample, manually verified HFM diagnostic split, it improves F1 over holistic fusion by 4.7 points on conflict cases, versus 0.7 on non-conflict cases. These results support a broader principle for multimodal inference: preserve disagreement long enough for it to determine how evidence should be reconciled.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal affective contradiction reasoning, conflict-aware emotion understanding, multimodal large language models, multimodal reasoning, affective computing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EbcKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the misinterpretation of multimodal emotion models under cue combinations. Specifically, it leverages cross-source affective disagreement to determine aggregation or arbitration, and propose MACR with recognize-then-arbitrate paradigm to preserve evidence and evaluate disagreement. Empirically, MACR improves F1 over holistic fusion by 4.7 points on conflict cases, versus 0.7 on non-conflict cases.

Critical strengths
  • Clear motivation: This paper studies multimodal fusion under emotion understanding tasks, which is an important problem. The authors provide clear background and motivation in the introduction.

  • Comprehensive experiments: In Sec.4.1, the authors provide detailed implementation descriptions, including learning rates. And Tab.5 reports results with standard deviation error bars.

Critical weaknesses
  • Lack of latest baselines: Tab.1 compares with CMGCN [ACL'22], SPFVTE [ICMR'24], MODA [ICML'25], while the latest models like A2SI [ACL’26] and SPP-SCL [AAAI’26] are not considered. Facing the growing MLLMs, it’s not convincing enough to just show olds baselines.

  • Lack of citation in Sec.4.1: The authors wrote citations for each dataset under “Datasets”. However, under “Baselines and reporting”, none of comparison methods are cited. As a result, it’s hard to connect the main paper with reference and know which baseline the author refers to. Moreover, proprietary MLLMs like “Claude 3 ops/QWen-VL-Max” only appear in Tab.1 while not described anywhere else.

  • Representation: Fig.1 shows a sparse layout, tighter layout helps to save space. In comparison, Fig.2 and Fig.3 contain dense information but in small font, where readers have to zoom in for details. Likewise, Tab.2 has only three columns, where half-column could be sufficient. Finally, I suggest numbering the equations in the appendix for easier reference.

Questions for authors
  • Could you include more baselines for comparison?

  • Is there any possibility of improving your efficiency?

Reviewer_gEAiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MACR, a multimodal affective contradiction reasoning framework that preserves source-specific features and uses predicted cross-modal disagreement to determine whether additional cross-source attention should be activated. The method further incorporates uncertainty estimation and adaptive source weighting for the final prediction.

The authors evaluate MACR on four multimodal sentiment benchmarks and report improvements over selected fine-tuned baselines. Additional analyses include a manually annotated HFM conflict subset, contradiction scorer diagnostics, component ablations, and counterfactual modality swaps.

Critical strengths

The paper addresses a relevant challenge in multimodal affective understanding: disagreements between modalities may carry meaningful information and should not necessarily be suppressed during fusion. The authors also attempt to analyze performance separately on conflicting and non-conflicting samples.

However, these positive aspects are substantially outweighed by concerns about methodological novelty, evaluation validity, and the reliability of the presented evidence.

Critical weaknesses

1. Limited methodological novelty.

The paper presents contradiction-driven arbitration as a central conceptual and methodological contribution. However, closely related ideas have already been explored in recent multimodal emotion recognition and sentiment analysis research.

For example, EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness (ACL Findings 2026) introduces CHASE, which includes a learned conflict detection router, modality trust estimation, and conflict-conditioned attention steering. These mechanisms substantially overlap with MACR's contradiction scorer, adaptive source weighting, and conditional attention.

Although MACR implements these components differently, the paper does not clearly establish what fundamentally new capability or inference principle its formulation introduces beyond existing conflict-aware routing and adaptive fusion techniques.

Furthermore, the paper overlooks a broader body of recent research on multimodal emotion conflicts and reasoning, including Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning (ACM MM 2025) and AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization (ICLR 2026). Although these works focus on different modality configurations and methodological approaches, they directly investigate cross-modal affective conflicts and how models can better reason over conflicting emotional evidence. The paper neither adequately discusses these studies nor clarifies how MACR differs from or advances beyond them. Such omissions make it difficult to assess the novelty and significance of the proposed framework within the broader research landscape of multimodal emotion reasoning.

2. Concerns about experimental validity and technical presentation.

First, standard preprocessing of MVSA-S and MVSA-M commonly removes image-text sentiment conflicts, yet the paper does not clarify whether such samples were retained. This potentially undermines the suitability of these benchmarks for evaluating contradiction reasoning.

Second, the paper reports only 494 HFM test samples, whereas the commonly used test split contains 2,409 samples. This discrepancy raises concerns about the validity and comparability of the reported results.

Third, the counterfactual modality-swap experiment does not establish improved contradiction reasoning. Replacing images may change the correct labels, while lower prediction flip rates could simply reflect reduced sensitivity to visual evidence.

Finally, the technical illustrations raise serious presentation concerns. Figure 2 appears poorly curated, with inconsistent modality descriptions and unclear architectural details.

Together, these issues substantially weaken the credibility of the experimental evidence and the paper's central claims.

Questions for authors

Q1. Missing related work and unclear novelty.

The discussion of affective conflicts in Sections 2.2 and 2.3 overlooks several highly relevant studies from 2025 and 2026, including:

  • Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning (ACM MM 2025).
  • Improving Audiovisual Emotion Reasoning with Preference Optimization (ICLR 2026).
  • EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness (ACL Findings 2026).

These works investigate multimodal affective disagreement, modality bias, and conflict-aware evidence integration from complementary methodological perspectives. In particular, EmoMM already explore conflict-driven adaptive integration and attention steering.

Could the authors provide a comprehensive discussion of these works and explicitly clarify the methodological novelty of MACR? Beyond the difference in input modalities (e.g., image-text versus audio-video) and the specific implementation of the fusion modules, what fundamentally distinguishes MACR from these existing approaches?

Q2. Validity of the evaluation benchmarks.

Did the authors retain samples with conflicting unimodal sentiment annotations in MVSA-S and MVSA-M? If conventional preprocessing was followed, how can these benchmarks be used to support conclusions about contradiction resolution?

Additionally, why does the paper report only 494 HFM test samples when the commonly used test split contains 2,409 samples? Were all baseline methods evaluated on exactly the same subset?

Q3. Validity of the counterfactual experiments.

Were the ground-truth labels manually revalidated after the image replacements? If not, how are the reported counterfactual error rates defined? How do the authors distinguish improved contradiction reasoning from simply reduced sensitivity to the replaced visual modality?

Q4. Technical accuracy of the illustrations.

Could the authors substantially revise Figure 2 to accurately reflect the modality configurations, operations, and mathematical definitions used in the implementation?

Reviewer_fatJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a failure mode in multimodal emotion understanding in which models may recognize individual affective cues yet misinterpret their combination when fusion suppresses meaningful disagreement between them. It introduces MACR, the Multimodal Affective Contradiction Reasoner, a recognize-then-arbitrate framework that preserves source-specific evidence and uses cross-source affective disagreement as a control signal before fusion. Disagreement determines when sources interact, while source uncertainty informs how much each source contributes. Strong disagreement triggers cross-source attention before weighted fusion, while low-conflict inputs follow direct aggregation. Across four multimodal benchmarks, the paper reports improvements over fine-tuned baselines, with F1 scores of 89.2 on HFM and 80.9 on MVSA-M. On a 480-sample manually verified HFM diagnostic split, the F1 gain over holistic fusion is 4.7 points on conflict cases and 0.7 points on non-conflict cases.

Critical strengths

S1 The inference policy separates conditional interaction from source allocation

MACR separates the decision to invoke cross-source attention from the allocation of source weights. This is useful because cross-source disagreement and source uncertainty describe different properties of the evidence. Two stable sources can disagree, while an uncertain source need not contradict the others. The maximum contradiction score determines whether cross-source attention is activated, while relevance, contradiction, and uncertainty jointly inform source allocation on both inference branches. Low-conflict inputs can therefore follow direct aggregation while retaining adaptive source weights. Computing contradiction in the shared embedding space before applying source-specific projections also avoids measuring the geometric separation introduced by those projections. This ordering is well motivated because identical pre-projection representations can become separated after projection. The resulting recognize-then-arbitrate policy has a clear interpretation and yields a concrete hypothesis about when cross-source attention should help.

S2 Consistent gains on the main benchmarks and a range of ablations

MACR reports higher F1 than the listed task-fine-tuned baselines on all four main benchmarks, with gains of 1.1 to 1.9 points over the fine-tuned MODA variant. The evaluation includes no-scorer and no-attention controls that share the backbone, training data, and optimization budget with full MACR. It also removes source decoupling, dynamic fusion, and conflict resolution individually and replaces the attention module with MODA attention or SEPM VTA. All five component and replacement variants yield lower F1 on both MVSA-M and HFM. Their means and standard deviations over three seeds provide an indication of the consistency of these differences. The benchmark gains and range of ablations support the empirical promise of the complete design and help assess the contribution of its main components.

S3 Main gains are consistent with the motivating failure mode

The diagnostic evaluation compares MACR with the holistic baseline on the 480-sample HFM diagnostic split with manually verified conflict labels. MACR improves F1 by 4.7 points on conflict cases, from 81.2 to 85.9, compared with 0.7 points on non-conflict cases. The results connect the gains of the complete system to the failure mode motivating the paper.

Critical weaknesses

W1 The reported HFM test size differs from the original dataset

Section 4.3 and Table 13 report a 494-sample HFM test set, which Appendix H calls the full test set. However, Table 1 of the original dataset paper by Cai et al. reports 2,409 test samples. The manuscript states that standard splits are used but does not specify a dataset version, filtering procedure, or subsampling rule that explains this discrepancy.

This leaves the test population and the comparability of the reported scores unresolved. Please identify the instances used for each HFM result, explain whether the Table 1 baselines were evaluated on the same population, and clarify how the 480-sample diagnostic split relates to it. A difference in sample counts does not by itself establish an unfair comparison, but its cause and implications need to be documented.

W2 The main benchmarks provide limited evidence for arbitration among multiple sources

Table 6 shows that all four main benchmarks activate only text and vision. Although MACR defines four source slots, its primary evaluation therefore tests disagreement between just two sources. This does not establish how the policy handles multiple simultaneous conflicts, a reliable minority source, or changing combinations of available sources. ASCERTAIN provides a richer source configuration, but its aggregate task results do not isolate these arbitration behaviors.

Please evaluate arbitration in the existing richer-source setting while varying the number and combination of active sources. Results separated by conflict structure would help establish whether the policy can identify reliable evidence when several sources disagree, rather than only improve text-image fusion. This would give the four-source formulation stronger empirical support and clarify the scope of the current two-source findings.

W3 The related-work review and novelty argument could be strengthened

The review of prior disagreement-aware emotion recognition research could be more complete. Since some existing studies already use cross-modal disagreement for emotion-state recognition, the current discussion leaves the specific advance of MACR over this line of work insufficiently clear.

This limits the persuasiveness of the current novelty argument. A fuller review and a clearer comparison of problem formulations and methodological choices would help explain the specific contribution of MACR. Supporting comparisons under comparable conditions, where feasible, would further strengthen this case.

W4 Threshold selection may limit the transferability of conflict decisions

MACR decides whether a sample requires arbitration by thresholding the maximum contradiction score. The threshold is selected from five predefined values on task validation data and set to 0.3. This makes the operational distinction between conflict requiring intervention and disagreement left to direct aggregation depend on a cutoff calibrated to a particular validation distribution. Small score fluctuations near the cutoff can change the routing decision, and the same numerical threshold need not correspond to the same degree of disagreement across datasets. The applicability of this conflict decision rule beyond the calibration setting is therefore uncertain. The authors should consider other reasonable ways of calculating the threshold and compare them with the current selection scheme. Please assess transfer without retuning through conflict-detection quality, trigger rates, and task performance.

Questions for authors

Q1 Why does the reported HFM test set contain 494 rather than the original 2,409 samples? Which instances underlie the main comparisons, and how does the 480-sample diagnostic split relate to that test set?

Q2 How does arbitration behave when more than two sources are active, several sources disagree, or the reliable evidence comes from a minority of sources? Can the existing richer-source setting support evaluations across active-source combinations and conflict structures?

Q3 Could the authors expand the review of disagreement-aware emotion recognition research, clarify the specific advance of MACR over that work, and provide further supporting comparisons where feasible?

Q4 Could the authors use other reasonable ways of calculating the threshold and compare them with the current discrete search? How well do the resulting conflict decisions and task performance transfer across datasets without retuning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.