- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HEAR, a hierarchical evidence-aware framework for multimodal sentiment analysis. It extracts temporal-spectral evidence, models text-aligned and text-deviating information, and progressively organizes evidence through local, relational, and global reasoning before integrating it into a frozen LLM. Experiments on four benchmarks demonstrate improvements over existing methods.
Critical strengths
-
HEAR explicitly preserves both text-aligned and text-deviating information and introduces hierarchical evidence modeling, providing a useful perspective for addressing cross-modal inconsistencies in sentiment analysis.
-
Experiments on four datasets and three LLM backbones demonstrate consistent improvements over the matched MSE-Adapter baseline. Extensive ablation studies further support the effectiveness of individual components.
Critical weaknesses
-
HEAR primarily combines existing techniques, such as cross-attention, shared-private decomposition, and hierarchical aggregation. Its fundamental differences from conventional hierarchical multimodal fusion remain unclear.
-
Text-aligned and text-deviating evidence are constructed through addition and subtraction of representations. However, these operations do not necessarily guarantee meaningful semantic alignment or discrepancy.
-
The paper emphasizes subtle sentiment cues and cross-modal discrepancies, but lacks systematic quantitative evaluation on genuinely conflicting multimodal samples. The weak-sentiment metric and qualitative visualizations are insufficient to fully support this claim.
-
The experiments lack comparisons with recent 2026 methods, such as ATGFB-MFF[1]. This limits the evaluation of HEAR's competitiveness against the latest advances in multimodal sentiment analysis.
Reference:
[1] ATGFB-MFF: Adaptive Text-Guided Fiber Bundle Feature Fusionwith LLMs for Multimodal Sentiment Analysis and EmotionRecognition in Conversations. WWW2026
Questions for authors
-
What are the key technical differences between HEAR's evidence reasoning and existing hierarchical multimodal fusion methods?
-
How can the authors verify that the addition/subtraction operations genuinely capture text-aligned and text-deviating evidence?
-
Can the authors provide quantitative results on samples with conflicting modalities to demonstrate the effectiveness of evidence-aware reasoning?
Response to Reviewer F4gr
We thank the reviewer for the constructive feedback and recognition of HEAR's strengths. We address all four concerns below with clarifications and new experiments.
Q1. Technical differences from conventional hierarchical multimodal fusion.
R1. We agree that cross-attention, shared-private decomposition, and hierarchical aggregation are established operators. HEAR differs from conventional fusion in three aspects: (i) Evidence Construction vs. Direct Fusion: Rather than collapsing multimodal features into one vector, HEAR isolates temporal-spectral shared/view-specific cues to build text-aligned and text-deviating evidence banks, preserving cross-modal discrepancies relative to text. (ii) Evidence Reasoning: Instead of black-box fusion layers, HEAR’s local, relational, and global stages progressively reason over the evidence bank under task supervision and consistency constraints. (iii) Incremental LLM Conditioning: Rather than projecting only the final vector, HEAR retains all three levels via attention-based residual updates to curb redundancy. Fig. 7 confirms this: incremental projection boosts CH-SIMS-V2 Acc-2 from 83.85% (independent) to 85.20%.
Q2. Interpretation and validity of aligned/deviating evidence.
R2. We agree that addition/subtraction with LayerNorm does not mathematically guarantee semantic separation; rather, they introduce text-conditioned inductive biases: integrating text context (h+) vs. preserving text-conditioned residual evidence (h−). Table 2 confirms their complementarity: Full HEAR achieves 55.60% Acc-7 on CMU-MOSEI (vs. 54.73% Aligned-Only / 54.30% Deviating-Only) and 85.20% Acc-2 on CH-SIMS-V2 (vs. 83.96% / 84.37%).
We additionally train linear probes on frozen pre-Local representations: nonverbal evidence (e), retrieved text context (c), additive (h+), and subtractive (h−). Excluding neutral-containing CH-SIMS-V2 unimodal pairs, we form
++,+-,-+,--groups. Shared Bal. Acc. classifies polarity on++/--; Four-group Macro Acc. averages accuracy equally across all four groups.Takeaways: h+ consistently outperforms h− on Shared Bal. Acc. (76.32% vs. 53.09% for T–V; 58.14% vs. 50.48% for T–A). Conversely, h− achieves the highest Four-group Macro Acc. across both pairs (55.57% for T–V; 56.49% for T–A). These findings support complementary encoding: h+ is more informative for shared polarity, whereas h− retains cues useful for distinguishing cross-modal polarity patterns.
Q3. Systematic cross-modal conflict evaluation.
R3. Beyond qualitative examples (Fig. 5a), we partitioned the CH-SIMS-V2 test set using independent unimodal labels into Agree (same polarity) and Conflict (opposing polarities) subsets for Text–Audio (n=624 Agree, n=81 Conflict) and Text–Visual (n=506 Agree, n=123 Conflict):
Findings: Removing text-conditioned verification increases T–V Conflict MAE by 0.0553 (approximately 3.6× the 0.0154 increase on Agree samples) and reduces Conflict Acc-2 by 4.07 pp (61.79% → 57.72%), with T–A Conflict MAE worsening by 0.0444. Removing hierarchical abstraction reduces T–A Conflict Acc-2 by 3.71 pp (53.09% → 49.38%). These results demonstrate the clear contributions of both components under genuine cross-modal polarity disagreement.
Q4. Comparisons with recent 2026 methods.
R4. We expand CMU-MOSEI comparisons to five 2026 SOTA methods (ATGFB-MFF: 3 seeds; HEAR: 5 seeds):
Observations: HEAR obtains the best regression results (0.487 MAE, 0.809 Corr), surpassing all CVPR'26, ICLR'26, and PR'26 baselines. With the same ChatGLM3-6B backbone, ATGFB-MFF is better at discrete classification, whereas HEAR cuts continuous MAE by 7.9% and raises Corr, showing stronger sensitivity to fine-grained sentiment intensity.