acceptodds
Under review as a conference paper at ICLR 2027

HEAR: Hierarchical Evidence-Aware Reasoning via Text Conditioning for Multimodal Sentiment Analysis

Abstract

Multimodal sentiment analysis (MSA) seeks to infer human sentiment from textual, acoustic, and visual signals. Existing approaches often rely on feature fusion or cross-modal interaction, where globally salient information may overshadow subtle yet sentiment-discriminative cues. This highlights a key challenge: perceptual saliency does not necessarily imply sentiment relevance, particularly when modalities convey discrepant information. To address this challenge, we propose HEAR, a Hierarchical Evidence-Aware Reasoning framework that shifts multimodal sentiment modeling from direct feature fusion toward structured evidence reasoning. HEAR first extracts fine-grained candidate evidence from complementary temporal and spectral views of acoustic and visual features while preserving shared and view-specific information. It then conditions the evidence on textual semantics to distinguish text-aligned from text-deviating evidence, explicitly preserving both cross-modal consistency and discrepancy. Rather than directly aggregating evidence into a single global representation, HEAR progressively reasons from local cues through relational interactions to global sentiment abstractions. Hierarchical evidence is incrementally projected into pseudo-tokens and integrated with textual embeddings for sentiment prediction using a frozen large language model. Experiments on CMU-MOSEI, CH-SIMS-V2, MELD, and CHERMA demonstrate consistent improvements over strong fusion-based and LLM-based baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Sentiment Analysis, Multimodal Learning, Evidence-Aware Reasoning, Cross-Modal Discrepancy, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F4grOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HEAR, a hierarchical evidence-aware framework for multimodal sentiment analysis. It extracts temporal-spectral evidence, models text-aligned and text-deviating information, and progressively organizes evidence through local, relational, and global reasoning before integrating it into a frozen LLM. Experiments on four benchmarks demonstrate improvements over existing methods.

Critical strengths
  1. HEAR explicitly preserves both text-aligned and text-deviating information and introduces hierarchical evidence modeling, providing a useful perspective for addressing cross-modal inconsistencies in sentiment analysis.

  2. Experiments on four datasets and three LLM backbones demonstrate consistent improvements over the matched MSE-Adapter baseline. Extensive ablation studies further support the effectiveness of individual components.

Critical weaknesses
  1. HEAR primarily combines existing techniques, such as cross-attention, shared-private decomposition, and hierarchical aggregation. Its fundamental differences from conventional hierarchical multimodal fusion remain unclear.

  2. Text-aligned and text-deviating evidence are constructed through addition and subtraction of representations. However, these operations do not necessarily guarantee meaningful semantic alignment or discrepancy.

  3. The paper emphasizes subtle sentiment cues and cross-modal discrepancies, but lacks systematic quantitative evaluation on genuinely conflicting multimodal samples. The weak-sentiment metric and qualitative visualizations are insufficient to fully support this claim.

  4. The experiments lack comparisons with recent 2026 methods, such as ATGFB-MFF[1]. This limits the evaluation of HEAR's competitiveness against the latest advances in multimodal sentiment analysis.

Reference:

[1] ATGFB-MFF: Adaptive Text-Guided Fiber Bundle Feature Fusionwith LLMs for Multimodal Sentiment Analysis and EmotionRecognition in Conversations. WWW2026

Questions for authors
  1. What are the key technical differences between HEAR's evidence reasoning and existing hierarchical multimodal fusion methods?

  2. How can the authors verify that the addition/subtraction operations genuinely capture text-aligned and text-deviating evidence?

  3. Can the authors provide quantitative results on samples with conflicting modalities to demonstrate the effectiveness of evidence-aware reasoning?

1 comment
AuthorsOpenReview ↗

Response to Reviewer F4gr

We thank the reviewer for the constructive feedback and recognition of HEAR's strengths. We address all four concerns below with clarifications and new experiments.

Q1. Technical differences from conventional hierarchical multimodal fusion.

R1. We agree that cross-attention, shared-private decomposition, and hierarchical aggregation are established operators. HEAR differs from conventional fusion in three aspects: (i) Evidence Construction vs. Direct Fusion: Rather than collapsing multimodal features into one vector, HEAR isolates temporal-spectral shared/view-specific cues to build text-aligned and text-deviating evidence banks, preserving cross-modal discrepancies relative to text. (ii) Evidence Reasoning: Instead of black-box fusion layers, HEAR’s local, relational, and global stages progressively reason over the evidence bank under task supervision and consistency constraints. (iii) Incremental LLM Conditioning: Rather than projecting only the final vector, HEAR retains all three levels via attention-based residual updates to curb redundancy. Fig. 7 confirms this: incremental projection boosts CH-SIMS-V2 Acc-2 from 83.85% (independent) to 85.20%.

Q2. Interpretation and validity of aligned/deviating evidence.

R2. We agree that addition/subtraction with LayerNorm does not mathematically guarantee semantic separation; rather, they introduce text-conditioned inductive biases: integrating text context (h+h_+) vs. preserving text-conditioned residual evidence (h−h_-). Table 2 confirms their complementarity: Full HEAR achieves 55.60% Acc-7 on CMU-MOSEI (vs. 54.73% Aligned-Only / 54.30% Deviating-Only) and 85.20% Acc-2 on CH-SIMS-V2 (vs. 83.96% / 84.37%).

We additionally train linear probes on frozen pre-Local representations: nonverbal evidence (ee), retrieved text context (cc), additive (h+h_+), and subtractive (h−h_-). Excluding neutral-containing CH-SIMS-V2 unimodal pairs, we form ++, +-, -+, -- groups. Shared Bal. Acc. classifies polarity on ++/--; Four-group Macro Acc. averages accuracy equally across all four groups.

PairRepr.Shared Bal. Acc. (%)↑Four-group Macro Acc. (%)↑
T–Aee/cc57.44 / 69.8450.20 / 54.68
T–Ah+h_+/h−h_-58.14 / 50.4849.26 / 56.49
T–Vee/cc75.74 / 73.2253.21 / 54.49
T–Vh+h_+/h−h_-76.32 / 53.0951.08 / 55.57

Takeaways: h+h_+ consistently outperforms h−h_- on Shared Bal. Acc. (76.32% vs. 53.09% for T–V; 58.14% vs. 50.48% for T–A). Conversely, h−h_- achieves the highest Four-group Macro Acc. across both pairs (55.57% for T–V; 56.49% for T–A). These findings support complementary encoding: h+h_+ is more informative for shared polarity, whereas h−h_- retains cues useful for distinguishing cross-modal polarity patterns.

Q3. Systematic cross-modal conflict evaluation.

R3. Beyond qualitative examples (Fig. 5a), we partitioned the CH-SIMS-V2 test set using independent unimodal labels into Agree (same polarity) and Conflict (opposing polarities) subsets for Text–Audio (n=624n=624 Agree, n=81n=81 Conflict) and Text–Visual (n=506n=506 Agree, n=123n=123 Conflict):

VariantT–A Agree (MAE↓/Acc-2↑)T–A Conflict (MAE↓/Acc-2↑)T–V Agree (MAE↓/Acc-2↑)T–V Conflict (MAE↓/Acc-2↑)
Full HEAR0.1910 / 93.110.5556 / 53.090.1810 / 94.860.4634 / 61.79
w/o text-cond. verification0.2019 / 89.740.6000 / 49.380.1964 / 91.900.5187 / 57.72
w/o hierarchical abstraction0.1968 / 92.150.5679 / 49.380.1976 / 93.870.4764 / 60.16

Findings: Removing text-conditioned verification increases T–V Conflict MAE by 0.0553 (approximately 3.6× the 0.0154 increase on Agree samples) and reduces Conflict Acc-2 by 4.07 pp (61.79% → 57.72%), with T–A Conflict MAE worsening by 0.0444. Removing hierarchical abstraction reduces T–A Conflict Acc-2 by 3.71 pp (53.09% → 49.38%). These results demonstrate the clear contributions of both components under genuine cross-modal polarity disagreement.

Q4. Comparisons with recent 2026 methods.

R4. We expand CMU-MOSEI comparisons to five 2026 SOTA methods (ATGFB-MFF: 3 seeds; HEAR: 5 seeds):

MethodAcc-2↑F1↑Acc-7↑MAE↓Corr↑
MMRest (CVPR'26)83.4283.6554.720.5310.769
MEERA (PR'26)86.0186.1055.720.5070.786
PaP-C (CVPR'26)87.1686.8754.820.4950.802
PaP-L (CVPR'26)87.1786.9156.240.4930.796
DecAlign (ICLR'26)86.4886.0755.020.5430.768
ATGFB-MFF [ChatGLM3-6B] (WWW'26)89.8389.4958.040.5290.803
HEAR [ChatGLM3-6B] (Ours)87.2987.0655.600.4870.809

Observations: HEAR obtains the best regression results (0.487 MAE, 0.809 Corr), surpassing all CVPR'26, ICLR'26, and PR'26 baselines. With the same ChatGLM3-6B backbone, ATGFB-MFF is better at discrete classification, whereas HEAR cuts continuous MAE by 7.9% and raises Corr, showing stronger sensitivity to fine-grained sentiment intensity.

Reviewer_V9AeOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HEAR, a hierarchical evidence-aware reasoning framework for multimodal sentiment analysis. It discovers fine-grained acoustic and visual evidence, interprets it through text-conditioned aligned and deviating evidence, and progressively organizes it from local to global reasoning before integrating it into a frozen LLM. Experiments on four benchmarks show consistent improvements over the compared baselines.

Critical strengths
  1. HEAR provides a coherent framework that integrates temporal-spectral evidence discovery, text-conditioned evidence interpretation, and hierarchical local-to-global reasoning for multimodal sentiment analysis.
  2. The empirical evaluation is relatively comprehensive, covering four benchmarks, three frozen LLM backbones, multiple ablation studies, and five random seeds. The consistent improvements under matched LLM backbones provide useful evidence for the effectiveness of the proposed framework.
Critical weaknesses
  1. The paper motivates fine-grained evidence discovery by arguing that salient patterns may obscure subtle sentiment cues, but applies this only to audio and video while treating text as a reliable semantic reference. The paper does not sufficiently justify why textual representations would not suffer from the same saliency-versus-relevance issue. In particular, it remains unclear how HEAR behaves when the text itself is ambiguous, sentiment-irrelevant, or misleading relative to the non-verbal modalities.
  2. The novelty of the proposed "evidence reasoning" perspective is not sufficiently distinguished from prior MSA methods that already employ hierarchical reasoning [1], language-guided multimodal interaction, or progressive sentiment reasoning.

[1] Xu Q, Wei Y, Wu C, et al. Towards multimodal sentiment analysis via hierarchical correlation modeling with semantic distribution constraints[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(20): 21788-21796.

Questions for authors
  1. The paper motivates fine-grained evidence discovery by arguing that salient patterns may obscure subtle sentiment cues, but applies this only to audio and video while treating text as a reliable semantic reference. The paper does not sufficiently justify why textual representations would not suffer from the same saliency-versus-relevance issue. In particular, it remains unclear how HEAR behaves when the text itself is ambiguous, sentiment-irrelevant, or misleading relative to the non-verbal modalities.
  2. Text is used as a semantic reference, but text relevance, cross-modal agreement, and sentiment relevance are conceptually distinct. The current formulation does not explicitly establish the link among them.
  3. The novelty of the proposed "evidence reasoning" perspective is not sufficiently distinguished from prior MSA methods that already employ hierarchical reasoning [1], language-guided multimodal interaction, or progressive sentiment reasoning.
  4. The comparison with recent SOTA methods is insufficient. The paper would benefit from including more recent state-of-the-art methods from 2026 to better position HEAR against current advances in multimodal sentiment analysis.
  5. The loss design is relatively complex. For example, LdiscL_{\text{disc}} itself combines reconstruction, consistency, and decorrelation losses, while Table 8 only ablates LdiscL_{\text{disc}} as a whole. A finer-grained ablation is needed to clarify the contribution of its internal terms.
5 comments

Since no response is received, I therefore keep my original rating.

AuthorsOpenReview ↗

Response to Reviewer V9Ae

We thank the reviewer for the constructive feedback. We address all five concerns below.

Q1. Modality-specific evidence discovery and behavior under unreliable text.

R1. (1) Why audio/video only? Text representations are contextualized via pretrained language encoders, whereas continuous acoustic/visual sequences contain dense temporal signals for which time–frequency decomposition offers an explicit means of discovering complementary evidence. This is a modality-specific choice, not an assumption that text is error-free. (2) Unreliable text & Pilot Study (n=30n=30): Text serves as contextual guidance rather than ground truth. To evaluate behavior under misleading text, we conducted a pilot test keeping audio, video, and labels fixed while varying captions (Control: neutral "scene transition"; Distractor: strong positive/negative phrase):

Model VariantnControl MAE↓Distractor MAE↓
Full HEAR300.24670.3267
w/o text-conditioned verification300.27330.3533

Under distractors, Full HEAR maintains a lower MAE than the ablated variant (0.3267 vs. 0.3533). Both exhibit the same absolute increase (+0.0800), showing HEAR retains its predictive advantage while remaining sensitive to misleading context.

Q2. Linking textual relevance, cross-modal agreement, and sentiment relevance.

R2. HEAR connects these via successive stages: (1) Textual relevance: Cross-attention (Eq. 7) retrieves text context (cc) for nonverbal cues (ee). (2) Cross-modal interpretation: Additive/subtractive text-conditioned representations provide complementary views of agreement and residuals without strict semantic guarantees. (3) Sentiment relevance: Both branches enter the evidence bank for hierarchical reasoning under task supervision, learning sentiment downstream.

In CH-SIMS-V2 Agree/Conflict evaluation: removing text-conditioned verification increases T–V Conflict MAE by 0.0553 and lowers Conflict Acc-2 by 4.07 pp (61.79% → 57.72%). Removing hierarchical abstraction reduces T–A Conflict Acc-2 by 3.71 pp (53.09% → 49.38%):

VariantT–A Agree (MAE↓/Acc-2↑)T–A Conflict (MAE↓/Acc-2↑)T–V Agree (MAE↓/Acc-2↑)T–V Conflict (MAE↓/Acc-2↑)
Full HEAR0.1910 / 93.110.5556 / 53.090.1810 / 94.860.4634 / 61.79
w/o text-cond. verification0.2019 / 89.740.6000 / 49.380.1964 / 91.900.5187 / 57.72
w/o hierarchical abstraction0.1968 / 92.150.5679 / 49.380.1976 / 93.870.4764 / 60.16

Q3. Distinction from prior hierarchical, language-guided, and progressive methods.

R3. HEAR's novelty lies in its explicit design: (1) Hierarchy vs. HCMNet (AAAI'25): Unlike HCMNet (focusing on distribution consistency via graph/optimal transport), HEAR builds an explicit evidence bank with supervision at Local, Relational, and Global levels. (2) Language Guidance vs. ALMT (EMNLP'23): Unlike ALMT (suppressing inconsistencies), HEAR preserves both aligned and deviating branches as complementary inputs, improving CH-SIMS-V2 Acc-2 from 83.96% to 85.20%. (3) Progression vs. TFPN (2025): Unlike methods iteratively refining a single vector, HEAR applies attention-based residual updates across levels to build level-aware pseudo-tokens for a frozen LLM (boosting CH-SIMS-V2 Acc-2 from 83.85% to 85.20%).

Q4. Comparison with recent 2026 SOTA methods.

R4. We expand CMU-MOSEI comparisons to five 2026 SOTA methods:

MethodAcc-2↑F1↑Acc-7↑MAE↓Corr↑
MMRest (CVPR'26)83.4283.6554.720.5310.769
MEERA (PR'26)86.0186.1055.720.5070.786
PaP-C (CVPR'26)87.1686.8754.820.4950.802
PaP-L (CVPR'26)87.1786.9156.240.4930.796
DecAlign (ICLR'26)86.4886.0755.020.5430.768
ATGFB-MFF [ChatGLM3-6B] (WWW'26)89.8389.4958.040.5290.803
HEAR [ChatGLM3-6B] (Ours)87.2987.0655.600.4870.809
HEAR achieves top regression (0.487 MAE, 0.809 Corr) and cuts MAE by 7.9% over ATGFB-MFF under the same ChatGLM3-6B backbone, showing superior sensitivity to intensity.

Q5. Fine-grained ablation of Ldisc\mathcal{L}_{\text{disc}} internal terms.

R5. Ablating internal terms of Ldisc\mathcal{L}_{\text{disc}} individually under identical settings on CH-SIMS-V2 and MELD:

VariantSIMS-V2 Acc-2↑F1↑Acc2-w↑MAE↓Corr↑MELD Acc↑WF1↑
Full HEAR85.2085.2177.850.2610.75967.2366.31
w/o Ldisc\mathcal{L}_{\text{disc}}83.8183.7576.230.2640.75066.0265.40
w/o Lrec\mathcal{L}_{\text{rec}}83.6283.5176.120.2650.74765.9565.41
w/o Lcon\mathcal{L}_{\text{con}}83.5383.4676.080.2690.74365.9165.34
w/o Ldiv\mathcal{L}_{\text{div}}83.6483.5176.140.2660.74865.9765.67

Removing any term degrades performance, confirming that reconstruction (LrecL_{\text{rec}}), consistency (LconL_{\text{con}}), and decorrelation (LdivL_{\text{div}}) each contribute to the discovery objective.

AuthorsOpenReview ↗

Response to Reviewer V9Ae

Thank you for your comment. We have just submitted our rebuttal addressing your concerns, which was posted after your comment. We kindly invite you to read our response and reconsider your score if you find our clarifications sufficient. We are happy to discuss further if you have any follow-up questions.

Reviewer_htL5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HEAR organizes multimodal sentiment and emotion reasoning into four evidence-centric stages, separating perceptual saliency from sentiment relevance and progressively interpreting, reasoning over, and projecting non-textual cues. Experiments across four datasets and three LLM backbones show promising results, though the paper would benefit from clearer methodological novelty, stronger theoretical motivation, and more recent comparisons.

Critical strengths

The paper has several strengths. It clearly distinguishes perceptual saliency from sentiment relevance and treats cross-modal discrepancies as useful affective evidence. HEAR provides a coherent evidence-aware pipeline, while incremental pseudo-token projection helps reduce redundancy before interacting with a frozen LLM. Evaluation across English and Chinese datasets, three LLM backbones, and matched MSE-Adapter comparisons supports the framework’s generality. Competitive results, ablations, and sensitivity analyses further provide solid empirical support.

Critical weaknesses
  1. Several design choices would benefit from stronger theoretical justification.
    (1) Text-Aligned and Text-Deviating Evidence (Sec. 3.2): The two branches are constructed through additive and subtractive interactions with text-conditioned representations, followed by LayerNorm. Although this formulation is simple and computationally efficient, LayerNorm alone does not ensure that the two branches capture semantically distinct types of evidence. It remains unclear whether the aligned branch truly represents cross-modal agreement and whether the deviating branch preserves complementary affective information rather than irrelevant discrepancies. Further analysis of the semantic characteristics of these two branches would strengthen the motivation for this component.
    (2) Incremental Projection: The residual subtraction in Eq. (16) provides an intuitive heuristic for reducing repeated information across reasoning levels. However, it does not mathematically guarantee orthogonality or complete removal of redundant information. The authors should clarify the intended interpretation of this operation and provide more direct evidence that it effectively reduces cross-level redundancy.

  2. The comparison with recent methods could be more comprehensive. Although the experimental tables include a broad range of established fusion-based, representation-learning, and LLM-based approaches, several recent methods discussed in the related work or appendix, such as DDSE, EMOE, AlignMamba, and DiffuFuse, are not included in the SOTA comparisons. In addition, CMU-MOSI, a widely used MSA benchmark, is absent from the evaluation. Including results on this dataset would further strengthen the empirical assessment.

  3. Figures 5 and 8–15 visualize local, relational, and global evidence processing, but the current visualizations are not fully convincing. In particular, the relational heatmaps exhibit prominent vertical stripe patterns, while the global abstraction heatmaps appear relatively uniform and block-like. From these plots alone, it is difficult to determine how selectively the model distinguishes individual evidence tokens and captures their interactions. The authors should clarify the normalization strategy used for visualization and consider providing quantitative measures of attention diversity or selectivity.

  4. HEAR introduces multiple auxiliary objectives, including reconstruction, consistency, decorrelation, hierarchical, and progressive losses, together with several architectural hyperparameters. Figure 3 shows non-monotonic sensitivity to different loss weights, and the optimal settings vary across evaluation metrics. Although Appendix B.5 provides useful sensitivity analysis, the overall framework appears to require careful tuning. A clearer discussion of the rationale behind these design choices, the robustness of the default settings, and the associated computational cost would improve the practical value of the proposed method.

Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer htL5

We thank the reviewer for the constructive feedback and appreciation of our work. We have conducted additional experiments and quantitative analyses to address all comments below.

Q1. Text-aligned/deviating branches and incremental projection.

R1. (1) Aligned/Deviating Evidence: Addition/subtraction with LayerNorm introduces text-conditioned inductive biases: integrating text context (h+h_+) vs. preserving residual evidence (h−h_-). Table 2 confirms their complementarity: Full HEAR achieves 55.60% Acc-7 on CMU-MOSEI (vs. 54.73% / 54.30% for single branches) and 85.20% Acc-2 on CH-SIMS-V2 (vs. 83.96% / 84.37%).

Linear probing on frozen pre-Local features over non-neutral CH-SIMS-V2 pairs (++, +-, -+, --) confirms complementary encoding: h+h_+ excels in Shared Bal. Acc. (polarity consistency), while h−h_- achieves the highest Four-group Macro Acc. across pairs (55.57% for T–V; 56.49% for T–A).

PairRepr.Shared Bal. Acc. (%)↑Four-group Macro Acc. (%)↑
T–Aee/cc57.44 / 69.8450.20 / 54.68
T–Ah+h_+/h−h_-58.14 / 50.4849.26 / 56.49
T–Vee/cc75.74 / 73.2253.21 / 54.49
T–Vh+h_+/h−h_-76.32 / 53.0951.08 / 55.57

(2) Incremental Projection (Eq. 16): Eq. (16) performs attention-conditioned residualization (cross-attention retrieval followed by subtraction) to suppress redundancy rather than strict mathematical orthogonalization. Beyond Fig. 7 (boosting CH-SIMS-V2 Acc-2 from 83.85% to 85.20%), mean absolute cosine similarity confirms reduced cross-level dependence.

Earlier → LaterBefore↓After↓
Local → Relation0.11780.0878
Local → Global0.12270.1005
Relation → Global0.03840.0203

Q2. Missing comparisons with recent methods and CMU-MOSI evaluation.

R2. (1) Recent SOTA Comparisons: HEAR consistently outperforms DDSE, EMOE, AlignMamba, and DiffuFuse across all available metrics on CMU-MOSEI:

MethodAcc-2 (%)↑F1 (%)↑Acc-7 (%)↑MAE↓Corr↑
DDSE85.4785.3853.810.5370.767
EMOE85.5085.5053.900.530–
AlignMamba86.6086.50–––
DiffuFuse86.7086.4055.400.5210.776
HEAR (Ours)87.2987.0655.600.4870.809

(2) CMU-MOSI Evaluation: On CMU-MOSI under standard splits, HEAR consistently surpasses MSE-Adapter (achieving up to 87.90% Acc-2 and 0.646 MAE with ChatGLM3-6B).

MethodAcc-2 (%)↑F1 (%)↑Acc-7 (%)↑MAE↓Corr↑
MSE-Adapter [Llama2-7B]80.7680.7739.800.8350.736
MSE-Adapter [ChatGLM3-6B]87.6287.3544.900.6780.832
HEAR [Llama2-7B]84.1183.9449.270.6120.838
HEAR [ChatGLM3-6B]87.9087.8950.000.6460.849

Q3. Clarification of visualization patterns and attention selectivity.

R3. (1) Normalization & Visualization Rationale: Heatmaps visualize row-centered attention weights (w−1/Kw-1/K, KK: number of source tokens). Combined with a symmetric color scale, this enhances contrast for above- and below-baseline variations. Vertical stripes reflect stable recurring source positions (shared contextual anchors), while block-like global patterns reflect broad aggregation over evidence groups.

(2) Quantitative Selectivity: Pre-centering attention analysis across datasets confirms non-collapsed, selective distributions:

DatasetClips (nn)Rel. Top-1 (%)Uniform Base (%)Rel. Top-1 LiftWithin-Local LiftWithin-Relation Lift
CMU-MOSEI1296.856.251.096×1.22×1.00×
CH-SIMS-V21,0346.966.251.114×1.24×1.00×
  • Relational Selectivity: Relational attention exceeds the uniform baseline (6.85–6.96% vs. 6.25%), confirming measurable non-uniform concentration corresponding to stable contextual anchors.
  • Global Functional Differentiation: Global queries show distinct selectivity: Within-Local top-1 lift reaches 1.22–1.24× (targeted filtering), whereas Within-Relation lift remains at 1.00× (comprehensive aggregation), validating our hierarchical design.

Q4. Design rationale, robustness of default hyperparameters, and computational cost.

R4. (1) Design rationale: Auxiliary objectives serve distinct roles: reconstruction preserves fidelity, consistency aligns shared cues, decorrelation reduces cross-view redundancy, and hierarchical/progressive losses supervise multi-level reasoning.

(2) Robustness of default settings: HEAR applies a single shared set of loss weights (λdisc=0.005\lambda_{\text{disc}}=0.005, λhier=0.2\lambda_{\text{hier}}=0.2, λprog=0.02\lambda_{\text{prog}}=0.02) and identical architectural hyperparameters across all four datasets without dataset-specific retuning, achieving consistently top performance (Appendix B.5).

(3) Computational cost: Freezing the LLM trains only 0.28%–0.94% of parameters, and compact sequences (L≪LLM contextL \ll \text{LLM context}) keep overhead low: on CH-SIMS-V2 (1× RTX 4090), one epoch takes ~3.1 min (train) and ~52 s (val).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.