acceptodds
Under review as a conference paper at ICLR 2027

Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning

Abstract

While multimodal large language models (MLLMs) have demonstrated exceptional capabilities in objective understanding tasks, their performance in affective reasoning still falls significantly short of human standards. We attribute it to a central capability gap: MLLMs are difficult to reliably distinguish semantically proximal emotions based on fine-grained visual evidence, which could be decoupled as two limitations: 1) Insufficient Attribution. The global reasoning paradigm of conventional MLLMs severely dilutes fine-grained emotion cues, where subtle emotional states are usually implicitly encoded, thereby generating emotional misjudgments in complex scenarios. 2) Insufficient Discrimination. Existing methods could only identify regions generally associated with emotions, which fails to distinguish discriminative regions between semantically similar emotions, leading to ambiguous emotion judgements. To overcome these limitations, we present a training-free inference-time optimization framework, named Decoding Affective Nuances (DAN). Specifically, we propose a Hierarchical Emotional Reasoning Chain (HERC) that enhances the insufficient attribution by harmonizing fine-grained scene/object-level cues and performing a soft-gated reasoning. Furthermore, to discriminate between semantically proximal emotions, we design a Contrastive Discriminative Visual Pruning (CDVP), which isolates discriminative visual tokens to reason the final emotion category by computing the absolute discrepancy between the attention distributions of similar emotions. Performances on several benchmarks demonstrate that DAN significantly improves discrimination for affective nuances without consuming additional training resources, especially achieving +10.47% improvements with Qwen3-VL-8B-Instruct on WebEmo25 dataset that contains 25 fine-grained emotion categories.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Emotional Understading, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ueUMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DAN, an inference-time hierarchical prompting and attention-guiding approach to improve MLLM visual (elicited) emotion recognition. A series of hierarchical prompts (starting with simple positive and negative emotion classification) combines polarity and fine-grained label probabilities into a soft-gated distribution. For uncertain or difficult samples, a second module issues one contrastive prompt per candidate emotion and keeps the visual tokens whose attention differs most across candidates. The model then re-selects the label from the masked image.

Critical strengths
  • The method is train-free and is evaluated across multiple MLLM backbones
  • The idea of masked images to guide the MLLM is nice and effective (based on the results)
  • The MLLMs are evaluated on a few elicited emotion datasets
  • Even in the train-free setting, ablations cover the proposed components, and the masked image samples appear reasonable.
Critical weaknesses
  • The evaluation of the method is overall weak; only one external method is compared (SEPM), while the paper itself cites several training-free methods. In addition, general training-free token-pruning and visual-cropping methods, or fine-tuned methods (Emotion-LLaMA, AffectGPT, Emo-R3) as upper references, are also missing from the evaluation.
  • The method makes up to 5 calls to rather large MLLMs (not including the multiple calls for the contrastive samples). Even though the method is train-free, it is quite computationally costly at inference time and most likely very slow. It is questionable whether that many FLOPs are needed for a simple 25-class classification. It would help to show how the FLOPs per additional accuracy justify the choice of such a method over simpler task-specific classifiers, or fewer targeted calls. For reference, Abstract8 is a ~230-image dataset, one image is +0.44 points in accuracy.
  • Another main issue is the limited novelty in the approach. HERC is chained multi-turn prompting, which is pretty established in general VQA tasks. The polarity to fine-grained used is practically a simplified Parrott’s hierarchy. Overall, this element is quite overstated.
  • The main argument of the method is that MLLMs struggle with the task; however, in reality, the task is inherently noisy (even humans disagree).
Questions for authors
  • How does SFT perform, and do the FLOPs-per-accuracy justify the multiple calls at inference time?
  • How does the method compare against other task-specific methods?
  • Does CDVP’s attention come only from prompt tokens, or also from the generated contrastive rationale, and is that rationale fed into the final selection? This part was unclear in the paper.
  • Zero-shot Emotion6 accuracy is identical (55.21) for Qwen3-VL-4B and 8B. Is this a coincidence or a reporting error?
Reviewer_kA9AOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a critical bottleneck in current MLLMs regarding affective understanding: their limited ability to distinguish between semantically proximal emotions across benchmarks, which stems from two specific limitations—insufficient attribution and insufficient discrimination. To address these issues, the authors propose a training-free, inference-time optimization framework named DAN. Specifically, DAN incorporates a HERC to hierarchically mine affective cues and enhance the robustness of preliminary emotion distribution predictions. Furthermore, it introduces CDVP, which leverages attention distribution discrepancies among ambiguous candidate emotions to isolate discriminative visual tokens, prompting the model to make a final re-selection based on the pruned image. Extensive experiments across five emotion benchmarks demonstrate that DAN consistently outperforms several mainstream training-free baselines, validating the superior effectiveness of the proposed framework.

Critical strengths

1.Simple yet Effective Discriminative Pruning (CDVP). The idea of exploiting attention discrepancies across competing emotional candidates to isolate emotion-discriminative regions is intuitive and well-motivated. It offers a clean, training-free solution to resolve ambiguity among semantically similar emotions.

2.Thorough Empirical Analysis and Generalization. The ablation studies are extensive, systematically validating the contributions of multi-level cues, soft-gating mechanisms, layer-wise attention choices, and pruning criteria. Notably, evaluating the framework on general Fine-Grained Image Classification benchmarks convincingly proves its generalizability and reinforces the overall empirical soundness.

Critical weaknesses

1.CDVP relies heavily on the candidate emotions generated by HERC. If the ground-truth category is absent from the candidate set, the subsequent contrastive reasoning cannot correct this error. The reported Recall@k of the ambiguous set on WebEmo25 is only 41.22% with k=2, suggesting that many samples may be excluded before entering the CDVP stage. Although the authors attribute this limitation to annotation ambiguity in fine-grained emotion categories, the single-label formulation itself may be insufficient for representing subtle, overlapping, or ambiguous affective states. This raises concerns about whether CDVP can robustly handle the inherent uncertainty of fine-grained emotion understanding.

2.The Positive/Negative hierarchical decomposition in HERC may oversimplify affective modeling. Although polarity-based soft gating reduces the search space, complex emotional states such as neutral, mixed, and ambivalent emotions cannot always be represented by a binary polarity distinction. Exploring richer affective representations, such as continuous affect dimensions or multi-label emotion modeling, could improve the generalizability of the proposed framework.

Questions for authors

1.Does attention discrepancy truly capture human-perceived emotion-discriminative evidence? Although Tables 5 and 6 demonstrate that discrepancy-based pruning improves classification performance, they do not directly validate whether the retained regions correspond to the visual cues that humans rely on to distinguish semantically similar emotions. More explicit grounding-level evaluation, such as human annotation or region-level verification, would strengthen this claim.

2.Can the authors provide a more comprehensive analysis of the computational overhead introduced by DAN? Although DAN avoids additional training, the CDVP module requires extra operations, including candidate-conditioned inference, attention extraction, discrepancy calculation, and another inference stage after pruning. A detailed breakdown of inference time, memory cost, and the overhead of each component would better demonstrate the practical efficiency of the proposed framework.

3.The baseline comparison could be further strengthened. Since DAN is proposed as a training-free inference-time framework, it would be important to compare it with more recent training-free emotion understanding approaches, such as MM-PEAR-CoT and MERMAID, which are discussed in the related work (A.1). Such comparisons would better demonstrate the relative advantages of DAN over existing inference-time approaches.

Reviewer_tbjXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DAN, a training-free inference framework for visual emotion classification. Its Hierarchical Emotion Reasoning Chain (HERC) first elicits scene-level and object-level cues, then combines coarse polarity scores with fine-grained emotion scores through soft gating. For uncertain examples, Contrastive Discriminative Visual Pruning (CDVP) constructs a small candidate set, obtains candidate-conditioned attention maps, retains visual tokens with large attention discrepancies, and asks the model to select the final emotion.

The experiments cover four MLLMs and five emotion-classification settings, including photographs and abstract paintings. DAN consistently improves over the reported zero-shot, zero-shot-CoT, and SEPM baselines. The paper also includes component ablations, pruning comparisons, parameter sensitivity analyses, candidate coverage, and efficiency measurements.

Critical strengths

Clear problem focus and intuitive design. Distinguishing semantically close emotion categories is a useful target for inference-time improvement. Combining scene/object cues with candidate-specific visual analysis provides a coherent approach.

Consistent improvements across the evaluated models. DAN improves over SEPM in all twenty dataset–backbone combinations in Table 1, suggesting that the results are not limited to one model or dataset.

Relevant component analyses. Tables 5–6 already distinguish candidate restriction, contrastive prompting, and pruning, and compare against random, query-related, and emotion-focused pruning. The appendix also acknowledges that attention discrepancy is a heuristic rather than a guarantee of causal importance. These are useful controls and qualifications.

Critical weaknesses
  1. Candidate restriction imposes a substantial unresolved bottleneck on the principal fine-grained benchmark. Table A.1 reports that, on WebEmo25, the ground-truth label appears among the top k+1k+1 predictions only 41.22% of the time at the default k=2k=2. Increasing kk to 3 raises this to only 42.11%. Since final reselection is restricted to the selected candidate set, CDVP cannot correct errors when the true label is excluded. Moreover, these top-k+1k+1 results need not equal recall of the actual variance-filtered set.

Please report actual candidate-set recall after thresholding, both overall and among triggered examples. A transition analysis should distinguish HERC errors corrected by CDVP, initially correct predictions damaged by CDVP, and errors that are unrecoverable because the true category is absent.

This analysis would also help substantiate the central claim about semantically proximal emotions. Aggregate accuracy does not reveal whether improvements primarily correct polarity mistakes or distinguish categories within the same semantic group. Per-class metrics and within-group versus cross-group error reductions would be more directly aligned with the paper’s motivation.

Finally, the attribution of low candidate recall to subjective annotation noise is not established by the presented experiment. It could also reflect limitations of HERC’s ranking. This explanation should be supported by an annotation audit or presented as a hypothesis.

  1. Clarify percentage improvements. The abstract’s “+10.47%” is a relative improvement over SEPM on WebEmo25, from 22.45% to 24.80%, not a 10.47-percentage-point gain. State the comparator and absolute improvement explicitly.
Questions for authors

Can the confidence-gating rule, Table A.5 trigger rates, Figure A.1 bin accuracies, and HERC accuracies be reconciled under one evaluation protocol?

What is the true-label recall of the actual filtered candidate set, and how often does CDVP correct versus damage HERC predictions?

Comments on the paper

1 comment

Dear Reviewers,

We’re now past the halfway point of the ICLR discussion period. Please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.