acceptodds
Under review as a conference paper at ICLR 2027

EmoPrune: Learning Where to Look for Multimodal Emotion Recognition

Abstract

Multimodal large language models (MLLMs) encode each image into hundreds of visual tokens, which make up most of the input sequence and thus most of the inference cost in multimodal emotion recognition (MER). Token pruning can reduce this cost, but for MER it is difficult to decide which tokens to keep: unlike visual question answering or grounding, an emotion prompt does not indicate where the evidence lies, and emotional cues are often small and not visually prominent. Generic pruning signals, which mainly reflect visual prominence, therefore tend to discard them. We propose , an emotion-aware visual token pruning method that learns which tokens to keep. A lightweight Evidence Head, trained offline from the frozen MLLM's answer loss with only image-level labels, scores visual tokens, and its scores are combined with [CLS] attention to retain both local cues and broader visual context. Unselected tokens are removed before the language model, without predefined regions or additional inference stages. Experiments on six benchmarks and two MLLM backbones show a favorable accuracy–compute trade-off. On LLaVA-1.5-7B, retaining 10% of visual tokens reduces computation from 8.8 to 1.78 TFLOPs per image while keeping accuracy close to full-token inference (% vs. 47.81%).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal large language models, Multimodal emotion recognition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xb61OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual token pruning for image emotion recognition. With the multimodal model frozen, the authors train a lightweight Evidence Head using image-level emotion labels to predict task-relevance scores for visual patches. During training, the head learns through soft masking and an answer-prediction loss, together with budget regularization. At inference, its scores are combined with CLS-to-patch attention to select the top-K tokens for the language model. Experiments evaluate accuracy, macro-F1, and FLOPs across two backbones and six emotion datasets.

Critical strengths
  1. Clear task motivation and a lightweight trainable module. Emotion cues can be subtle and spatially scattered, and generic visual saliency may fail to preserve them. The method trains a lightweight scoring head using only image-level labels, without requiring evidence-region annotations or updating the backbone, resulting in a relatively simple design.

  2. Evaluation across multiple datasets with component ablations. The authors report accuracy, macro-F1, and results from three independent training runs. Comparisons of Head-only, CLS-only, combined scoring, and different loss components provide some support for the task-relevant scoring mechanism.

Critical weaknesses
  1. The full method depends on CLS attention, but its applicability to CLS-free architectures remains unvalidated. At inference, the method combines Evidence Head scores with CLS-to-patch attention using a fixed weight. Both evaluated backbones provide this signal, so the experiments do not establish how the method applies to visual encoders without a CLS token. Although the Head-only ablation shows that the scoring head can operate independently, it does not demonstrate performance on CLS-free architectures. Furthermore, a separate head is trained for each backbone, which should not be interpreted as direct transfer of the same head. The paper should clarify this architectural dependency and the method’s scope of applicability.

  2. Evidence of practical inference efficiency is insufficient. The paper primarily measures efficiency using per-image TFLOPs, without reporting corresponding end-to-end latency, TTFT, or throughput. Reduced computation does not directly establish actual speedup, particularly because the implementation also involves the scoring head, CLS attention extraction, and top-K selection. Measurements covering these operations under the same optimization settings as the baselines are needed to assess the practical deployment benefits.

Questions for authors
  1. How can the full method be applied to visual encoders without a CLS token?

  2. Could you report the actual latency or TTFT of the complete inference pipeline, including visual encoding, CLS attention extraction, the scoring head, token selection, and language model computation?

  3. Could you provide Head-only results from three independent training runs on both backbones to assess the actual benefit of the CLS signal and whether this benefit is stable across different datasets and token retention budgets?

1 comment

We sincerely thank the reviewer for the time and effort spent reviewing our work. If our responses have addressed your concerns, we would be grateful if you could kindly consider raising your score. If any concerns remain, we would be happy to discuss them further and address them.

Reviewer_Lt4hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EMOPRUNE, an emotion-aware visual token pruning method that trains a lightweight Evidence Head on 2,000 emotion-labeled EmoSet images while keeping the multimodal backbone frozen. During training, soft masks scale visual tokens, and the head is optimized using answer cross-entropy and a budget regularizer on the mean mask value. At inference, the method standardizes and combines the head scores with the vision encoder’s [CLS] attention, retaining the top-K original, unweighted tokens as input to the language model. On LLaVA-1.5-7B, retaining 10% of visual tokens yields an average accuracy of 47.59 ± 0.61% across six datasets, compared with 47.81% for full-token inference, while reducing reported computation from 8.8 to 1.78 TFLOPs per image—a reduction of approximately 79.8%.

Critical strengths

The paper addresses a well-defined problem: preserving emotion-relevant information under aggressive visual token pruning. The proposed method is simple and practical, using a lightweight selector with a frozen backbone to remove visual tokens before they enter the language model, without additional LLM inference passes. Its improvements over existing pruning methods at very low token budgets, particularly on LLaVA, demonstrate the value of using limited task-specific supervision to improve the accuracy–computation trade-off.

The evaluation spans six datasets, two backbones, and multiple token retention ratios, with comparisons against FastV, SparseVLM, FasterVLM, VisionZip, PruneSID, and the emotion-oriented SEPM. Reporting both accuracy and macro-F1 provides a more balanced assessment, capturing performance across emotion classes that overall accuracy alone may obscure. The main results for the proposed method are also averaged over three independent training runs with resampled training data and different parameter initializations. This breadth of evaluation, together with the assessment of training variability, strengthens the empirical support for the method.

Critical weaknesses

The methodological contribution appears incremental and is insufficiently distinguished from closely related work. EMOPRUNE combines a lightweight token-scoring head, soft-mask training with task and budget losses, and inference-time fusion with [CLS] attention. However, existing approaches such as VisionSelector already explore learnable visual token selection with a frozen backbone, task-driven optimization, and soft training followed by hard Top-K selection. The paper neither compares against this close alternative nor sufficiently establishes what its specific design contributes beyond adaptation to emotion recognition. This omission makes it difficult to assess the significance of the claimed methodological advance.

The experiments do not adequately separate the benefits of emotion supervision from those of the proposed selection mechanism. EMOPRUNE uses 2,000 emotion-labeled images, whereas the principal generic pruning baselines do not receive comparable task-specific supervision. Although these comparisons demonstrate practical utility, they do not establish whether a similarly trained alternative selector would achieve comparable gains. The improvements are also uneven: while the low-budget LLaVA results are encouraging, at 10% retention on InternVL3, the gains over the strongest baselines are only 0.74 percentage points in accuracy and 0.83 points in macro-F1. The reported three-run standard deviations do not establish the uncertainty of these paired differences. A closely related learnable baseline trained under the same supervision and validation budget, together with uncertainty estimates for the key comparisons, is needed to substantiate the contribution of the proposed design.

Questions for authors
  1. Could the authors report measured end-to-end inference latency, peak GPU memory usage, and Evidence Head training time, together with the relevant hardware, numerical precision, batch size, and decoding settings? The appendix specifies the FLOPs accounting, but these measurements are needed to assess whether the computational savings translate into practical deployment benefits and how much offline training they require.

  2. How consistent are the gains from score fusion and the selected fusion weight across random seeds? Although the main results include three independent training runs, the relevant ablations use a single run, and some differences are small—for example, adding [CLS] fusion improves average accuracy over the head alone by only 0.10 percentage points at 5% retention. Could the authors repeat the key ablations and report their variability to establish whether these design choices provide reproducible benefits?

  3. Could the authors compare EMOPRUNE with a closely related learnable selector, such as VisionSelector, using the same frozen backbone, 2,000 emotion-labeled training images, validation protocol, and token budgets? The current comparisons demonstrate the usefulness of task-specific training, but do not adequately isolate the contribution of the proposed selection mechanism. This controlled comparison would help determine how much of the improvement is attributable to EMOPRUNE’s design beyond the benefit of emotion supervision.

1 comment

We are sincerely grateful for the reviewer's careful reading of our work. If our responses have resolved your concerns, we would greatly appreciate it if you could consider increasing your score. If anything remains unclear, we are more than happy to continue the discussion.

Reviewer_FcXqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EmoPrune proposes a lightweight, task-aware visual token pruning method for multimodal emotion recognition (MER). It trains an Evidence Head using the frozen MLLM’s answer loss and combines its token scores with [CLS] attention to preserve emotion-relevant visual cues. Experiments across six datasets and two MLLM backbones demonstrate favorable accuracy–compute trade-offs, including approximately 4.9× lower computation on LLaVA-1.5-7B while maintaining comparable accuracy.

Critical strengths
  1. The Evidence Head learns token rankings from answer-level supervision without modifying the backbone or requiring additional inference passes, which is a practical approach to preserving subtle emotional cues that generic saliency-based pruning may discard.
  2. The method is evaluated across six datasets and two backbones, with comparisons against multiple pruning baselines. On LLaVA-1.5-7B, retaining 10% of visual tokens yields 47.59% average accuracy versus 47.81% without pruning, while reducing reported computation from 8.8 to 1.78 TFLOPs per image. The macro-F1 results also provide useful evidence beyond overall accuracy.
Critical weaknesses
  1. The training objective does not directly optimize hard token selection at the evaluated budgets. The Evidence Head is trained using soft masks and a budget regularizer targeting 25% retention, whereas inference uses unweighted top-k selection, including budgets of 5% and 10%. Softly scaling all token embeddings is not equivalent to removing most tokens. Although the empirical results are promising, the paper does not establish how well the learned ranking objective aligns with hard selection across these substantially different budgets.
  2. The experiments cover only two MLLM backbones and image emotion recognition. Since the Evidence Head is explicitly trained using emotion-label supervision, it remains unclear whether the approach generalizes to other vision-language tasks or substantially different architectures. Additional experiments are needed to establish whether the method is broadly applicable or primarily benefits emotion-specific inference.
Questions for authors
  1. Have the authors evaluated training the Evidence Head with budget targets matched to the inference budgets, particularly 5% and 10%? How sensitive are the results to the discrepancy between soft-mask training and hard top-kselection?
  2. What are the measured end-to-end latency and throughput of EmoPrune versus full-token inference and the strongest vision-side pruning baselines? Do the reported FLOPs reductions translate into comparable real-world speedups after accounting for the Evidence Head and token-selection overhead?
  3. Could the authors evaluate a leave-one-dataset-out protocol in which neither training nor hyperparameter selection uses images from the held-out benchmark domain? This would help establish whether the learned evidence ranking transfers beyond the domains used for model selection.

Comments on the paper

1 comment

Dear Reviewers,

We’re now past the halfway point of the ICLR discussion period. Please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.