acceptodds
Under review as a conference paper at ICLR 2027

Contextual Cue Replay: Training-Free Whole-Image Emotion Recognition via Progressive Re-reading in MLLMs

Abstract

Multimodal large language models (MLLMs) have made substantial progress in visual understanding, yet whole-image emotion recognition remains challenging, as it requires interpreting visual cues in context and weighing their contributions to the overall emotional impression. Emotion-specific fine-tuning incurs training and data-preparation costs when adapting MLLMs to different emotion-category systems, motivating training-free approaches. While existing training-free methods such as visual token selection help retain relevant evidence, such filtering alone does not establish an explicit feedback path for task-conditioned scene information to refine visual representations. In image-first autoregressive MLLMs, causal attention prevents visual positions from directly accessing such information formed at subsequent instruction positions. We propose **Contextual Cue Replay (CCR)**, a training-free framework that enables progressive contextual re-reading within frozen MLLMs. At selected decoder depths, an auxiliary block evaluation constructs contextualized visual states with access to the complete image and image-conditioned states of a generic task instruction. The base and contextualized states then query a shared memory of the current visual representations. Their retrieval difference captures context-induced changes in retrieved evidence and is written back as an incremental correction to the visual residual. Native decoder blocks integrate each correction before the next replay stage, allowing subsequent re-reading to build on visual evidence and instruction states shaped by earlier updates. Experiments across diverse benchmarks and emotion-category systems demonstrate that CCR yields consistent performance improvements without parameter updates or image-specific human guidance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
whole-image emotion recognition, multimodal large language models, training-free inference, vision-language understanding, visual representation refinement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qf5YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CCR, a training-free method that changes the inference of a frozen MLLM for whole-image emotion recognition. Causal attention prevents image tokens from seeing the instruction that comes after them. To work around this, at three decoder depths CCR re-evaluates one block for the image tokens with a mask that lets them see the whole image and a generic instruction. It then reads the visual memory at the next block with both the contextualized and the original query, and adds the difference, scaled by a correction weight, back into the image tokens. On three 7-8B backbones and five dataset settings, CCR beats the strongest baseline (SEPM) by 1.95-3.40 points on average, and ablations cover each component.

Critical strengths
  1. CCR gives consistent gains without training. It beats zero-shot in all 15 backbone-setting pairs and SEPM in 14 of 15, without updating any parameters. This makes it easy to reuse across different sets of emotion categories.

  2. The ablations test each design choice. Table 2 and Fig. 3 compare CCR with meaningful alternatives: token selection, direct context, restricted feedback sources, other correction forms and open-loop replay. CCR wins in nearly every case.

Critical weaknesses
  1. The protocol for choosing hyperparameters is not stated, and there are no significance tests. The paper describes no validation split. Its only sensitivity analyses (number of stages and correction weight, Fig. 3) are run on the evaluation settings, and the layer schedule (about 1/4, 1/2 and 3/4 of depth; exact indices are not reported) is fixed, with no other depths tested. Some test sets are small. The paper does not report split sizes, but all Abstract accuracies are multiples of 1/280, which suggests 280 test images, so one image is about 0.36 points, and no confidence intervals are given. The SEPM numbers for LLaVA and VILA match SEPM's published table, but the Qwen2.5-VL numbers are the authors' own reproduction, which is not described. Together, these make the 2-3 point margin hard to judge.

  2. The evidence for the mechanism is thin, and baselines are missing. Variants that remove one ingredient of the claimed mechanism already recover much of the gain over zero-shot. The image-blind variant keeps the differential retrieval and replay but uses instruction states that are not conditioned on the image. It alone recovers about 60% of the gain on LLaVA and VILA and about half on Qwen (LLaVA: zero-shot 34.04, image-blind instruction 39.06, CCR 41.99). So part of the gain may come from an image-independent signal or a generic shift in label predictions, rather than from image-conditioned re-reading. Apart from the Direct-context ablation, the paper does not compare against the future-aware attention of Pei et al. (2025), on which CCR builds, or against simply placing the instruction before the image. Inference cost is described only qualitatively, and only emotion recognition is tested.

Questions for authors
  1. How were the correction weight, the number of stages and the layer schedule chosen? Please report results with these chosen on a validation split, with bootstrap confidence intervals or paired tests against SEPM.
  2. Please compare against future-aware attention (Pei et al., 2025), prompting with the instruction before the image, and PAI. Please also report CCR's latency and memory overhead.
  3. To rule out a shift in label priors, please report per-class accuracy or confusion matrices. Please also add two controls: a correction of the same norm in a random direction, and an irrelevant instruction in place of Q_ctx.
  4. Your zero-shot and SEPM numbers for LLaVA and VILA match SEPM's published Table 1. Does your zero-shot baseline use SEPM's prompt or CCR's prompt with the added contextual instruction? If the prompts differ, please report zero-shot with CCR's prompt, so that the gain from the prompt is separated from the gain from CCR. How was SEPM reproduced for Qwen2.5-VL?
Reviewer_SUPqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CCR lets visual tokens in a frozen multimodal model read information from the later task instruction. It compares how the original and contextualized visual states attend to the same visual tokens, adds the difference back to the model, and repeats this at several layers. On three models and five emotion-classification settings, it improves average accuracy over SEPM by 1.95–3.40 percentage points.

Critical strengths
  1. CCR uses frozen weights and keeps all visual tokens, while excluding the candidate labels from the information fed back to those tokens (Section 3).
  2. The gains appear across three backbones, and the component ablations and comparison with corrections computed on the unmodified model give useful evidence for the proposed update (Tables 1–2; Section 4.4).
Critical weaknesses
  1. The paper gives fixed correction strengths but not the test splits, sample counts, exact replay layers, or how those strengths were chosen, so I cannot assess whether the reported gains are independent of test-set tuning (Sections 4.1 and 4.4).
  2. The Direct-context baseline is described only as using contextualized states directly, so it is unclear whether it fairly tests the simpler future-aware attention approach on which CCR builds (Table 2).
Questions for authors
  1. Which test splits and replay layers were used, and what data were used to choose the correction strengths?
  2. Does Direct-context replace the visual states or add a correction, and how does its implementation relate to the cited future-aware attention method?
Reviewer_UCiZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Contextual Cue Replay (CCR) for whole-image emotion recognition using MLLMs. For the fact that existing training-free methods mostly extract general visual tokens for relavant scene cues via the underlying causal attention mechanism, which might lead to ambiguous and limited representativeness. This paper proposes to perform depth-wise, context-guided rereading of the visual representations, and compare the outputs (contextual read) to the results of original contextualized queries (base read), and add the differences back to the visual representations. The depth-wise here means repeating this at multiple decoder layers.

Experiments are conducted across three MLLMs and five benchmarks, and the average results are improved over all evaluated settings. Authors also provide reasonable ablation study and analysis.

Critical strengths
  • The motivation, method and underlying mechanism are clearly stated. The focus is worth investigating both for acadamic research and for industrial applications. And the paper is easy to follow overall.

  • Informative component ablations in section 4.3 and 4.4, which help with the understanding.

Critical weaknesses
  • The employed models are out-dated. As the models keep advancing rapidly, author should be sure that the method still works and will present the claimed benefits at least at the time of submission. Some newer models (then qwen2.5-vl) are worth validating.

  • CCR’s incremental contribution remains insufficiently established, as several relevant rereading and visual-feedback or re-reading methods are omitted, which are reasonable and fair comparisons to this paper. The broad principle of using an initial understanding to guide another read predates CCR in both language and vision-language models. Missing baselines and related works include: [1] uses prior LLM context to condition a second visual extraction; [2] uses middle-layer language context to recover visual details and reintegrate them through an intermediate skip connection; and [3] performs training-free visual-memory retracing through the FFN. The first two require trained components, whereas MemVR updates the decoding hidden state rather than CCR's visual-prefix residuals. Representative textual precedents such as [4] and [5] also address causal context limitations through rereading, although repeated input positions are not equivalent to CCR's internal correction.

[1] Wang, et al. "Browse and Concentrate: comprehending multimodal content via prior-llm context fusion." ACL 2024.

[2] Li, et al. "Fine-tuning multimodal llms to follow zero-shot demonstrative instructions." ICLR 2024.

[3] Zou, et al. "Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models." ICML 2025

[4] Xu, et al. "Re-Reading Improves Reasoning in Large Language Models." EMNLP 2024

[5] Jacob, et al. "Repetition improves language model embeddings." ICLR 2025.

Questions for authors
  1. Can authors explain why the selection-only setting of LLaVA produce substaintially high score on WEBEmo-7 compared to other settings and even the full CCR setting in Table 2? And the differences in the series are as large as 17.28, what indeed does the different ablation settings produce when using LLaVA for the WEBEmo-7? Could it be some uncovered issue of the data itslf? Examples rather then pure numeric comparison might help with the understanding.

  2. Can the authors provide runtime and memory measurements to analyse the accuracy-cost trade-off?

  3. Section 4.3 says Direct-context uses contextualized visual states directly, but does not specify whether it replaces, adds to, or interpolates with the original states, or exactly where the update occurs. Please specify the update rule and application location for Direct-context. Does it implement a published future-aware attention variant, or is it a separate ablation?

  4. Not a question, but to mention that figure 3 and 4 are toooooo small. While table 1 and 2 take too much space comapred to these extremely small plots and cases. Layout should be re-designed.

Comments on the paper

1 comment

Dear Reviewers,

We’re now past the halfway point of the ICLR discussion period. Please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.