acceptodds
Under review as a conference paper at ICLR 2027

AffectReveal: Event-Grounded Emotion Recognition Beyond Visual Appearances

Abstract

Visual emotion recognition commonly assumes that all evidence required for prediction is contained in the observed image or video. Yet the same visible reaction can convey different emotions depending on events beyond the input: tears, for example, may indicate grief or joy. We formulate **Event-Grounded Emotion Recognition (EGER)**, where emotion recognition requires recovering the affect-determining event. We construct **EGER-Bench**, comprising 10,052 videos and 10,734 images across 11 emotions, two source domains, and four visual settings. A study with six annotators shows that event context raises human recognition accuracy from 33.96% to 72.08%, confirming that visual evidence alone is often insufficient. Semantic relevance alone does not solve EGER: a plausible event may imply the wrong emotion if its identity, focal-person role, relationship, or outcome is misinterpreted. We therefore propose **AffectReveal**, a tuning-free framework that first constructs and independently verifies evidence-grounded alternatives over these affect-critical factors. It then cross-checks the recovered event against face-masked in-media facts through bidirectional atomic evidence support, while retaining the original unmasked input for final prediction. Across three downstream models and four input settings, AffectReveal yields average UAR gains of 5.26–10.53 points. For three fine-tunable models, it also enables untuned models to outperform their fine-tuned visual-only counterparts in all 12 accuracy comparisons, without updating downstream parameters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Affective Computing, Event-Grounded Emotion Recognition, Visual Emotion Recognition, Event-Centric Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dwhCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Event-Grounded Emotion Recognition (EGER), in which a person’s emotion depends on an event not visible in the image or video. For this new task, the work introduces a new benchmark of video and static images, and AffectReveal, a training-free pipeline that runs web searches to get the context of a given input to better inform the classification.

Critical strengths
  • The dataset is the main contribution. It covers two domains and is large. Text-removed and text-retained versions share the same frames, and the source-grounded reference context is kept for diagnostics.
  • Human study supports the claims on context.
Critical weaknesses
  • The task novelty is narrower than claimed. Context-aware emotion recognition (EMOTIC, CAER, EmotiCon), emotion-cause extraction, conversational context in MELD, and Social-IQ all frame emotion as depending on context beyond facial appearance. EGER is not that far from these previous tasks.
  • AffectReveal is mostly web-search RAG, and it is well established that this helps MLLMs' reliability. In addition, the paper has no RAG baselines. Every comparison is visual-only vs. +AffectReveal, and search is disabled for the baselines.
  • There is possible label leakage in the RAG; phrases such as “tears of joy” and “forcing a smile” were used to compose the dataset, which may now be revealed in the reverse image search for context.
  • The “outperforms fine-tuned models” claim is unfair. Fine-tuned visual-only models have no access to the missing event, while AffectReveal does. The fair comparison is fine-tuning with retrieved context.
  • The pipeline identifies real people through reverse image search and web lookup. This is a privacy risk that the ethics statement doesn’t address. Licensing for redistributing clips from TV series (Friends and Empresses in the Palace) is also unresolved.
Questions for authors
  • How does AffectReveal compare with a plain RAG baseline under the same search budget across all models and settings, and with AffectAgent?
  • How often does the retrieved context contain emotion words, the seed phrases, or the source page/article of the sample?
  • What is emotion accuracy for Match, Partial, and Mismatch recoveries separately?
  • How many annotators labeled each benchmark sample, what was their agreement, and how were disagreements resolved?
  • How were the fine-tuned baselines trained, and how do they perform when fine-tuned with retrieved context?

I am willing to raise the score if the weaknesses and ethics concerns, are resolved and the over-claimed novelty is addressed is toned down.

5 comments
AuthorsOpenReview ↗

Response to Reviewer dwhC (Part 1/3)

Dear Reviewer dwhC,

Thank you for your careful and constructive review. We appreciate your recognition of EGER-Bench as the main contribution, particularly its scale, two-domain coverage, paired visual settings sharing the same frames, and source-grounded reference contexts for diagnostics. We also appreciate your acknowledgment of the human study supporting the value of event context. We address your concerns below with clarified task boundaries, retrieval-controlled comparisons, and additional fine-tuning and annotation analyses.

W1. Task novelty and difference from context-aware emotion recognition

(1) From interpreting supplied context to actively acquiring event evidence.

EGER extends emotion recognition beyond inference over a fixed observation. EMOTIC, CAER, and EmotiCon exploit context within the supplied media; MELD, emotion-cause extraction, and Social-IQ provide conversations, documents, or videos for contextual reasoning [1–6]. EGER instead makes acquiring missing, sample-specific event evidence part of the recognition problem. A system can search external sources to establish what happened and how the focal person was involved, rather than relying solely on the supplied media or knowledge already encoded in its parameters. This also allows access to evolving event information without requiring model retraining.

 

(2) External acquisition introduces challenges that additional context alone does not solve.

The system must distinguish the relevant event from visually or semantically similar events, establish the focal person’s role and relationship to its outcome, and determine how retrieved facts should interact with the observed reaction. External evidence supplements—not replaces—in-media context: observable facts can help validate the recovered event, while the original expression remains necessary for emotion inference. Incorrect or distracting context can otherwise lead to an incorrect emotional interpretation.

 

(3) EGER-Bench opens this problem to systematic investigation.

The human study demonstrates the missing-evidence bottleneck: event context raises accuracy from 33.96% to 72.08%. Our controlled retrieval comparisons further show that single-search and budget-matched multi-search baselines do not fully resolve it. Evidence integration also matters: adding an in-media description to recovered event context changes UAR from 27.84% to 27.75%, whereas explicitly evaluating their support relation raises it to 29.25%. The contribution is therefore an externally grounded emotion-recognition task and benchmark that expose research questions in evidence acquisition, event disambiguation, verification, and internal–external evidence integration. AffectReveal provides an initial approach to these challenges, rather than treating web search itself as the solution.

 

References

[1] EMOTIC: Emotions in Context Dataset. CVPRW, 2017.

[2] Context-Aware Emotion Recognition Networks. ICCV, 2019.

[3] EmotiCon: Context-Aware Multimodal Emotion Recognition Using Frege’s Principle. CVPR, 2020.

[4] MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. ACL, 2019.

[5] Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts. ACL, 2019.

[6] Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence. CVPR, 2019.

W3/Q2. Discovery phrases and retrieval leakage

Audit. Discovery phrases such as “tears of joy” and “forcing a smile” are not supplied at inference, but may reappear in retrieved webpages. Exact annotation-source URL overlap is 8.17% under VID-V and 9.18% under VID-VS. Explicit emotion descriptions appear in 5.82% and 5.68% of final evidence packages.

 

Controls. We filtered explicit affective descriptions, including seed phrases when present; excluded annotation-source URLs and matching tool results before event recovery; and combined both restrictions. Under both restrictions, UAR remains at 29.28% and 33.00%, retaining gains of 12.17 and 11.15 points over Visual-Only inference (Table 2). Although these controls do not directly assess pretraining exposure to television content, the same pretrained downstream model is used across all conditions, keeping its prior knowledge consistent. The gains are therefore not primarily explained by exact-source reuse or explicit descriptions in final evidence.

Table 2. Leakage controls using Qwen2.5-Omni-7B (%). Each cell reports UAR / Acc. / WAF.

MethodVID-VVID-VS
AffectReveal29.25 / 29.80 / 27.1132.83 / 33.29 / 30.61
Emotion Filtering29.22 / 29.70 / 26.8433.06 / 33.46 / 30.76
Source Exclusion29.86 / 30.33 / 27.5933.89 / 33.46 / 31.32
Both Restrictions29.28 / 29.77 / 26.9133.00 / 33.42 / 30.74
AuthorsOpenReview ↗

Response to Reviewer dwhC (Part 2/3)

W2/Q1. Comparisons with RAG baselines and AffectAgent

(1) Task-specific differences from generic retrieval augmentation.

AffectReveal differs from retrieve-and-append baselines in two ways. First, it verifies competing event interpretations: evidence-anchored alternatives change only one factor—event identity, focal role, relationship, or outcome—and independent searches test the discriminative fact. Second, it checks applicability to the observed sample: bidirectional atomic support compares external facts with non-facial in-media facts. Faces are masked only in this branch to reduce circular use of misleading expressions. The support score and explanation accompany both contexts for emotion prediction.

 

(2) Controlled comparisons and results.

Primary Retrieval uses the initial event; Generic Multi-Search refines queries without factorized alternatives or expression-disentangled cross-verification. Comparisons use the same downstream models, visual inputs, retrieval model, tools, context format, and prediction template. Generic Multi-Search and AffectReveal share the same maximum retrieval budget. Across five models, AffectReveal improves average UAR by 0.75–4.58 points over Primary Retrieval and 0.21–2.06 over Generic Multi-Search (Table 1), supporting additional gains beyond retrieval access or search count. Gains vary across individual settings.

Table 1. Retrieval-baseline comparisons (%). Cells report UAR / Acc. / WAF. Base denotes Visual-Only inference without EGER-Bench fine-tuning. Bold marks the highest UAR per model and setting.

MethodVID-VVID-VSIMG-VIMG-VS
Emotion-LLaMA
Base20.42 / 20.90 / 15.6921.36 / 21.99 / 16.0518.87 / 19.72 / 15.3618.38 / 19.63 / 15.11
+ Primary Retrieval28.15 / 28.80 / 23.1728.47 / 29.14 / 23.5923.27 / 24.01 / 19.2723.35 / 24.01 / 19.39
+ Generic Multi-Search28.90 / 29.57 / 24.3229.02 / 29.70 / 24.2823.56 / 24.41 / 18.8723.63 / 24.44 / 19.02
+ AffectReveal29.10 / 29.77 / 24.5129.13 / 29.80 / 24.3824.47 / 25.16 / 20.0324.52 / 25.25 / 19.92
AffectGPT
Base25.63 / 25.59 / 24.4425.82 / 26.08 / 24.5124.60 / 24.66 / 22.7624.11 / 25.06 / 23.85
+ Primary Retrieval26.01 / 26.31 / 24.8125.82 / 26.08 / 24.5425.34 / 25.64 / 24.0225.70 / 26.01 / 24.35
+ Generic Multi-Search27.20 / 27.85 / 25.5627.92 / 28.49 / 26.2629.68 / 30.15 / 28.2528.15 / 28.65 / 26.39
+ AffectReveal29.48 / 30.00 / 28.0229.93 / 30.46 / 28.5130.70 / 30.25 / 28.7031.07 / 30.87 / 29.44
Qwen2.5-Omni-7B
Base17.11 / 17.94 / 11.1321.85 / 22.52 / 17.0216.66 / 18.04 / 13.7718.95 / 20.37 / 16.99
+ Primary Retrieval28.09 / 29.80 / 25.0930.63 / 31.13 / 27.7725.72 / 26.99 / 25.6327.76 / 29.07 / 27.58
+ Generic Multi-Search28.61 / 29.07 / 26.3231.73 / 32.16 / 29.1826.02 / 27.08 / 25.6327.80 / 28.76 / 27.43
+ AffectReveal29.25 / 29.80 / 27.1132.83 / 33.29 / 30.6126.58 / 27.67 / 25.7928.02 / 29.13 / 27.20
InternVL3.5-8B
Base21.01 / 21.53 / 15.6829.98 / 30.02 / 25.5617.46 / 18.73 / 13.8522.07 / 22.89 / 19.25
+ Primary Retrieval32.68 / 32.62 / 29.0736.68 / 36.45 / 33.2630.51 / 30.68 / 27.8932.86 / 32.80 / 29.97
+ Generic Multi-Search33.34 / 33.32 / 29.9737.76 / 37.41 / 34.8330.51 / 30.87 / 28.1633.27 / 33.23 / 30.87
+ AffectReveal33.42 / 33.36 / 29.9238.48 / 38.21 / 35.3530.49 / 30.87 / 28.2533.34 / 33.26 / 30.70
InternVL3.5-30B-A3B
Base23.20 / 23.39 / 17.9332.23 / 32.23 / 27.7418.93 / 19.32 / 15.4225.44 / 25.71 / 22.21
+ Primary Retrieval34.89 / 34.85 / 31.1739.86 / 39.97 / 37.4632.15 / 32.27 / 30.2634.90 / 35.03 / 33.12
+ Generic Multi-Search35.79 / 35.91 / 32.7239.89 / 40.00 / 37.7032.78 / 32.83 / 30.9534.90 / 35.06 / 33.41
+ AffectReveal36.00 / 36.08 / 32.8541.12 / 41.10 / 38.5032.91 / 32.95 / 31.0835.30 / 35.53 / 33.87

(3) Relation to AffectAgent.

AffectAgent retrieves supportive, confusing, and countering emotion exemplars and optimizes its agents through SFT and MAPPO. AffectReveal instead tests competing factual interpretations of the event underlying the observation, then checks them against non-facial visual evidence, without model training. We could not obtain an accessible implementation for faithful reproduction; adapting AffectAgent’s retrieval and training components to EGER also requires validation. We therefore provide controlled open-web RAG comparisons and do not claim superiority over AffectAgent.

AuthorsOpenReview ↗

Response to Reviewer dwhC (Part 3/3)

W4/Q5. Fine-tuning with retrieved context

Evidence-matched comparison. Original FT models were trained on visual inputs and emotion labels. We added retrieval-aware fine-tuning (AR-FT), which includes automatically recovered context by AffectReveal during training. FT + AffectReveal and AR-FT + AffectReveal use the same retrieval pipeline at inference. AR-FT reaches 32.48% and 34.11% UAR, compared with 36.55% and 36.48% for visual-only FT followed by AffectReveal (Table 3). In this tested configuration, adding context during training does not outperform introducing it at inference. We therefore emphasize the complementary value of fine-tuning and event acquisition, rather than an unconditional superiority claim over fine-tuning.

Table 3. Visual-only and retrieval-aware fine-tuning of Qwen2.5-Omni-7B (%). Each cell reports UAR / Acc. / WAF. FT uses visual-only training inputs, while AR-FT incorporates automatically recovered context during training. Both augmented variants receive AffectReveal evidence at inference.

MethodVID-VVID-VS
FT25.34 / 25.45 / 21.5433.19 / 32.72 / 29.50
FT + AffectReveal36.55 / 36.08 / 35.4436.48 / 36.78 / 33.54
AR-FT + AffectReveal32.48 / 32.43 / 28.8634.11 / 34.65 / 30.67

Q3. Emotion accuracy by recovery quality

We grouped 2,359 VID-V samples with valid Qwen3.5-Plus recovery assessments and evaluated Qwen2.5-Omni-7B + AffectReveal. Accuracy is 34.04%, 30.21%, and 28.12% for Match, Partial, and Mismatch recoveries (Table 4). This shows an association between higher automatically assessed recovery quality and better emotion-recognition performance.

Table 4. Emotion recognition grouped by Qwen3.5-Plus-assessed recovery quality (%).

Recovery QualitySamplesUARAcc.WAF
Match32933.4034.0432.67
Partial68229.0930.2127.74
Mismatch1,34828.1928.1225.60

Q4. Annotation protocol and agreement

Original protocol and independent audit. Original labels were assigned by one annotator per sample using visual materials and source-grounded event context. We subsequently audited 500 randomly sampled instances with eight annotators: two independently labeled each sample, with disagreements adjudicated by a third. Pre-adjudication agreement was 68.60% (343/500), with Fleiss’ (\kappa=0.64). Adjudicated labels agreed with the benchmark on 91.60% (458/500), leaving 42 discrepancies (Table 5). This audit measures annotation consistency; all reported experiments retain the same original benchmark labels.

Table 5. Independent annotation audit. Agreement with benchmark labels uses adjudicated labels; other agreement measures are calculated before adjudication.

SamplesAnnotatorsRaw AgreementFleiss’κ\kappaAgreement with Benchmark
500868.60%0.6491.60%

W5/Ethics concern. Privacy and copyright

Please refer to the corresponding Ethics section in the General Response for details.

Reviewer_nVBsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper frames visual emotion recognition as a mis-specified closed input problem. According to the authors, the visible reaction can correspond to different emotions depending on an event which is not observed in the media. This paper presents this problem as an Event-Grounded Emotion Recognition (EGER), in which the affect-determining event must be recovered from an external information set rather than read off the input.

This paper has 3 different contributions:

EGER-Bench, 10,052 video clips (avg. 6.1 s) and 10,734 images over 11 emotion classes, which are drawn from news or interview footage (57%) and television series (43%). This dataset is also annotated against the main source-grounded reference context and released in four input settings crossing video and image with visible-text-retained. Human annotation with 6 different annotator and showing accuracy increase from 34% to 72% when reference event is provided. They also introduce AffectReveal, a tuning-free pipeline with two different stages. : (i) Affect-Critical Event Recovery (ii) Expression-Disentangled Event Verification.

This paper utilised 4 different models : (i) Qwen3.5-Omni-Plus, (ii) Qwen2.5-Omni-7B, (iii) Emotion-LLaMA, (iv) AffectGPT, with 4 different settings. They also fine-tuned three different models with (i) a reference-event diagnostic, (ii) an event-recovery comparison against primary retrieval, and (iii) generic multi-search.

Critical strengths
  1. The problem formulation looks good to me. Also, the context-aware work expands on the frame while preserving the closed input assumption. Isolating the case where the evidence is outside the sample entirely, and then validating that gap with a controlled between-subjects study where no annotator sees an instance under both conditions (33.96% → 72.08%, in Table 1), is a good way to establish a benchmark. This matters to my assessment because it means the dataset is not merely “harder emotion recognition”, but the difficulty has an identified and manipulable cause.
  2. From my POV, the finding in-media context is useful as a verifier rather than as appended text is a non-obvious result with implications. Table 3(b) is an interesting table in the paper: adding C_int directly moves UAR from 27.84 to 27.75, while using the same information to score compatibility with the recovered event reaches 29.25.
Critical weaknesses
  1. From my POV, the headline gains come from retrieval, not from the proposed framework. Table 2 compares each model against its visual-only self, bundling “retrieve an event at all” with AffectReveal’s contribution. Appendix D separates them: (i) for Qwen2.5-Omni-7B under VID-V, visual-only is 17.11 UAR, (ii) single-shot Primary Retrieval is already 28.09, and (iii) full AffectReveal reaches 29.25. Table 3(b) also does not look good: Affect-Critical Event Recovery alone yields 27.84 UAR, below Primary Retrieval, so the recovery gains in Table 4 (where Match is flat at 13.25→13.95 and mismatch stays at 57%) do not transfer to the primary task. This is consequential because contributions 1 and 3 are framed around affect-critical factorisation being the mechanism, and the authors’ own numbers locate it elsewhere.
  2. Retrieval may be recovering the answer key, and nothing is measured or controlled. Labels were assigned from source webpages and headlines. AffectReveal then searches that same website, so retrieval can surface the annotator used. 3.1’s assurance that K is not sample-specific does not address this, and while emotion labels are stripped from the reference context in 5.3, no equivalent filter is applied to retrieved content. The TV-series half compounds this, since summaries with scene-level affect descriptions sit in every evaluated model’s pretraining data.
Questions for authors
  1. Please add “Primary Retrieval” and, ideally “Generic Multi-Search” as rows in Table 2 for all models × all settings, so that AffectReveal’s contribution is separated from the contribution of retrieval.
  2. Table 3(b) gives C_ext = 27.84 UAR, while Table 7 gives Primary Retrieval = 28.09 under what I take to be the identical configuration (VID-V, Qwen2.5-Omni-7B). Is this comparison valid as I have read it?
  3. What fraction of retrieved documents overlap with the source webpages used during annotation?
  4. What is the per-sample cost of AffectReveal, such as the number of search calls, MLLM calls, and tokens?
  5. Appendix C reports 64.4% agreement between the Qwen judge and human annotators, with the confusion matrix showing systematic optimism. Table 4’s method differences are 2–6 points. How does the paper establish that those differences survive judge bias?
  6. Human performance with reference context is 72.08%, and labels were assigned from that same context. What is the inter-annotator agreement on the benchmark labels themselves, and should 72% be read as the task ceiling?
  7. Qwen2.5-Omni-7B is used inside AffectReveal and is also the downstream model showing the largest gain. Have you checked that the gain is not partly an artefact of the evidence package being written in a form that the model is disposed to consume?
  8. Can you please report the actual numbers rather than the derived averages for 5.2?
  9. On MER2023 (Table 6), Qwen3.5-Omni-Plus UAR decreases (69.66 → 69.60) while AffectGPT UAR is 20.57 against 48.44 accuracy, suggesting degenerate class coverage. Could you explain on both?
5 comments
AuthorsOpenReview ↗

Response to Reviewer nVBs (Part 1/3)

Dear Reviewer nVBs,

Thank you for your careful and constructive review. We appreciate your recognition of the EGER formulation and controlled human study, as well as your observation that in-media context is more useful for event verification than as appended input. We address your concerns below with controlled comparisons, additional human evaluation, and corrected results.

W1/Q1–Q2. Gains beyond retrieval and the two context configurations

Controlled comparisons. Primary Retrieval uses the initial retrieved event; Generic Multi-Search adds unconstrained query refinement. Both use the same retrieval model, tools, and prediction protocol as AffectReveal. Generic Multi-Search and AffectReveal share the same maximum retrieval budget. AffectReveal improves average UAR for all five models, by 0.75–4.58 points over Primary Retrieval and 0.21–2.06 points over Generic Multi-Search (Table 1). For Qwen2.5-Omni-7B under VID-V, UAR is 17.11% with Visual-Only input, 28.09% with Primary Retrieval, 28.61% with Generic Multi-Search, and 29.25% with AffectReveal. Retrieval supplies the dominant gain; the complete framework provides additional, model-dependent improvements.

Table 1. Retrieval-baseline comparisons (%). Each cell reports UAR / Acc. / WAF. Base denotes Visual-Only inference without EGER-Bench fine-tuning. Bold marks the highest UAR within each model and setting. Gains are not universal across individual settings or metrics.

MethodVID-VVID-VSIMG-VIMG-VS
Emotion-LLaMA
Base20.42 / 20.90 / 15.6921.36 / 21.99 / 16.0518.87 / 19.72 / 15.3618.38 / 19.63 / 15.11
+ Primary Retrieval28.15 / 28.80 / 23.1728.47 / 29.14 / 23.5923.27 / 24.01 / 19.2723.35 / 24.01 / 19.39
+ Generic Multi-Search28.90 / 29.57 / 24.3229.02 / 29.70 / 24.2823.56 / 24.41 / 18.8723.63 / 24.44 / 19.02
+ AffectReveal29.10 / 29.77 / 24.5129.13 / 29.80 / 24.3824.47 / 25.16 / 20.0324.52 / 25.25 / 19.92
AffectGPT
Base25.63 / 25.59 / 24.4425.82 / 26.08 / 24.5124.60 / 24.66 / 22.7624.11 / 25.06 / 23.85
+ Primary Retrieval26.01 / 26.31 / 24.8125.82 / 26.08 / 24.5425.34 / 25.64 / 24.0225.70 / 26.01 / 24.35
+ Generic Multi-Search27.20 / 27.85 / 25.5627.92 / 28.49 / 26.2629.68 / 30.15 / 28.2528.15 / 28.65 / 26.39
+ AffectReveal29.48 / 30.00 / 28.0229.93 / 30.46 / 28.5130.70 / 30.25 / 28.7031.07 / 30.87 / 29.44
Qwen2.5-Omni-7B
Base17.11 / 17.94 / 11.1321.85 / 22.52 / 17.0216.66 / 18.04 / 13.7718.95 / 20.37 / 16.99
+ Primary Retrieval28.09 / 29.80 / 25.0930.63 / 31.13 / 27.7725.72 / 26.99 / 25.6327.76 / 29.07 / 27.58
+ Generic Multi-Search28.61 / 29.07 / 26.3231.73 / 32.16 / 29.1826.02 / 27.08 / 25.6327.80 / 28.76 / 27.43
+ AffectReveal29.25 / 29.80 / 27.1132.83 / 33.29 / 30.6126.58 / 27.67 / 25.7928.02 / 29.13 / 27.20
InternVL3.5-8B
Base21.01 / 21.53 / 15.6829.98 / 30.02 / 25.5617.46 / 18.73 / 13.8522.07 / 22.89 / 19.25
+ Primary Retrieval32.68 / 32.62 / 29.0736.68 / 36.45 / 33.2630.51 / 30.68 / 27.8932.86 / 32.80 / 29.97
+ Generic Multi-Search33.34 / 33.32 / 29.9737.76 / 37.41 / 34.8330.51 / 30.87 / 28.1633.27 / 33.23 / 30.87
+ AffectReveal33.42 / 33.36 / 29.9238.48 / 38.21 / 35.3530.49 / 30.87 / 28.2533.34 / 33.26 / 30.70
InternVL3.5-30B-A3B
Base23.20 / 23.39 / 17.9332.23 / 32.23 / 27.7418.93 / 19.32 / 15.4225.44 / 25.71 / 22.21
+ Primary Retrieval34.89 / 34.85 / 31.1739.86 / 39.97 / 37.4632.15 / 32.27 / 30.2634.90 / 35.03 / 33.12
+ Generic Multi-Search35.79 / 35.91 / 32.7239.89 / 40.00 / 37.7032.78 / 32.83 / 30.9534.90 / 35.06 / 33.41
+ AffectReveal36.00 / 36.08 / 32.8541.12 / 41.10 / 38.5032.91 / 32.95 / 31.0835.30 / 35.53 / 33.87

The 27.84% versus 28.09% comparison (Q2). Both configurations omit cross-verification, but use different contexts: consolidated CextC_{\mathrm{ext}} from structured recovery versus the initial event from Primary Retrieval. Structured recovery alone is slightly worse here; adding verification raises UAR to 29.25%. We therefore attribute the downstream improvement to the complete evidence package, not factorization alone.

AuthorsOpenReview ↗

Response to Reviewer nVBs (Part 2/3)

W2/Q3. Source overlap and emotion-description leakage

Audit. Sample-level annotation-source URL overlap is 8.17% under VID-V and 9.18% under VID-VS. A Qwen3.5-Flash audit finds explicit emotion descriptions in 5.82% and 5.68% of final evidence packages.

 

Controls. We removed explicit emotion descriptions while preserving event facts, excluded annotation-source URLs during retrieval and filtered matching tool results, and applied both restrictions together. Under both restrictions, UAR remains at 29.28% and 33.00%, retaining gains of 12.17 and 11.15 points over Visual-Only inference (Table 2). Although these controls do not directly assess pretraining exposure to television content, using the same pretrained downstream model with and without AffectReveal keeps such exposure constant across the comparison. The gains are therefore not primarily explained by exact-source reuse or explicit emotion descriptions in downstream evidence.

Table 2. Leakage controls using Qwen2.5-Omni-7B (%). Each cell reports UAR / Acc. / WAF.

MethodVID-VVID-VS
AffectReveal29.25 / 29.80 / 27.1132.83 / 33.29 / 30.61
Emotion Filtering29.22 / 29.70 / 26.8433.06 / 33.46 / 30.76
Source Exclusion29.86 / 30.33 / 27.5933.89 / 33.46 / 31.32
Both Restrictions29.28 / 29.77 / 26.9133.00 / 33.42 / 30.74

Q4. Per-sample cost

AffectReveal averages 7.17 tool calls—including search and webpage extraction—and 7.78 MLLM calls: 4.78 recovery API calls plus three local verification calls. Final emotion prediction is excluded. The token count and dollar cost below cover Qwen3.5 API usage, including tool operations, but exclude local inference costs.

Table 3. Average computational cost per sample.

MetricAverage
External tool calls7.17
MLLM calls7.78
API token consumption107,858
API cost$0.015

Q5. Recovery quality assessed independently of the original judge

Human evaluation. Eight annotators assessed the three methods on the same 2,851 successfully processed samples, out of 3,010 VID-V test samples. Each recovery received two independent assessments and third-person adjudication. Method identities were concealed, outputs randomized, and references and criteria held constant. Human-assessed Match + Partial recovery increases from 55.31% for Primary Retrieval and 57.14% for Generic Multi-Search to 59.10% for AffectReveal. Relationship correctness improves by 5.83 points over Generic Multi-Search (Table 4). Human assessment is now the primary evidence; automatic evaluations are supplementary.

Table 4. Event recovery quality under VID-V (%). Sample IDs are identical across methods within each evaluator. API evaluation restrictions result in different sample coverage across evaluators; therefore, absolute scores are not directly comparable.

MethodEvent IdentityFocal RoleRelationshipEvent OutcomeMatchPartialMismatch
Human — 2,851 samples
Primary Retrieval23.4045.2133.2921.9920.4834.8344.69
Generic Multi-Search24.8047.8136.5824.4820.8336.3042.86
AffectReveal27.1850.0942.4127.8121.9237.1840.90
Qwen3.5-Plus — 2,359 samples
Primary Retrieval16.3536.6723.3312.2513.2521.5565.20
Generic Multi-Search15.4438.9025.5314.3212.9625.9261.12
AffectReveal16.5240.6431.0615.0313.9528.9157.14
GPT-5 — 2,589 samples
Primary Retrieval10.5128.0522.067.939.3719.8770.76
Generic Multi-Search10.5928.3723.148.229.8623.1866.96
AffectReveal10.6628.6224.108.5110.2726.4563.27

Q6. Label agreement and the 72.08% human result

Label audit. The original benchmark used one annotator per sample. We independently audited 500 samples with eight annotators, using two independent labels per sample and third-person adjudication. Pre-adjudication agreement was 68.60%, with Fleiss’ (\kappa=0.64). Adjudicated labels agreed with the benchmark on 91.60% (458/500).

Table 5. Independent annotation audit.

SamplesAnnotatorsRaw AgreementFleiss’κ\kappaAdjudicated Agreement with Benchmark
500868.60%0.6491.60%

All experiments retain the original labels; the audit documents consistency and residual uncertainty rather than selectively relabeling the audited subset. 72.08% is a diagnostic result, not a ceiling. The separate six-participant, 480-sample study measures the benefit of reference context. Incomplete reference descriptions and differences in interpretation limit its use as an upper bound.

AuthorsOpenReview ↗

Response to Reviewer nVBs (Part 3/3)

Q7. Evidence-generator and predictor coupling

We fixed Qwen2.5-Omni-7B as the predictor and replaced the model performing both in-media extraction and atomic verification. External context, prompts, visual inputs, evidence format, and prediction settings remained unchanged. UAR reaches 30.59% with Qwen3.5-Flash and 29.81% with GPT-5, versus 29.25% originally (Table 6). The gains therefore do not require the evidence producer and predictor to be the same model.

Table 6. Verification-model replacement under VID-V (%), with Qwen2.5-Omni-7B fixed as the predictor.

Verification MLLMUARAcc.WAF
Qwen2.5-Omni-7B29.2529.8027.11
Qwen3.5-Flash30.5930.9628.47
GPT-529.8130.2727.39

Q8. Reporting individual results

Table 1 reports the absolute results for each model and input setting. We will also revise Section 5.2 to explicitly report the per-setting improvements instead of relying solely on averages.

Q9. MER2023 results and evaluation corrections

Qwen’s UAR decrease. Surprise predictions fall from six to four correct among 14 samples, reducing recall from 42.86% to 28.57%. Equal class weighting gives this change a 2.38-point UAR contribution, largely offset by improvements elsewhere. Overall UAR decreases by 0.06 points, while accuracy and WAF improve (Tables 7–8). One API-rejected Angry sample is now counted as incorrect rather than excluded. Corrected evaluation includes all 834 samples and all 183 Angry instances.

 

AffectGPT correction. The erroneous 11-class configuration affected prediction parsing, filtering, and metric aggregation. We re-scored saved predictions using MER2023’s six classes, retaining all 834 samples and counting unparseable or out-of-set outputs as incorrect. Corrected results improve across all three metrics with AffectReveal; the manuscript table has been updated.

Table 7. MER2023 performance on all 834 test samples (%).

MethodUARAcc.WAF
Qwen3.5-Omni-Plus69.5978.5479.15
Qwen3.5-Omni-Plus + AffectReveal69.5381.4181.56
AffectGPT44.6747.7248.47
AffectGPT + AffectReveal47.9351.4454.67

Table 8. MER2023 per-class recall on all 834 test samples (%).

EmotionSamplesQwen3.5-Omni-PlusQwen+AffectRevealAffectGPTAffectGPT+AffectReveal
Happy16991.7293.4939.0541.42
Sad25787.5589.8826.4637.74
Angry18369.4073.7782.5177.05
Worried4555.5655.5662.2253.33
Surprise1442.8628.577.1421.43
Neutral16670.4875.9050.6056.63
UAR—69.5969.5344.6747.93

Ethics concern: Privacy, annotator protection, and copyright

Please refer to the corresponding Ethics section in the General Response for details.

Reviewer_SXsuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

the paper formulates Event-Grounded Emotion Recognition (EGER), in which emotion of a focal person depends on an event that is not directly visible in the image or video. Without this context, visual emotion recognition may not be accurate since it will just miss important context and models may give wrong answer (e.g when person is crying depending on context it might be "tears of joy" or "sadness"). the paper introduces EGER-Bench with more than 10k videos and images with 11 emotions from two source domains. It proposes AffectReveal, a tuning free pipeline which helps to recover external event from visual clues and web search, verifies affect-critical alternatives, cross checks recovered events against non-facial in-media facts at the atomic level. It improves UAR by 5.26 - 10.53 points over the 4 downstream models.

Critical strengths
  1. new task and substantial benchmark. the paper builds more than 10k videos and images across 11 emotions, two domains, and four input settings. Human study is clean demonstration that visual input is often insufficient and relevant event that is outside video/image content matters for better visual emotion recognition.
  2. within-model gains are consistent and oracle diagnostic is well designed. adding AffectReveal improves performance for every downstream model on every setting (+5.26 to +10.53), and reference-context diagnostic (with AffectReeal closing ~70% of the gap) is strong test the event is the bottleneck. This makes paper's premise credible even where the mechanism is not.
Critical weaknesses
  1. "verified event" mechanism is not established. the claim is that AffectReveal recovers the affect-determining event and verifies it, so the prediction is grounded in the right cause. However, Its own evidence points the other way. Table 4 labels 57.14% of recovered events as mismatches, 28.91% as partial, and only 13.95% as full matches. So in most cases the event given to the model is not the correct one. that diagnostic is itself produced by an automatic judge that agrees with human annotators only 64.4% of the time, so the number is too noisy to support fine-grained claims.
  2. retrieval branch may be reading the answer. authors describe that benchmark comes from news/interview clips with source pages, and their proposed AffectReveal searches the web using person and text. However, the paper does not show emotion-descriptive language is filtered from retrieved context. if retrieved event/context states emotion reaction, gains might be just a lookup effect rather than even-grounded reasoning.
  3. baseline experiments are not comprehensive. authors only report 4 models whereas different open source models with different capabilities exist already for example InternVL3_5 ranging from 8 bln to 30 bln params https://huggingface.co/OpenGVLab/InternVL3_5-30B-A3B
Questions for authors
  1. does retrieved info contain explicit emotion-descriptive words ?
  2. why only 11 emotion labels ? what is the reasoning to choose only this emotion set?
  3. could authors also compare their methods with different models with different scale ?
3 comments
AuthorsOpenReview ↗

Response to Reviewer SXsu (Part 1/2)

Dear Reviewer SXsu,

Thank you for your careful and constructive review. We appreciate your recognition of the substantial EGER-Bench, the importance of external event context demonstrated by the human study, and the consistent gains across models and input settings. We also appreciate your positive assessment of the reference-context diagnostic. We address your concerns below with additional human evaluation, leakage controls, and model comparisons.

W1. Event recovery quality and verification reliability

Human evaluation. We now use human assessments as the primary evidence for event recovery quality. Eight annotators evaluated all three recovery methods on the same 2,851 VID-V samples successfully processed by AffectReveal, out of 3,010 candidate samples. Each recovered event received two independent assessments, with disagreements adjudicated by a third annotator. Outputs were randomized, method identities were concealed, and assessments used the same source-grounded references and criteria.

Results. As shown in Table 1, the human-assessed Match + Partial rate increases from 55.3% for Primary Retrieval and 57.1% for budget-matched Generic Multi-Search to 59.1% for AffectReveal. All four affect-critical factors improve; relationship correctness shows the largest gain over Primary Retrieval, from 33.29% to 42.41%. Complete Match rates are 20.48%, 20.83%, and 21.92%, respectively. Qwen3.5-Plus and GPT-5 evaluations are supplementary. Their valid sample sets differ because some API-based evaluations were restricted; absolute scores across evaluators are therefore not directly comparable.

Table 1. Event recovery quality under VID-V (%). Comparisons use identical sample IDs across methods within each evaluator.

MethodEvent IdentityFocal RoleRelationshipEvent OutcomeMatchPartialMismatch
Human — 2,851 samples
Primary Retrieval23.4045.2133.2921.9920.4834.8344.69
Generic Multi-Search24.8047.8136.5824.4820.8336.3042.86
AffectReveal27.1850.0942.4127.8121.9237.1840.90
Qwen3.5-Plus — 2,359 samples
Primary Retrieval16.3536.6723.3312.2513.2521.5565.20
Generic Multi-Search15.4438.9025.5314.3212.9625.9261.12
AffectReveal16.5240.6431.0615.0313.9528.9157.14
GPT-5 — 2,589 samples
Primary Retrieval10.5128.0522.067.939.3719.8770.76
Generic Multi-Search10.5928.3723.148.229.8623.1866.96
AffectReveal10.6628.6224.108.5110.2726.4563.27

What verification establishes. Verification checks evidential support for event details and compatibility with non-facial visual observations; it does not certify complete event correctness. Human evaluation still identifies 40.90% of AffectReveal recoveries as mismatches. Its contribution to downstream prediction is assessed separately: providing the explicit support relation raises UAR from 27.75% to 29.25%, compared with supplying the same external and in-media contexts without that relation (Table 3(b) of the main paper).

 

Downstream association. Table 2 reports an exploratory analysis using Qwen3.5-Plus recovery assessments. Among its 2,359 valid evaluations, Qwen2.5-Omni-7B + AffectReveal achieves accuracies of 34.04%, 30.21%, and 28.12% for Match, Partial, and Mismatch recoveries. This shows an association between higher automatically assessed recovery quality and better emotion-recognition performance.

Table 2. Emotion recognition performance (%) of Qwen2.5-Omni-7B + AffectReveal on VID-V, grouped by Qwen3.5-Plus-assessed recovery quality. Only samples with valid recovery evaluations are included.

Recovery QualitySamplesUARAcc.WAF
Match32933.4034.0432.67
Partial68229.0930.2127.74
Mismatch1,34828.1928.1225.60
AuthorsOpenReview ↗

Response to Reviewer SXsu (Part 2/2)

W2/Q1. Explicit emotion descriptions and annotation-source overlap

Evidence audit. Explicit emotion descriptions appear in 5.82% of final evidence packages under VID-V and 5.68% under VID-VS, according to a Qwen3.5-Flash audit. Sample-level exact URL overlap with annotation-source webpages is 8.17% and 9.18%, respectively.

Controlled evaluation. We tested three conditions:

  1. Emotion Filtering: remove explicit emotion descriptions from the final evidence while preserving factual event information.
  2. Source Exclusion: exclude annotation-source URLs during retrieval and filter matching tool results.
  3. Both Restrictions: apply both controls.

With both restrictions, UAR is 29.28% under VID-V and 33.00% under VID-VS, compared with 29.25% and 32.83% for the original pipeline (Table 3). The gains over Visual-Only inference remain 12.17 and 11.15 percentage points. These controls indicate that the gains are not primarily explained by exact annotation-source reuse or explicit emotion descriptions in the downstream evidence.

Table 3. Leakage-control experiments using Qwen2.5-Omni-7B (%). Each cell reports UAR / Acc. / WAF.

MethodVID-VVID-VS
AffectReveal29.25 / 29.80 / 27.1132.83 / 33.29 / 30.61
Emotion Filtering29.22 / 29.70 / 26.8433.06 / 33.46 / 30.76
Source Exclusion29.86 / 30.33 / 27.5933.89 / 33.46 / 31.32
Both Restrictions29.28 / 29.77 / 26.9133.00 / 33.42 / 30.74

W3/Q3. Additional model families and scales

We added InternVL3.5-8B, InternVL3.5-30B-A3B, and GPT-5, each evaluated with and without AffectReveal under all four visual settings. AffectReveal improves UAR in all 12 untuned model–setting comparisons (Table 4). Under VID-V, the gains are 12.41, 12.80, and 12.34 percentage points, respectively. It also improves the fine-tuned InternVL3.5 variants. Within each model, paired evaluations use identical visual inputs, and external search is disabled for the downstream predictor. Exact checkpoints, API configurations, and frame-sampling details will be included in the revised appendix.

Table 4. Additional model comparisons under four visual settings (%). Each cell reports UAR / Acc. / WAF. Base denotes models without EGER-Bench fine-tuning, while FT denotes task-specific fine-tuning on EGER-Bench. Bold indicates the UAR improvement within each baseline–AffectReveal pair.

MethodVID-VVID-VSIMG-VIMG-VS
InternVL3.5-8B
Base21.01 / 21.53 / 15.6829.98 / 30.02 / 25.5617.46 / 18.73 / 13.8522.07 / 22.89 / 19.25
+ AffectReveal33.42 / 33.36 / 29.9238.48 / 38.21 / 35.3530.49 / 30.87 / 28.2533.34 / 33.26 / 30.70
FT28.88 / 29.20 / 23.1439.33 / 39.44 / 36.2420.78 / 22.36 / 17.4933.64 / 35.00 / 31.99
FT + AffectReveal40.18 / 40.20 / 36.4344.38 / 44.22 / 43.3133.72 / 34.07 / 32.1040.49 / 41.86 / 39.61
InternVL3.5-30B-A3B
Base23.20 / 23.39 / 17.9332.23 / 32.23 / 27.7418.93 / 19.32 / 15.4225.44 / 25.71 / 22.21
+ AffectReveal36.00 / 36.08 / 32.8541.12 / 41.10 / 38.5032.91 / 32.95 / 31.0835.30 / 35.53 / 33.87
FT33.66 / 33.95 / 30.7045.72 / 45.78 / 44.5427.87 / 29.60 / 26.0337.95 / 39.01 / 36.68
FT + AffectReveal43.67 / 43.92 / 42.1849.24 / 49.40 / 48.6439.10 / 40.62 / 39.3543.45 / 44.53 / 43.68
GPT-5
Base19.48 / 20.26 / 16.8021.93 / 22.52 / 19.5415.77 / 16.56 / 13.6118.37 / 19.00 / 16.20
+ AffectReveal31.82 / 31.85 / 30.4433.63 / 33.69 / 32.3429.32 / 29.12 / 27.7631.95 / 31.58 / 30.09

Q2. Why these 11 emotion categories?

We retain six basic emotions plus neutral, following the seven-class setting of CAER [1], and add pride, embarrassment, contempt, and respect to cover socially evaluative reactions involving roles, relationships, and event outcomes. Pride, embarrassment, and contempt are also represented in ADFES [2]. Attributional and empirical research supports the relevance of event and social evaluation to these emotions [3,4]. The resulting 11 categories form a task-oriented label set, not an exhaustive taxonomy of human emotions.

 

References

[1] Context-Aware Emotion Recognition Networks. ICCV, 2019.

[2] Moving Faces, Looking Places: Validation of the Amsterdam Dynamic Facial Expression Set (ADFES). Emotion, 2011.

[3] An Attributional Analysis of Moral Emotions: Naïve Scientists and Everyday Judges. Emotion Review, 2014.

[4] The Everyday Moral Judge – Autobiographical Recollections of Moral Emotions. PLOS ONE, 2016.

Ethics concern. Copyright and benchmark release

Please refer to the corresponding Ethics section in the General Response for details.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your careful review and constructive suggestions. We want to highlight the additions addressing your main concerns briefly:

  1. Recovery reliability: Human assessments now provide the primary evidence for event recovery quality.
  2. Potential leakage: Source-exclusion and emotion-filtering experiments assess whether gains depend on annotation-source reuse or explicit emotion descriptions.
  3. Model coverage: Additional InternVL3.5 and GPT-5 experiments broaden the evaluation across model families and scales.
  4. Copyright: We clarified permission-dependent release boundaries, including restrictions on redistributing original media.

Please let us know if any concerns remain or further clarification would be helpful. We appreciate your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you for the constructive feedback and recognition of the benchmark and human study. We highlight three points:

(1) EGER opens emotion recognition to active external evidence acquisition. Rather than only interpreting supplied context, it requires establishing what happened and how it relates to the observed person. EGER-Bench makes this problem systematically testable.

(2) AffectReveal addresses task-specific challenges beyond retrieving relevant text. It distinguishes competing event interpretations and verifies external evidence against non-facial visual facts, connecting event recovery with evidence integration.

(3) Our responses directly examine the principal concerns. We added controlled RAG comparisons, human recovery assessments, annotation audits, leakage controls, broader model evaluations, and retrieval-aware fine-tuning, alongside explicit privacy and release boundaries.

The contribution is a new research setting and benchmark, together with an initial framework for acquiring and validating the events needed to understand visual reactions.

 

Ethics: Privacy, annotator protection, and copyright

Privacy. Event recovery submits representative frames, potentially containing identifiable faces, to third-party reverse-image-search and web-search services. Retrieved names, roles, and relationships are used for event identification rather than identity tracking. Public releases exclude unrelated personal information and restrict samples involving private individuals, minors, bystanders, or sensitive circumstances. Face masking applies only to event verification, not external retrieval.

Annotator protection. Public releases exclude annotator identities, contact details, and individual participation records, retaining only task-relevant annotations and aggregate statistics. The revised Ethics Statement will clarify the annotator protocol and consent procedure.

Copyright and benchmark release. We will not redistribute original television-series clips, news videos, or extracted frames without appropriate permissions, including materials from Friends and Empresses in the Palace. The intended release comprises independently created annotations, sample identifiers, and source-localization metadata, subject to source-specific copyright and platform-term review. URLs, timestamps, and reproduction tools will be included only where permitted. We will also provide copyright notices, usage restrictions, and a mechanism for reporting concerns and requesting removal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.