- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Event-Grounded Emotion Recognition (EGER), in which a person’s emotion depends on an event not visible in the image or video. For this new task, the work introduces a new benchmark of video and static images, and AffectReveal, a training-free pipeline that runs web searches to get the context of a given input to better inform the classification.
Critical strengths
- The dataset is the main contribution. It covers two domains and is large. Text-removed and text-retained versions share the same frames, and the source-grounded reference context is kept for diagnostics.
- Human study supports the claims on context.
Critical weaknesses
- The task novelty is narrower than claimed. Context-aware emotion recognition (EMOTIC, CAER, EmotiCon), emotion-cause extraction, conversational context in MELD, and Social-IQ all frame emotion as depending on context beyond facial appearance. EGER is not that far from these previous tasks.
- AffectReveal is mostly web-search RAG, and it is well established that this helps MLLMs' reliability. In addition, the paper has no RAG baselines. Every comparison is visual-only vs. +AffectReveal, and search is disabled for the baselines.
- There is possible label leakage in the RAG; phrases such as “tears of joy” and “forcing a smile” were used to compose the dataset, which may now be revealed in the reverse image search for context.
- The “outperforms fine-tuned models” claim is unfair. Fine-tuned visual-only models have no access to the missing event, while AffectReveal does. The fair comparison is fine-tuning with retrieved context.
- The pipeline identifies real people through reverse image search and web lookup. This is a privacy risk that the ethics statement doesn’t address. Licensing for redistributing clips from TV series (Friends and Empresses in the Palace) is also unresolved.
Questions for authors
- How does AffectReveal compare with a plain RAG baseline under the same search budget across all models and settings, and with AffectAgent?
- How often does the retrieved context contain emotion words, the seed phrases, or the source page/article of the sample?
- What is emotion accuracy for Match, Partial, and Mismatch recoveries separately?
- How many annotators labeled each benchmark sample, what was their agreement, and how were disagreements resolved?
- How were the fine-tuned baselines trained, and how do they perform when fine-tuned with retrieved context?
I am willing to raise the score if the weaknesses and ethics concerns, are resolved and the over-claimed novelty is addressed is toned down.
5 comments
Response to Reviewer dwhC (Part 2/3)
W2/Q1. Comparisons with RAG baselines and AffectAgent
(1) Task-specific differences from generic retrieval augmentation.
AffectReveal differs from retrieve-and-append baselines in two ways. First, it verifies competing event interpretations: evidence-anchored alternatives change only one factor—event identity, focal role, relationship, or outcome—and independent searches test the discriminative fact. Second, it checks applicability to the observed sample: bidirectional atomic support compares external facts with non-facial in-media facts. Faces are masked only in this branch to reduce circular use of misleading expressions. The support score and explanation accompany both contexts for emotion prediction.
(2) Controlled comparisons and results.
Primary Retrieval uses the initial event; Generic Multi-Search refines queries without factorized alternatives or expression-disentangled cross-verification. Comparisons use the same downstream models, visual inputs, retrieval model, tools, context format, and prediction template. Generic Multi-Search and AffectReveal share the same maximum retrieval budget. Across five models, AffectReveal improves average UAR by 0.75–4.58 points over Primary Retrieval and 0.21–2.06 over Generic Multi-Search (Table 1), supporting additional gains beyond retrieval access or search count. Gains vary across individual settings.
Table 1. Retrieval-baseline comparisons (%). Cells report UAR / Acc. / WAF. Base denotes Visual-Only inference without EGER-Bench fine-tuning. Bold marks the highest UAR per model and setting.
| Method | VID-V | VID-VS | IMG-V | IMG-VS |
| Emotion-LLaMA | ||||
| Base | 20.42 / 20.90 / 15.69 | 21.36 / 21.99 / 16.05 | 18.87 / 19.72 / 15.36 | 18.38 / 19.63 / 15.11 |
| + Primary Retrieval | 28.15 / 28.80 / 23.17 | 28.47 / 29.14 / 23.59 | 23.27 / 24.01 / 19.27 | 23.35 / 24.01 / 19.39 |
| + Generic Multi-Search | 28.90 / 29.57 / 24.32 | 29.02 / 29.70 / 24.28 | 23.56 / 24.41 / 18.87 | 23.63 / 24.44 / 19.02 |
| + AffectReveal | 29.10 / 29.77 / 24.51 | 29.13 / 29.80 / 24.38 | 24.47 / 25.16 / 20.03 | 24.52 / 25.25 / 19.92 |
| AffectGPT | ||||
| Base | 25.63 / 25.59 / 24.44 | 25.82 / 26.08 / 24.51 | 24.60 / 24.66 / 22.76 | 24.11 / 25.06 / 23.85 |
| + Primary Retrieval | 26.01 / 26.31 / 24.81 | 25.82 / 26.08 / 24.54 | 25.34 / 25.64 / 24.02 | 25.70 / 26.01 / 24.35 |
| + Generic Multi-Search | 27.20 / 27.85 / 25.56 | 27.92 / 28.49 / 26.26 | 29.68 / 30.15 / 28.25 | 28.15 / 28.65 / 26.39 |
| + AffectReveal | 29.48 / 30.00 / 28.02 | 29.93 / 30.46 / 28.51 | 30.70 / 30.25 / 28.70 | 31.07 / 30.87 / 29.44 |
| Qwen2.5-Omni-7B | ||||
| Base | 17.11 / 17.94 / 11.13 | 21.85 / 22.52 / 17.02 | 16.66 / 18.04 / 13.77 | 18.95 / 20.37 / 16.99 |
| + Primary Retrieval | 28.09 / 29.80 / 25.09 | 30.63 / 31.13 / 27.77 | 25.72 / 26.99 / 25.63 | 27.76 / 29.07 / 27.58 |
| + Generic Multi-Search | 28.61 / 29.07 / 26.32 | 31.73 / 32.16 / 29.18 | 26.02 / 27.08 / 25.63 | 27.80 / 28.76 / 27.43 |
| + AffectReveal | 29.25 / 29.80 / 27.11 | 32.83 / 33.29 / 30.61 | 26.58 / 27.67 / 25.79 | 28.02 / 29.13 / 27.20 |
| InternVL3.5-8B | ||||
| Base | 21.01 / 21.53 / 15.68 | 29.98 / 30.02 / 25.56 | 17.46 / 18.73 / 13.85 | 22.07 / 22.89 / 19.25 |
| + Primary Retrieval | 32.68 / 32.62 / 29.07 | 36.68 / 36.45 / 33.26 | 30.51 / 30.68 / 27.89 | 32.86 / 32.80 / 29.97 |
| + Generic Multi-Search | 33.34 / 33.32 / 29.97 | 37.76 / 37.41 / 34.83 | 30.51 / 30.87 / 28.16 | 33.27 / 33.23 / 30.87 |
| + AffectReveal | 33.42 / 33.36 / 29.92 | 38.48 / 38.21 / 35.35 | 30.49 / 30.87 / 28.25 | 33.34 / 33.26 / 30.70 |
| InternVL3.5-30B-A3B | ||||
| Base | 23.20 / 23.39 / 17.93 | 32.23 / 32.23 / 27.74 | 18.93 / 19.32 / 15.42 | 25.44 / 25.71 / 22.21 |
| + Primary Retrieval | 34.89 / 34.85 / 31.17 | 39.86 / 39.97 / 37.46 | 32.15 / 32.27 / 30.26 | 34.90 / 35.03 / 33.12 |
| + Generic Multi-Search | 35.79 / 35.91 / 32.72 | 39.89 / 40.00 / 37.70 | 32.78 / 32.83 / 30.95 | 34.90 / 35.06 / 33.41 |
| + AffectReveal | 36.00 / 36.08 / 32.85 | 41.12 / 41.10 / 38.50 | 32.91 / 32.95 / 31.08 | 35.30 / 35.53 / 33.87 |
(3) Relation to AffectAgent.
AffectAgent retrieves supportive, confusing, and countering emotion exemplars and optimizes its agents through SFT and MAPPO. AffectReveal instead tests competing factual interpretations of the event underlying the observation, then checks them against non-facial visual evidence, without model training. We could not obtain an accessible implementation for faithful reproduction; adapting AffectAgent’s retrieval and training components to EGER also requires validation. We therefore provide controlled open-web RAG comparisons and do not claim superiority over AffectAgent.
Response to Reviewer dwhC (Part 3/3)
W4/Q5. Fine-tuning with retrieved context
Evidence-matched comparison. Original FT models were trained on visual inputs and emotion labels. We added retrieval-aware fine-tuning (AR-FT), which includes automatically recovered context by AffectReveal during training. FT + AffectReveal and AR-FT + AffectReveal use the same retrieval pipeline at inference. AR-FT reaches 32.48% and 34.11% UAR, compared with 36.55% and 36.48% for visual-only FT followed by AffectReveal (Table 3). In this tested configuration, adding context during training does not outperform introducing it at inference. We therefore emphasize the complementary value of fine-tuning and event acquisition, rather than an unconditional superiority claim over fine-tuning.
Table 3. Visual-only and retrieval-aware fine-tuning of Qwen2.5-Omni-7B (%). Each cell reports UAR / Acc. / WAF. FT uses visual-only training inputs, while AR-FT incorporates automatically recovered context during training. Both augmented variants receive AffectReveal evidence at inference.
| Method | VID-V | VID-VS |
| FT | 25.34 / 25.45 / 21.54 | 33.19 / 32.72 / 29.50 |
| FT + AffectReveal | 36.55 / 36.08 / 35.44 | 36.48 / 36.78 / 33.54 |
| AR-FT + AffectReveal | 32.48 / 32.43 / 28.86 | 34.11 / 34.65 / 30.67 |
Q3. Emotion accuracy by recovery quality
We grouped 2,359 VID-V samples with valid Qwen3.5-Plus recovery assessments and evaluated Qwen2.5-Omni-7B + AffectReveal. Accuracy is 34.04%, 30.21%, and 28.12% for Match, Partial, and Mismatch recoveries (Table 4). This shows an association between higher automatically assessed recovery quality and better emotion-recognition performance.
Table 4. Emotion recognition grouped by Qwen3.5-Plus-assessed recovery quality (%).
| Recovery Quality | Samples | UAR | Acc. | WAF |
| Match | 329 | 33.40 | 34.04 | 32.67 |
| Partial | 682 | 29.09 | 30.21 | 27.74 |
| Mismatch | 1,348 | 28.19 | 28.12 | 25.60 |
Q4. Annotation protocol and agreement
Original protocol and independent audit. Original labels were assigned by one annotator per sample using visual materials and source-grounded event context. We subsequently audited 500 randomly sampled instances with eight annotators: two independently labeled each sample, with disagreements adjudicated by a third. Pre-adjudication agreement was 68.60% (343/500), with Fleiss’ (\kappa=0.64). Adjudicated labels agreed with the benchmark on 91.60% (458/500), leaving 42 discrepancies (Table 5). This audit measures annotation consistency; all reported experiments retain the same original benchmark labels.
Table 5. Independent annotation audit. Agreement with benchmark labels uses adjudicated labels; other agreement measures are calculated before adjudication.
| Samples | Annotators | Raw Agreement | Fleiss’ | Agreement with Benchmark |
| 500 | 8 | 68.60% | 0.64 | 91.60% |
W5/Ethics concern. Privacy and copyright
Please refer to the corresponding Ethics section in the General Response for details.
Response to Reviewer dwhC (Part 1/3)
Dear Reviewer dwhC,
Thank you for your careful and constructive review. We appreciate your recognition of EGER-Bench as the main contribution, particularly its scale, two-domain coverage, paired visual settings sharing the same frames, and source-grounded reference contexts for diagnostics. We also appreciate your acknowledgment of the human study supporting the value of event context. We address your concerns below with clarified task boundaries, retrieval-controlled comparisons, and additional fine-tuning and annotation analyses.
(1) From interpreting supplied context to actively acquiring event evidence.
EGER extends emotion recognition beyond inference over a fixed observation. EMOTIC, CAER, and EmotiCon exploit context within the supplied media; MELD, emotion-cause extraction, and Social-IQ provide conversations, documents, or videos for contextual reasoning [1–6]. EGER instead makes acquiring missing, sample-specific event evidence part of the recognition problem. A system can search external sources to establish what happened and how the focal person was involved, rather than relying solely on the supplied media or knowledge already encoded in its parameters. This also allows access to evolving event information without requiring model retraining.
(2) External acquisition introduces challenges that additional context alone does not solve.
The system must distinguish the relevant event from visually or semantically similar events, establish the focal person’s role and relationship to its outcome, and determine how retrieved facts should interact with the observed reaction. External evidence supplements—not replaces—in-media context: observable facts can help validate the recovered event, while the original expression remains necessary for emotion inference. Incorrect or distracting context can otherwise lead to an incorrect emotional interpretation.
(3) EGER-Bench opens this problem to systematic investigation.
The human study demonstrates the missing-evidence bottleneck: event context raises accuracy from 33.96% to 72.08%. Our controlled retrieval comparisons further show that single-search and budget-matched multi-search baselines do not fully resolve it. Evidence integration also matters: adding an in-media description to recovered event context changes UAR from 27.84% to 27.75%, whereas explicitly evaluating their support relation raises it to 29.25%. The contribution is therefore an externally grounded emotion-recognition task and benchmark that expose research questions in evidence acquisition, event disambiguation, verification, and internal–external evidence integration. AffectReveal provides an initial approach to these challenges, rather than treating web search itself as the solution.
References
[1] EMOTIC: Emotions in Context Dataset. CVPRW, 2017.
[2] Context-Aware Emotion Recognition Networks. ICCV, 2019.
[3] EmotiCon: Context-Aware Multimodal Emotion Recognition Using Frege’s Principle. CVPR, 2020.
[4] MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. ACL, 2019.
[5] Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts. ACL, 2019.
[6] Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence. CVPR, 2019.
Audit. Discovery phrases such as “tears of joy” and “forcing a smile” are not supplied at inference, but may reappear in retrieved webpages. Exact annotation-source URL overlap is 8.17% under VID-V and 9.18% under VID-VS. Explicit emotion descriptions appear in 5.82% and 5.68% of final evidence packages.
Controls. We filtered explicit affective descriptions, including seed phrases when present; excluded annotation-source URLs and matching tool results before event recovery; and combined both restrictions. Under both restrictions, UAR remains at 29.28% and 33.00%, retaining gains of 12.17 and 11.15 points over Visual-Only inference (Table 2). Although these controls do not directly assess pretraining exposure to television content, the same pretrained downstream model is used across all conditions, keeping its prior knowledge consistent. The gains are therefore not primarily explained by exact-source reuse or explicit descriptions in final evidence.
Table 2. Leakage controls using Qwen2.5-Omni-7B (%). Each cell reports UAR / Acc. / WAF.