- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the ability of general-purpose multimodal large language models to understand observed viewer reactions and relate them to video stimuli. The authors introduce ReactionBench with three tasks: Intensity Profiling (IP), which ranks reaction intensity; Stimulus–Reaction Matching (SRM); and Grounded Rationale Generation (GRG), which generates stimulus-grounded explanations of reactions. They collect 31,834 synchronized stimulus–reaction candidate clips, obtain 14,894 human intensity ratings, and retain 447 human-written rationale references after quality review. The final evaluation sets comprise 250 IP videos, 112 SRM items, and 200 GRG items. Experiments with seven multimodal large language models show that current models have limited performance overall across the three tasks and that the results are highly sensitive to input format and evaluation protocol.
Critical strengths
- The paper focuses on observed viewer reactions and explores models' ability to understand them through three tasks covering intensity ranking, stimulus matching, and grounded explanation. This perspective offers some novelty.
- The authors evaluate seven multimodal large language models and examine how input formats and evaluation settings affect the results, including the Joint/Independent comparison for IP, the Stimulus-only and Shuffled-reaction controls for SRM, and experiments with native video inputs. They also report human annotation agreement and sensitivity analyses for some evaluation metrics. These experiments provide a relatively thorough analysis of the evaluation protocol.
Critical weaknesses
-
The validity of the benchmark has not been sufficiently established. Through the complementary IP, SRM, and GRG tasks, the paper aims to evaluate models' understanding of viewer reactions and their relationship to stimuli. However, there is insufficient evidence that the task scores effectively measure this core capability. IP mainly measures viewer reaction intensity without involving stimulus content. SRM directly evaluates stimulus–reaction matching, but has not been validated against an independent human baseline to establish whether the correct answer can be reliably identified from the reaction and candidate stimuli alone. Moreover, GPT-5's performance differences across the correct-reaction, no-reaction, and substituted-reaction conditions are not statistically significant. GRG primarily evaluates agreement between model explanations and human references, which does not adequately establish whether the model has identified the stimulus that actually elicited the reaction. Although the three tasks address different aspects of reaction understanding, it remains unclear to what extent the benchmark's overall results reflect an understanding of stimulus–reaction relationships.
-
There are substantial concerns about the reliability of the annotations and evaluation references. First, agreement among human intensity ratings for IP is low (Krippendorff's α = 0.357), and 58% of the test videos contain only two moments to rank. Although the authors apply rater-bias correction and leave-one-rater-out analysis, the stability of the final human-consensus rankings has not been sufficiently established. Second, SRM determines the correct stimulus directly through temporal alignment. Viewer reactions may be delayed, however, and a reaction may have multiple plausible triggers, so temporal alignment does not necessarily identify the actual stimulus–reaction relationship. Finally, GRG references are written after annotators watch continuous stimulus videos with audio and may explain reactions using dialogue or sound, whereas models receive only silent frames sampled at 1 FPS. This information asymmetry may make some references difficult to infer reliably from the model inputs and thereby complicate interpretation of the evaluation results.
Questions for authors
-
Given the low agreement among human intensity ratings for IP (Krippendorff's α = 0.357) and the fact that 58% of test videos contain only two moments to rank, could the authors report human agreement on relative intensity rankings separately for videos with 2, 3, 4, and 5 moments? Could they also assess how sensitive the final consensus rankings are to the choice of annotators?
-
The correct answer in SRM is determined by temporal alignment, but viewer reactions may be delayed and different stimuli may elicit similar reactions. Is there further evidence that the correct answer can be reliably identified from the provided reaction and candidate stimuli alone? For example, are independent human matching results available under the same input conditions as those used for the models?
-
GRG references are written after viewing continuous stimulus videos with audio, whereas models see only silent frames sampled at 1 FPS. How do the authors verify that these references can be inferred from the information actually available to the models? In addition, the human references in Figure 3(b) differ substantially in specificity. How do the authors ensure that reference-based evaluation fairly assesses different plausible explanations produced by the models?
Thank you for the detailed review and for the strengths you note: the focus on observed viewer reactions through three tasks, and the analysis of the evaluation protocol (joint vs. independent IP, the SRM stimulus-only and shuffled-reaction controls, native-video inputs, human agreement and metric sensitivity). The new experiments extend these analyses (General Response G1–G4).
W1 / Q2 / W2 (SRM label) — validity, human baseline, temporal alignment.
The SRM label. Reaction and stimulus are synchronized recordings, and the correct option is the stimulus segment that was playing when the reaction occurred (Appendix B.5). Because the label comes from the recording rather than from annotator judgment, we respectfully disagree that a human baseline is needed to establish it; a human score would instead describe how hard the task is for people. Construction adds three checks: the reaction view spans 2 s before to 1 s after the annotated range, following the sub-second onset of spontaneous facial reactions (Dimberg et al., 1998; 2000); option start times differ by at least 6 s (median 11.2 s) and options are description-dissimilar; and a human reviewer replayed every item to confirm the candidates are distinguishable against the reaction. For the camera-ready version, we would be happy to add such a human baseline on the models' inputs as a reference point for readers.
The controls. We see the GPT-5 result you point out as the benchmark's central finding rather than a weakness, and it now replicates across six models (General Response G3): the real reaction adds +2.9 points over stimulus-only and +3.5 over a shuffled reaction, and no single model's contrast is significant. Current models reach most of their SRM score without the observed reaction.
IP and GRG. IP is reaction-only by design: it tests whether a model can read the reaction signal, and SRM and GRG test the link to the stimulus. For GRG, the cue sub-score measures whether the model names the stimulus cue that the references identify; we will report it as the trigger-identification measure.
W2 / Q1 — α = .357, two-moment videos, consensus stability. The consensus rankings used as references are stable: removing any one rater leaves them at Spearman .866 [.847, .884] to the full consensus over 1,073 removals, with the full ordering unchanged in 78% (the mean scores the 117 all-tied remainders as 0, per the paper's ρ₀ convention; .972 without them), flat across 2 / 3 / 4 / 5+ moments (.862 / .867 / .876 / .891). α measures agreement on the absolute 0–3 scale, while IP is scored on within-video order. Stratified rater-vs-others ranking agreement, as requested: .826 [.731, .907] for 2 moments, .490 for 3, .495 for 4 and .312 for 5+ (video-cluster CIs; as in the paper, pairs with an all-tied score vector are dropped, which removes 59 of the 145 two-moment videos). Both tables go into the appendix.
W2 / Q3 — GRG audio and reference specificity. A text-only screen flags 13 of 200 references as potentially audio-dependent; without them every model's composite moves by .01–.08 and the larger gaps are unchanged (General Response G4). We will report both sets and mark the flagged items; a human check of the sampled frames remains the direct test of inferability. On specificity, the judge scores cue against at least one of the 1–5 accepted references, so a model matching any reference at that reference's level of specificity receives full credit, and the human anchor (84 pairs, Spearman .593 with the human mean) shows that the judge tracks human judgments under this rule.
We hope these responses clarify the design of the benchmark and address your concerns, and we would be grateful if you would consider them in your final assessment. We are happy to answer any further questions.