acceptodds
Under review as a conference paper at ICLR 2027

AffToM: Benchmarking Recursive Affective Theory of Mind in Dynamic Multi-Party Interactions

Abstract

Understanding emotion in realistic social interactions requires more than recognizing facial expressions or assigning affective labels. The same event can elicit different emotions depending on what each participant knows, what they believe others know or feel, and how these nested mental states evolve over time. Existing benchmarks largely study higher-order epistemic Theory of Mind (ToM) and multimodal emotion reasoning in isolation, leaving recursive affective reasoning under asymmetric and dynamically changing information underexplored. Therefore, we introduce AffToM, a benchmark for affective ToM with 1,922 test pairs and 19,759 training pairs. AffToM evaluates Individual Affective Reasoning (IAR), which infers emotions from events and socio-cognitive context, and Multiparty Affective Reasoning (MPAR), which recursively tracks participants’ nested beliefs and emotions as new evidence emerges. We develop an event-driven multiagent generation pipeline grounded in dynamic character interaction graphs that explicitly capture event sequences, interpersonal relations, and mental-state updates. Furthermore, we propose AffToM-R1, a two-stage post-training framework combining supervised fine-tuning with perspective-aware rubric-guided reinforcement learning. Experiments show that MPAR remains challenging, with even the strongest evaluated MLLM achieving less than 60% accuracy, while AffToM-R1 improves MPAR accuracy by 13.38 percentage points over its base model.

Primary area:
datasets and benchmarks
Keywords:
Theory of Mind, Multimodal Reasoning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_9iuQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AffToM to evaluate individual affective reasoning and multi-party affective reasoning (MPAR) in videos, with the latter focusing on nested belief and emotion attribution and how these attributions change with new information. The evaluation set contains 258 videos, 1,822 multiple-choice questions, and 100 open-ended questions, constructed through a multi-agent pipeline based on character interaction graphs and subsequently subjected to automatic screening and manual review. The authors also construct 19,759 training QA pairs and propose AffToM-R1, which combines perspective-structured SFT with rubric-guided GRPO. Under the default video-only setting, AffToM-R1 improves the MPAR accuracy of Qwen3.5-4B from 37.22% to 50.60% and reports gains on external benchmarks.

Critical strengths
  1. The evaluation setting has clear value. MPAR combines affective attribution with nested character perspectives, and the paired pre- and post-event questions further examine changes in characters’ mental states after they receive new information.

  2. The resource quality and training value are validated from multiple angles. Approximately 150 hours of manual review, annotation consistency checks, human baselines, and validation of the open-ended response scorer support data quality. The open-ended scoring also explicitly checks recursive perspective structure. Evaluations across multiple models and input conditions, a vanilla GRPO control, a dedicated reward ablation, and external benchmark results jointly support the resource’s usefulness and training value.

Critical weaknesses

None. I did not identify any critical weaknesses sufficient to warrant rejection; the points requiring clarification are given in the questions below.

Questions for authors
  1. How many second- and third-order MPAR multiple-choice questions are there, respectively, and what are the model’s accuracies on each? Did the final review check whether some questions can be answered solely from event facts, belief contents, or lower-order character perspectives, without requiring full affective attribution? Could you provide the corresponding review criteria or examples?

  2. What video-level split and near-duplicate checking rules were used between the training data and the AffToM and R³-Bench evaluation sets? How were clips from the same movie or the same episode handled, and what were the results of the content-overlap checks?

  3. How is the 47.35% MPAR accuracy in Table 3 computed? It cannot be obtained by rounding a single-run integer number of correct answers over 755 questions; please specify the actual denominator and aggregation method. Are the complete QA-generation, SFT, and reward-scoring prompts, the actual contents corresponding to the knowledge-base and causal-rule placeholders in Appendix G, and the final data split configuration included in the supplementary materials?

  4. Why do the main tables use video frames only (V) rather than video plus subtitles (V+T)? In Table 6, adding subtitles raises the MPAR accuracy of Qwen3.5-27B from 45.03% to 51.79%. Did the authors check whether the sampled frames (including any embedded subtitles) provide the dialogue evidence needed to answer the questions? For pre-event questions, are the sampled frames drawn from the full video or only from the segment before the queried time point?

4 comments

[1/2]

Thank you for the specific questions. We respond to the four questions in turn below.

Response to Q1: Distribution of MPAR Recursion Orders and Genuine Reasoning Demand

Recursion orders. AffToM contains 755 second-order MPAR multiple-choice questions and no third-order multiple-choice questions. The 100 third-order questions are open-ended, evaluated using Emotion Correctness (EC) and Factual Consistency (FC).

Question review. Human reviewers assess each question based on answer plausibility, video dependency, distractor quality, and question validity (Appendix C.1). Reference answers must be supported by the video from the specified character's perspective and at the specified time, rather than relying on information unavailable to that character.

Lower-order shortcut analysis. To examine whether second-order questions can be answered by directly reusing lower-order emotions, we compared all 755 MPAR answers with their corresponding single-character emotion labels recorded during dataset construction (Figure 2(b)). 551 questions (72.98%, Mismatched group) have different higher-order and lower-order answers, while 204 (27.02%, Matched group) have matching answers.

MPAR accuracy (%) on the Matched and Mismatched groups.

ModelsAll questions (n=755)Matched group (n=204)Mismatched group (n=551)
AffToM-R150.6056.3748.46
Qwen3.5-27B45.0354.9041.38

Both models achieve substantial accuracy on the Mismatched group, where directly copying the lower-order emotion would be always incorrect. These results show that lower-order answer reuse cannot fully explain model performance.

Response to Q2: Train–Test Data Isolation

Thank you for this important question. Our dataset uses a video-level split, ensuring that all questions from the same video belong to a single split. To further check for potential data leakage, we compared the training set against both AffToM and R3R^3Bench at three levels:

  1. Video IDs: We checked for shared source video IDs.
  2. Visual content: We sampled frames every 2 seconds, applied pHash and dHash matching, and manually verified candidate overlaps using visual and dialogue evidence.

Train–test overlap analysis.

ComparisonShared Video IDsReused Clips
Train vs. AffToM00
Train vs. R3R^3Bench00

No overlapping video IDs or reused clips were detected. These results provide additional evidence against train–test contamination at the video and clip level. We will include the verification procedure and results in the revised appendix.

Response to Q3: Accuracy Calculation and Reproducibility

Accuracy correction. Thank you for identifying this inconsistency. We found that our evaluation script mistakenly excluded unparsable responses (ERROR) from the accuracy denominator rather than counting them as incorrect. For example, the first-row MPAR accuracy was calculated as 357/754 = 47.35%, whereas the correct value is 357/755 = 47.28%.

We have corrected five affected cells in the first two rows of Table 3 and rechecked the remaining results, finding no further discrepancies. The corrected results are shown below.

RperspR_{\mathrm{persp}}RfaithR_{\mathrm{faith}}RconsR_{\mathrm{cons}}IARMPAROverall
–––64.7647.2857.52
✓––64.8548.0857.90
✓✓–65.0449.8058.73
✓–✓65.3249.0158.56
✓✓✓65.5150.6059.33

Importantly, the corrected results preserve our main finding: the full method outperforms standard GRPO by 3.32 percentage points on MPAR and 1.81 points overall. We apologize for the oversight and will correct the table and related text.

Reproduction materials. We will provide the actual content of the knowledge base and causal rules in Appendix G, the complete data-generation prompts, the sample splits of the SFT and PAFRO subsets, and the prompts used for the supplementary reward scoring.

[2/2]

Response to Q4: Video Input Settings and Temporal Leakage

Why the main experiments use video-only frames (V). Thank you for raising this. The visual evidence AffToM tests (who is present, who witnesses a key event, who has not yet learned a fact, and the characters' expressions, gaze, and actions) exists only in the frames, and dialogue cannot replace it. V is the configuration least able to exploit language priors, so it best tests whether a model reasons from visual evidence. We do not hide the gain that subtitles bring. Table 6 reports V and V+T side by side (Qwen3.5-27B improves from 59.22% to 64.71%) and already includes V+T results for some models. We will add more models to Table 6.

Video-only setting. We adopted video-only (V) as the default because AffToM focuses on visual evidence, such as character presence, actions, and expressions. V is the configuration least able to exploit language priors, so it best tests whether a model reasons from visual evidence. However, we acknowledge that sampled frames may contain embedded subtitles and that dialogue can provide useful clues. To assess the contribution of visual information beyond subtitles, we conducted additional subtitle-only (T) experiments and compared T, V, and video-plus-subtitle (V+T) settings on the same test set.

ModelTVV+T
Qwen3.5-4B36.1637.2240.40
AffToM-R148.8750.6052.72
Qwen3.5-27B43.0545.0351.79
GPT-5.552.8552.0556.95

V+T consistently outperforms both single-modality settings, suggesting that visual and textual information provide complementary evidence for MPAR.

Frame sampling We also considered that uniform sampling may miss key evidence, so Appendix E.2 (Table 7) compares frame sampling strategies. With the frame budget and all other inference settings fixed, we replaced uniform sampling with two keyframe selection methods, BOLT [1] and WFS-SB [2], to see whether more targeted frame selection can capture key evidence and improve answers.

MPAR accuracy by frame sampling strategy (%).

ModelsUniformBOLTWFS-SB
Qwen3-VL-32B40.7944.9044.37
Qwen3.5-27B45.0349.6748.21

Both adaptive sampling strategies beat uniform sampling, and BOLT improves MPAR accuracy of the two models by 4.11 and 4.64 percentage points. This supports the value of keyframe selection for collecting the evidence that multi-party affective reasoning needs.

We also inspected the 64-frame uniform inputs of 10 questions, and 9 of them kept the key evidence needed to answer. In the video of Figure 10(a), for example, a man wears a head covering against the cold while choosing a kitchen knife, and a woman mistakes him for a robber. Frame 3 ("With this on I won't be cold riding") gives his real purpose, and frame 10 ("You bad man"), together with the woman's spraying and frightened reaction, shows his inference about how she sees him.

Frame range for pre-event questions. We extract the sampled frames from the complete video for all questions, so pre-event and post-event questions have identical frame sets.

Temporal leakage. In the original evaluation, pre- and post-event questions use frames sampled from the full video, with the target time specified in the question. We acknowledge that this may expose models to information unavailable to the character at the target time. To investigate this concern, we conducted a same-question truncation experiment on 210 pre-event questions, keeping the questions and answers unchanged.

ModelFull VideoTruncated VideoImprovement
Qwen3.5-4B30.0031.90+1.90
AffToM-R139.0542.86+3.81
Qwen3.5-9B28.1032.38+4.28
Qwen3.5-27B27.6237.62+10.00

All four evaluated models improve when later footage is removed (+1.90 to +10.00 percentage points). This suggests that post-event information can interfere with judgments about earlier mental states. These findings suggest that current models struggle to reason from a character's time-specific knowledge rather than all information available in the video. In particular, additional context may introduce hindsight bias instead of improving mental-state reasoning. This highlights an important challenge captured by AffToM: models must not only understand social events but also track who knows what and when. We will clarify this interpretation and the limitations of the truncation control in the revised paper.

[1] Liu, S., Zhao, C., Xu, T., and Ghanem, B. BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding. CVPR 2025.

[2] Chen, W., Zeng, Y., Luo, Y., Xie, T., Lin, L., Ji, J., Zhang, Y., and Zheng, X. Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding. CVPR 2026.

The author provided responses regarding "Distribution of MPAR Recursion Orders" and "Data Isolation," which cleared up some of my confusion.

The author corrected an issue where samples with incorrect responses were excluded from scoring, and re-evaluated the scores. The revised results are consistent with the experimental trends and conclusions presented earlier in the paper.

The authors provided an additional comparison between the text-only, vision-only, and text+vision modalities, which addressed my question. The experimental results show that the text-only and vision-only modalities perform at roughly the same level, while combining the modalities leads to improved performance.

The author’s mention of the presence of explicit subtitles in the videos has sparked a new line of inquiry for me: In the practical application of an "affective theory of mind," the input typically consists of video alone, without subtitles. However, it might be worth exploring the addition of audio-modality inputs to conduct further experiments.

By comparing different frame sampling strategies and providing case studies, the authors answered my question: "Did the authors check whether the sampled frames (including any embedded subtitles) provide the dialogue evidence needed to answer the questions?"

Based on feedback and experimental results, the author identifies a significant challenge: "models must not only understand social events but also track who knows what and when." This insight offers valuable direction for future research in related fields, particularly concerning personal agents and human-computer interaction.

I have a positive view of this work and recommend its acceptance.

Reviewer_oMJbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AffToM, a benchmark for affective Theory of Mind in social videos, covering individual and recursive multi-party affective reasoning. It contains 1,922 evaluation questions and 19,759 training questions constructed through a multi-agent pipeline with human validation of the test set. The authors also propose AffToM-R1, combining supervised fine-tuning with perspective-aware reinforcement learning, and evaluate it through benchmark comparisons, a human baseline, and training ablations.

Critical strengths
  1. ToM research needs benchmarks grounded in real-world social videos, where beliefs and emotions unfold through dialogue, behavior, and changing access to information. AffToM addresses this need by focusing on how people interpret one another’s emotions from different perspectives. Collecting these scenarios and constructing questions around them is a substantial effort and a useful contribution to social reasoning research.

  2. The paper goes beyond reporting low model accuracy. It includes a human baseline, a training dataset, and a dedicated training approach with reward ablations and evaluation on external benchmarks. The reported human MPAR accuracy of 84.55% provides evidence that the sampled questions are answerable, while the training results offer a practical starting point for improving models on this task.

Critical weaknesses
  1. The evidence supporting the mental-state labels is not sufficiently clear. In Figure 1, the husband is said to believe that the wife believes the mother feels regretful. What evidence supports this emotional attribution and each character’s understanding of the others? The paper does not explain how annotators distinguish video-supported inferences from plausible commonsense assumptions, especially when several interpretations are reasonable. Human agreement alone cannot rule out shared assumptions.

  2. The necessity of visual input remains insufficiently tested. Table 6 compares video-only with video-plus-subtitle inputs, but does not report subtitle-only performance. Language shortcuts are a recurring concern in multimodal understanding benchmarks. For example, DeSIQ [1] found that models could exploit answer biases in Social-IQ without access to the video or even the question. A subtitle-only baseline is therefore important for assessing whether AffToM requires visual evidence beyond dialogue and language priors. The current comparison shows that subtitles help, but does not establish whether the visual input is necessary.

  3. The necessity of higher-order reasoning is not directly validated. Nested question wording does not guarantee that answering requires recursive reasoning. If a higher-order answer matches a lower-order or factual answer, models may answer correctly without tracking the intermediate perspectives. The paper does not quantify how often these answers differ or report performance on such cases, leaving it unclear whether the benchmark reliably tests the intended recursive reasoning ability.

I am open to raising my assessment if these concerns are adequately addressed during the rebuttal.

[1] Guo, X.-Y., Li, Y.-F., and Haf, R. DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding. EMNLP 2023.

Questions for authors
  1. In Figure 1, what evidence supports each step of the nested belief chain? How do annotators distinguish evidence-supported mental states from plausible commonsense assumptions, and handle cases with multiple reasonable interpretations?
  2. Could you add a subtitle-only baseline, particularly for MPAR? The current V versus V+T comparison does not show whether the questions can be answered without visual input.
  3. How many higher-order questions have answers that differ from the corresponding lower-order or factual answers? Could you report performance on these cases to test whether models actually need to track the nested perspectives?
3 comments

[1/2]

Thank you for the careful assessment and constructive suggestions. We added a subtitle-only experiment and an analysis of questions whose higher-order and lower-order emotions differ. Our point-by-point responses follow.

Response to W1 & Q1: Evidence for the Nested Belief Chain in Figure 1

Evidence for the nested belief chain. We thank you for this important concern. Figure 1 illustrates three perspectives grounded in the video:

Mother-in-law: She initially urges her son to divorce his wife because she cannot conceive. After reading a forged report claiming that her son is infertile, she reverses her stance and opposes the divorce.

Wife (from the husband's perspective): The husband observes his wife reading the report, witnessing the mother-in-law's reversal, and questioning her double standard. Since she has not discovered the real report, he infers that she believes the mother-in-law accepts the diagnosis and has changed her stance accordingly.

Husband (nested inference): Based on his understanding of what his wife knows, the husband further infers that she interprets the mother-in-law's reversal as regret over previously urging divorce.

Thus, the inference follows the nested perspective husband believes → wife believes → mother-in-law feels, with each step grounded in the character's available information. We will clarify this reasoning in Figure 1.

Answer determination and ambiguity. We agree that multiple interpretations may be plausible. Our annotation process selects the answer best supported by the video from the specified character's perspective and at the specified time, rather than relying on commonsense assumptions alone. Annotators examine answer plausibility, video dependency, distractor quality, and question validity (Appendix C.1). Questions with insufficient evidence or unresolved ambiguity are excluded.

These checks are intended to ensure that the reference answer follows the character's available evidence, rather than merely a plausible interpretation of the scene.

Response to W2 & Q2: Subtitle-Only Baseline

AffToM is a video-based benchmark designed to evaluate affective reasoning from visual evidence, including character presence, event observation, facial expressions, and actions. We therefore adopted video-only (V) as the default setting in the original submission. Inspired by DeSIQ, we first conducted an options-only experiment to examine potential answer biases. Given only the answer options, Qwen3.5-4B and AffToM-R1 achieved accuracies of 33.15% and 31.01%, only modestly above the 25% chance level. We will cite this paper in the revised manuscript.

Following your suggestion, we conducted additional subtitle-only (T) experiments and compared T, V, and video-plus-subtitle (V+T) settings on the same test set to assess the contribution of visual information beyond dialogue.

Effect of input modality. MCQ accuracy (%); EC and FC are the open-ended scores (1–5).

ModelsModalityIARMPARMCQ Avg.ECFCOpen-ended Avg.
Qwen3.5-4BT56.7936.1648.241.831.841.84
V57.8337.2249.291.771.501.64
V + T63.0740.4053.681.781.571.68
AffToM-R1T66.6448.8759.282.071.952.01
V65.5150.6059.332.171.892.03
V + T66.3552.7260.702.171.962.07
Qwen3.5-27BT68.2343.0557.792.512.492.50
V69.2645.0359.222.452.232.34
V + T73.8551.7964.712.692.662.68
GPT-5.5T74.7952.8565.703.293.323.31
V71.7052.0563.563.062.993.03
V + T76.1056.9568.173.513.463.49

On all four models, V+T beats both single modalities on MPAR, exceeding the stronger single modality by 3.18, 2.12, 6.76, and 4.10 percentage points. Subtitles only tell us who said what. They cannot show who is present, who witnessed a key event, or who has not yet learned a fact, and these determine each character's information set and therefore the attribution of nested beliefs and emotions. Visual input therefore provides gains that subtitles cannot replace.

[2/2]

Response to W3 & Q3: Necessity of Higher-Order Reasoning

We thank you for raising this important concern. To examine whether MPAR questions can be answered by simply reusing lower-order emotion labels, we compared the higher-order answers of all 755 second-order MPAR questions with their corresponding single-character emotion labels recorded before recursive question generation (Figure 2(b)). Semantically equivalent emotions were treated as matches. We found that 551 questions (72.98%, Mismatched group) have different higher-order and lower-order answers, while only 204 (27.02%, Matched group) share the same answer.

MPAR accuracy (%) on the Matched and Mismatched groups.

ModelsAll questions (n=755)Matched group (n=204)Mismatched group (n=551)
AffToM-R150.6056.3748.46
Qwen3.5-27B45.0354.9041.38

As shown above, both models maintain accuracy well above chance (25%) on the Mismatched group, where directly reusing lower-order emotions would produce incorrect answers. These findings indicate that lower-order answer reuse cannot fully explain model performance and provide further evidence that AffToM evaluates reasoning beyond direct emotion attribution.

I thank the authors for the information provided above as well as response to other reviwers, which align with my expectation that strong property models do not really needs video in many multimodal reasoning benchmarks. However this do not diminish the value of this work, I therefore raise my score to 3.

Comments on the paper

3 comments

[1/3]

Thank you for the in-depth review. We respond to the eight points one by one below.

Response to W1: Construct Validity and Controlled Information Intervention

Lower-order answer shortcut. To examine whether MPAR questions can be answered by simply reusing lower-order emotions, we compared all 755 MPAR MCQs with their corresponding single-character emotion labels recorded before recursive question generation (Figure 2(b)). We found that 551 questions (72.98%, Mismatched group) have different higher-order and lower-order answers, while only 204 (27.02%, Matched group) share the same answer.

MPAR accuracy (%) on the Matched and Mismatched groups.

ModelsAll questions (n=755)Matched group (n=204)Mismatched group (n=551)
AffToM-R150.6056.3748.46
Qwen3.5-27B45.0354.9041.38

Both models achieve accuracy above chance (25%) on the Mismatched group, where directly reusing lower-order answers would be incorrect. This suggests that lower-order answer reuse cannot explain their performance on MPAR.

Additional experiment: controlled information intervention. We also ran an information intervention on 50 pre/post-event pairs. For each pre-event question, we fixed the truncated video (cut at the time point before the event, with 64 sampled frames), question, options, and inference settings, while adding hypothetical information about what a character knew, without revealing the target emotion.

Information intervention results for Qwen3.5-27B.

SettingAdded informationChoose Pre answerChoose Post answerOther
Pre–12/50 (24.0%)17/50 (34.0%)21/50 (42.0%)
Pre + new information✓8/50 (16.0%)28/50 (56.0%)14/50 (28.0%)

For Qwen3.5-27B, the proportion of responses selecting the post-event answer increased from 34% (17/50) to 56% (28/50) after the intervention, a gain of 22 percentage points. These findings provide complementary evidence that MPAR answers are sensitive to character-specific information and cannot be explained solely by lower-order emotion labels or nested question wording.

Response to W2: Ambiguity of Emotion Labels

Question review. We agree that emotion inference can be ambiguous and that agreement on question quality does not establish agreement on emotion labels. To address this, our review process checks whether the reference answer is better supported by the video than the distractors, considering the scene, time point, and character perspective. Ambiguous questions are identified during reflection and revision (§3.3) and further screened through four human-review criteria (Appendix C.1).

Additional human evaluation. To further assess answer reliability, we collected a second independent human response for each of the 250 human-baseline questions, with each question answered by two different annotators.

Human baseline across two rounds (%).

Question TypeRound 1Round 2
Preference Prediction91.3086.96
Integrated Mental-State Reasoning92.5992.59
Strategic Perspective-Taking90.4885.71
Hidden Intention95.0090.00
Personality Reasoning92.0092.00
Counterfactual Reasoning91.6791.67
Multi-Party Affective Reasoning84.5585.45
Overall88.8088.00

The results remain consistent across both rounds, with only a 0.80 percentage-point difference overall and a 0.90-point difference on MPAR. This provides additional evidence of stable human performance across annotator assignments. We will include the second-round results in the revised appendix.

Response to W3: Isolation of Training and Test Material

We thank the reviewer for highlighting the importance of source-level data isolation. Our training and evaluation sets were originally separated at the video level, ensuring that all questions from the same video belong to a single split. To further rule out potential data leakage, we conducted additional checks against both AffToM and R3R^3Bench.

Video and content overlap. We first compared source video IDs and found no overlap. To detect re-uploaded or duplicated footage, we further extracted frames every 2 seconds and applied pHash and dHash matching, followed by temporal alignment and manual verification.

ComparisonShared Video IDsReused Clips
Train vs. AffToM00
Train vs. R3R^3Bench00

No overlapping source video IDs or reused footage were detected in either evaluation set. These results provide additional evidence against train–test contamination at the video and clip level. We will include the verification procedure and results in the revised appendix.

[2/3]

Response to W4: Source of the Performance Gains

We thank the reviewer for this suggestion. We agree that the overall improvement should be distinguished from the contribution of the proposed rewards. To clarify this, we provide a breakdown of performance gains at each training stage.

Stage-wise accuracy of Qwen3.5-4B (%). Δ = change relative to the previous row (pp).

StageMPAROverallΔ MPARΔ Overall
Qwen3.5-4B (base)37.2249.29––
+ SFT47.0256.20+9.80+6.91
+ GRPO (Accuracy + Format rewards)47.2857.52+0.26+1.32
+ Full method (three added rewards)50.6059.33+3.32+1.81

As shown above, SFT improves MPAR by 9.80 percentage points, while standard GRPO provides an additional 0.26-point gain. Importantly, our three proposed rewards further improve MPAR by 3.32 points and overall accuracy by 1.81 points over standard GRPO. We will revise the abstract and Table 3 to clearly distinguish the 13.38-point total MPAR improvement from the 3.32-point contribution of the proposed rewards.

Response to W5: Statistical Reliability

We thank the reviewer for raising this concern. To assess the reliability of our results, we conducted two additional analyses: (1) a second training seed (1106) for all five reward configurations, and (2) video-level statistical tests, treating the 258 source videos as independent clusters.

1. Multiple training seeds.

Seed-1106 accuracy (%).

RperspR_{\mathrm{persp}}RfaithR_{\mathrm{faith}}RconsR_{\mathrm{cons}}IARMPAROverall
–––64.2047.1557.14
✓––64.3947.9557.57
✓✓–64.6750.2058.67
✓–✓64.8548.6158.12
✓✓✓65.1450.8659.22

Seed-42 accuracy (%).

RperspR_{\mathrm{persp}}RfaithR_{\mathrm{faith}}RconsR_{\mathrm{cons}}IARMPAROverall
–––64.7647.2857.52
✓––64.8548.0857.90
✓✓–65.0449.8058.73
✓–✓65.3249.0158.56
✓✓✓65.5150.6059.33

2. Video-level statistical analysis (seed-42). We further computed 95% confidence intervals using 10,000 video-clustered bootstrap resamples and conducted two-sided paired permutation tests with 100,000 permutations. The full method improves overall accuracy from 57.52% to 59.33%, with a reported paired-test p-value of 0.01.

RperspR_{\mathrm{persp}}RfaithR_{\mathrm{faith}}RconsR_{\mathrm{cons}}IARMPAROverallp-value against base GRPO
–––64.76 [61.76, 67.74]47.28 [43.93, 50.62]57.52 [55.27, 59.78]—
✓––64.85 [61.81, 67.87]48.08 [44.63, 51.51]57.90 [55.62, 60.17]0.02
✓✓–65.04 [62.03, 68.12]49.80 [46.34, 53.23]58.73 [56.55, 60.87]0.03
✓–✓65.32 [62.37, 68.28]49.01 [45.63, 52.53]58.56 [56.35, 60.80]0.04
✓✓✓65.51 [62.34, 68.67]50.60 [46.95, 54.19]59.33 [56.93, 61.74]0.01

Together, these results show that the full reward configuration performs consistently better across two training seeds, with additional statistical evidence supporting its improvement over standard GRPO.

[3/3]

Response to W6: Validation of the Automatic Evaluator

We thank the reviewer for this suggestion. To strengthen the evaluation reliability, we conducted an additional validation with three independent human annotators and two LLM judges (GPT-5-mini and GPT-5.5). All evaluators scored the same 100 responses from Qwen3.5-27B using identical Emotion Correctness (EC) and Factual Consistency (FC) rubrics.

Inter-annotator agreement.

Human AnnotatorsECFC
All Annotators, ICC(3,k)0.96520.9623

Agreement between each annotator and the model judges. The table gives the Spearman correlation of the two judges with each annotator and with the mean of the human scores:

LLM-as-a-judge validation. Spearman ρ with human scores; Annotator Mean averages three human annotators.

LLM JudgeMetricAnnotator 1Annotator 2Annotator 3Annotator Mean
GPT-5-miniEC0.76660.76560.73300.7778
FC0.72680.76270.72850.7548
GPT-5.5EC0.81080.84170.76630.8300
FC0.74210.81130.77890.7963

Human annotators show high agreement (ICC > 0.96), while both LLM judges correlate strongly with the mean human scores (Spearman ρ = 0.75–0.83). Moreover, correlations with individual annotators remain above 0.72 across both dimensions.

These results provide stronger evidence that our automatic evaluation aligns with human judgments and is not dependent on a single annotator or LLM judge. We will include the complete agreement results in the revised appendix.

Response to W7: Pre- and Post-Event Question Difficulty

We thank the reviewer for this concern. To examine whether the performance gap is simply due to differences in question difficulty, we conducted a same-question truncation experiment on 210 pre-event questions. We kept the questions, answer options, and reference answers unchanged, comparing the full video with a video truncated before the question event. Both settings used 64 uniformly sampled frames.

*Pre-event accuracy with the full video and the truncated video (n=210, %).

ModelFull VideoTruncated VideoImprovement
Qwen3.5-4B30.0031.90+1.90
AffToM-R139.0542.86+3.81
Qwen3.5-9B28.1032.38+4.28
Qwen3.5-27B27.6237.62+10.00

All four models perform better when post-event footage is removed, with improvements ranging from 1.90 to 10.00 percentage points. Since the questions and answers remain identical, these results suggest that later events can interfere with reasoning about earlier mental states. Although this control does not fully eliminate intrinsic difficulty differences between pre- and post-event questions, it demonstrates that access to more video information does not necessarily make earlier-state reasoning easier.

Response to W8: Presentation and Clarity

We will clarify that AffToM contains 1,922 evaluation questions, comprising 1,822 multiple-choice questions and 100 open-ended questions, and use these numbers consistently throughout the paper. We will also improve figure readability, clarify axis labels, and standardize table formatting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.