- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AffToM to evaluate individual affective reasoning and multi-party affective reasoning (MPAR) in videos, with the latter focusing on nested belief and emotion attribution and how these attributions change with new information. The evaluation set contains 258 videos, 1,822 multiple-choice questions, and 100 open-ended questions, constructed through a multi-agent pipeline based on character interaction graphs and subsequently subjected to automatic screening and manual review. The authors also construct 19,759 training QA pairs and propose AffToM-R1, which combines perspective-structured SFT with rubric-guided GRPO. Under the default video-only setting, AffToM-R1 improves the MPAR accuracy of Qwen3.5-4B from 37.22% to 50.60% and reports gains on external benchmarks.
Critical strengths
-
The evaluation setting has clear value. MPAR combines affective attribution with nested character perspectives, and the paired pre- and post-event questions further examine changes in characters’ mental states after they receive new information.
-
The resource quality and training value are validated from multiple angles. Approximately 150 hours of manual review, annotation consistency checks, human baselines, and validation of the open-ended response scorer support data quality. The open-ended scoring also explicitly checks recursive perspective structure. Evaluations across multiple models and input conditions, a vanilla GRPO control, a dedicated reward ablation, and external benchmark results jointly support the resource’s usefulness and training value.
Critical weaknesses
None. I did not identify any critical weaknesses sufficient to warrant rejection; the points requiring clarification are given in the questions below.
Questions for authors
-
How many second- and third-order MPAR multiple-choice questions are there, respectively, and what are the model’s accuracies on each? Did the final review check whether some questions can be answered solely from event facts, belief contents, or lower-order character perspectives, without requiring full affective attribution? Could you provide the corresponding review criteria or examples?
-
What video-level split and near-duplicate checking rules were used between the training data and the AffToM and R³-Bench evaluation sets? How were clips from the same movie or the same episode handled, and what were the results of the content-overlap checks?
-
How is the 47.35% MPAR accuracy in Table 3 computed? It cannot be obtained by rounding a single-run integer number of correct answers over 755 questions; please specify the actual denominator and aggregation method. Are the complete QA-generation, SFT, and reward-scoring prompts, the actual contents corresponding to the knowledge-base and causal-rule placeholders in Appendix G, and the final data split configuration included in the supplementary materials?
-
Why do the main tables use video frames only (V) rather than video plus subtitles (V+T)? In Table 6, adding subtitles raises the MPAR accuracy of Qwen3.5-27B from 45.03% to 51.79%. Did the authors check whether the sampled frames (including any embedded subtitles) provide the dialogue evidence needed to answer the questions? For pre-event questions, are the sampled frames drawn from the full video or only from the segment before the queried time point?
4 comments
[2/2]
Response to Q4: Video Input Settings and Temporal Leakage
Why the main experiments use video-only frames (V). Thank you for raising this. The visual evidence AffToM tests (who is present, who witnesses a key event, who has not yet learned a fact, and the characters' expressions, gaze, and actions) exists only in the frames, and dialogue cannot replace it. V is the configuration least able to exploit language priors, so it best tests whether a model reasons from visual evidence. We do not hide the gain that subtitles bring. Table 6 reports V and V+T side by side (Qwen3.5-27B improves from 59.22% to 64.71%) and already includes V+T results for some models. We will add more models to Table 6.
Video-only setting. We adopted video-only (V) as the default because AffToM focuses on visual evidence, such as character presence, actions, and expressions. V is the configuration least able to exploit language priors, so it best tests whether a model reasons from visual evidence. However, we acknowledge that sampled frames may contain embedded subtitles and that dialogue can provide useful clues. To assess the contribution of visual information beyond subtitles, we conducted additional subtitle-only (T) experiments and compared T, V, and video-plus-subtitle (V+T) settings on the same test set.
| Model | T | V | V+T |
|---|---|---|---|
| Qwen3.5-4B | 36.16 | 37.22 | 40.40 |
| AffToM-R1 | 48.87 | 50.60 | 52.72 |
| Qwen3.5-27B | 43.05 | 45.03 | 51.79 |
| GPT-5.5 | 52.85 | 52.05 | 56.95 |
V+T consistently outperforms both single-modality settings, suggesting that visual and textual information provide complementary evidence for MPAR.
Frame sampling We also considered that uniform sampling may miss key evidence, so Appendix E.2 (Table 7) compares frame sampling strategies. With the frame budget and all other inference settings fixed, we replaced uniform sampling with two keyframe selection methods, BOLT [1] and WFS-SB [2], to see whether more targeted frame selection can capture key evidence and improve answers.
MPAR accuracy by frame sampling strategy (%).
| Models | Uniform | BOLT | WFS-SB |
|---|---|---|---|
| Qwen3-VL-32B | 40.79 | 44.90 | 44.37 |
| Qwen3.5-27B | 45.03 | 49.67 | 48.21 |
Both adaptive sampling strategies beat uniform sampling, and BOLT improves MPAR accuracy of the two models by 4.11 and 4.64 percentage points. This supports the value of keyframe selection for collecting the evidence that multi-party affective reasoning needs.
We also inspected the 64-frame uniform inputs of 10 questions, and 9 of them kept the key evidence needed to answer. In the video of Figure 10(a), for example, a man wears a head covering against the cold while choosing a kitchen knife, and a woman mistakes him for a robber. Frame 3 ("With this on I won't be cold riding") gives his real purpose, and frame 10 ("You bad man"), together with the woman's spraying and frightened reaction, shows his inference about how she sees him.
Frame range for pre-event questions. We extract the sampled frames from the complete video for all questions, so pre-event and post-event questions have identical frame sets.
Temporal leakage. In the original evaluation, pre- and post-event questions use frames sampled from the full video, with the target time specified in the question. We acknowledge that this may expose models to information unavailable to the character at the target time. To investigate this concern, we conducted a same-question truncation experiment on 210 pre-event questions, keeping the questions and answers unchanged.
| Model | Full Video | Truncated Video | Improvement |
|---|---|---|---|
| Qwen3.5-4B | 30.00 | 31.90 | +1.90 |
| AffToM-R1 | 39.05 | 42.86 | +3.81 |
| Qwen3.5-9B | 28.10 | 32.38 | +4.28 |
| Qwen3.5-27B | 27.62 | 37.62 | +10.00 |
All four evaluated models improve when later footage is removed (+1.90 to +10.00 percentage points). This suggests that post-event information can interfere with judgments about earlier mental states. These findings suggest that current models struggle to reason from a character's time-specific knowledge rather than all information available in the video. In particular, additional context may introduce hindsight bias instead of improving mental-state reasoning. This highlights an important challenge captured by AffToM: models must not only understand social events but also track who knows what and when. We will clarify this interpretation and the limitations of the truncation control in the revised paper.
[1] Liu, S., Zhao, C., Xu, T., and Ghanem, B. BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding. CVPR 2025.
[2] Chen, W., Zeng, Y., Luo, Y., Xie, T., Lin, L., Ji, J., Zhang, Y., and Zheng, X. Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding. CVPR 2026.
The author provided responses regarding "Distribution of MPAR Recursion Orders" and "Data Isolation," which cleared up some of my confusion.
The author corrected an issue where samples with incorrect responses were excluded from scoring, and re-evaluated the scores. The revised results are consistent with the experimental trends and conclusions presented earlier in the paper.
The authors provided an additional comparison between the text-only, vision-only, and text+vision modalities, which addressed my question. The experimental results show that the text-only and vision-only modalities perform at roughly the same level, while combining the modalities leads to improved performance.
The author’s mention of the presence of explicit subtitles in the videos has sparked a new line of inquiry for me: In the practical application of an "affective theory of mind," the input typically consists of video alone, without subtitles. However, it might be worth exploring the addition of audio-modality inputs to conduct further experiments.
By comparing different frame sampling strategies and providing case studies, the authors answered my question: "Did the authors check whether the sampled frames (including any embedded subtitles) provide the dialogue evidence needed to answer the questions?"
Based on feedback and experimental results, the author identifies a significant challenge: "models must not only understand social events but also track who knows what and when." This insight offers valuable direction for future research in related fields, particularly concerning personal agents and human-computer interaction.
I have a positive view of this work and recommend its acceptance.
[1/2]
Thank you for the specific questions. We respond to the four questions in turn below.
Response to Q1: Distribution of MPAR Recursion Orders and Genuine Reasoning Demand
Recursion orders. AffToM contains 755 second-order MPAR multiple-choice questions and no third-order multiple-choice questions. The 100 third-order questions are open-ended, evaluated using Emotion Correctness (EC) and Factual Consistency (FC).
Question review. Human reviewers assess each question based on answer plausibility, video dependency, distractor quality, and question validity (Appendix C.1). Reference answers must be supported by the video from the specified character's perspective and at the specified time, rather than relying on information unavailable to that character.
Lower-order shortcut analysis. To examine whether second-order questions can be answered by directly reusing lower-order emotions, we compared all 755 MPAR answers with their corresponding single-character emotion labels recorded during dataset construction (Figure 2(b)). 551 questions (72.98%, Mismatched group) have different higher-order and lower-order answers, while 204 (27.02%, Matched group) have matching answers.
MPAR accuracy (%) on the Matched and Mismatched groups.
Both models achieve substantial accuracy on the Mismatched group, where directly copying the lower-order emotion would be always incorrect. These results show that lower-order answer reuse cannot fully explain model performance.
Response to Q2: Train–Test Data Isolation
Thank you for this important question. Our dataset uses a video-level split, ensuring that all questions from the same video belong to a single split. To further check for potential data leakage, we compared the training set against both AffToM and R3Bench at three levels:
Train–test overlap analysis.
No overlapping video IDs or reused clips were detected. These results provide additional evidence against train–test contamination at the video and clip level. We will include the verification procedure and results in the revised appendix.
Response to Q3: Accuracy Calculation and Reproducibility
Accuracy correction. Thank you for identifying this inconsistency. We found that our evaluation script mistakenly excluded unparsable responses (ERROR) from the accuracy denominator rather than counting them as incorrect. For example, the first-row MPAR accuracy was calculated as 357/754 = 47.35%, whereas the correct value is 357/755 = 47.28%.
We have corrected five affected cells in the first two rows of Table 3 and rechecked the remaining results, finding no further discrepancies. The corrected results are shown below.
Importantly, the corrected results preserve our main finding: the full method outperforms standard GRPO by 3.32 percentage points on MPAR and 1.81 points overall. We apologize for the oversight and will correct the table and related text.
Reproduction materials. We will provide the actual content of the knowledge base and causal rules in Appendix G, the complete data-generation prompts, the sample splits of the SFT and PAFRO subsets, and the prompts used for the supplementary reward scoring.