- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SMORE, a multimodal benchmark containing 427 minimal video pairs, 854 videos, and 1,281 evaluation questions covering belief-conditioned rationality, goal-directed social reasoning, and moral judgments. There are 350 pairs produced by VirtualHome and 77 moral-reasoning pairs generated by Gemini Omni. Evaluation reports paired-video accuracy (P), independent single-video accuracy (S), and Joint-Pair Accuracy (J), which requires both members of a pair to be classified correctly. Human majority-vote consensus achieves 96.0% overall J, compared with 63.2% for the strongest evaluated model. Additional analyses examine agent-favourable prediction asymmetries, the effects of goal achievement and path efficiency on rationality judgments, and the linear decodability of benchmark labels from internal representations.
Critical strengths
-
The controlled behavioral contrasts have clear diagnostic value.Evaluating relative discrimination, absolute judgments, and correctness of both pair members helps determine whether model judgments consistently respond to behavior that changes the label.
-
The evaluation is broad and includes useful reliability checks. Human evaluation covers all questions, with three independent responses per question from a pool of 65 annotators and reported Fleiss' κ values. The experiments include specialized human-behavior models beyond general-purpose models, answer-order checks, and five-fold probe evaluation that keeps pair members together, with inner cross-validation for regularization selection. These choices strengthen the credibility of the benchmark scores and behavioral asymmetries and increase the practical value of the evaluation.
Critical weaknesses
-
Limited domain coverage. All videos are rendered in VirtualHome or generated with Gemini Omni. The VirtualHome portion uses limited scenario templates and scripted actions. This introduces a potential domain gap from naturalistic videos, and transfer remains unverified.
-
Class imbalance in conditional subsets weakens the probe conclusions. Subsets selected by model correctness are not necessarily class-balanced. Thus, high raw accuracy does not establish strong decodability on error cases. Balanced accuracy and per-class recall are needed to reassess the cross-domain conclusions.
Questions for authors
-
Could the authors evaluate a small set (due to time constraints) of real-world videos with comparable reasoning scenarios to assess whether the reported failure patterns generalize beyond the synthetic domains? If consistent failure patterns persist on these videos, I would consider raising my score.
-
Can authors use the existing out-of-fold probe predictions to report class counts, majority-class baselines, balanced accuracy, per-class recall, and uncertainty for each model-correct/model-wrong subset in Table 2? Does the contrast between retained social-label decodability and reduced rationality-label decodability survive under these metrics?
4 comments
Response to Question 2 (2/2)
Q2. Class imbalance in probe analysis
We thank the reviewer for highlighting this concern. We agree that class imbalance within the model-correct and model-wrong subsets can make raw probe accuracy misleading. We therefore re-examined the existing out-of-fold probe predictions and report class counts, majority-class baselines, balanced accuracy, per-class recall, and 95% confidence intervals below.
Evaluation protocol. Following Table 2, we use the final-layer Last Query Token representation with grouped 5-fold cross-validation, ensuring both videos of each minimal pair belong to the same fold. We use the held-out predictions from the same cross-validated probes and partition videos according to whether the model’s own single-video prediction is correct or incorrect. No separate probes are trained on these subsets. We compute 95% percentile confidence intervals using 10,000 bootstrap resamples of minimal pairs. All values are percentages.
The majority-class baselines illustrate the class imbalance within the model-correct and model-wrong subsets. However, the probes are trained on the full balanced dataset, without access to the model's correctness, and are not refitted on these subsets. Balanced accuracy accounts for class imbalance by giving equal weight to both classes, while per-class recall reports performance on each class separately.
1. Goal-Directed Social Reasoning (Help/Hinder). Representative results below; full five-model results are in Appendix A.11.3 (Table 12).
| Model | Subset | Help/Hinder | Majority | Probe Acc. [95% CI] | Balanced Acc. [95% CI] | Recall Help [95% CI] | Recall Hinder [95% CI] |
|---|---|---|---|---|---|---|---|
| InternVL3.5-8B Inst | Model Correct | 132/20 | 86.8 | 79.6 [73.1, 85.6] | 79.8 [69.6, 88.5] | 79.5 [72.4, 86.1] | 80.0 [60.9, 95.7] |
| Model Wrong | 18/130 | 87.8 | 77.7 [70.8, 84.2] | 72.9 [60.8, 84.3] | 66.7 [42.9, 88.0] | 79.2 [72.1, 86.0] | |
| Qwen3.5-9B Think | Model Correct | 125/52 | 70.6 | 84.2 [78.7, 89.4] | 84.9 [79.0, 90.3] | 83.2 [76.5, 89.5] | 86.5 [76.6, 95.3] |
| Model Wrong | 25/98 | 79.7 | 83.7 [77.0, 89.9] | 83.8 [75.1, 91.3] | 84.0 [68.2, 96.3] | 83.7 [76.0, 90.5] | |
| Qwen3.5-27B Think | Model Correct | 123/83 | 59.7 | 89.3 [84.8, 93.4] | 89.1 [84.3, 93.4] | 90.2 [84.7, 95.2] | 88.0 [80.5, 94.6] |
| Model Wrong | 27/67 | 71.3 | 83.0 [75.2, 90.0] | 84.7 [76.9, 91.5] | 88.9 [75.0, 100.0] | 80.6 [70.5, 89.4] |
2. Belief-Conditioned Rationality (Rational/Irrational). Representative results below; full five-model results are in Appendix A.11.3 (Table 13).
| Model | Subset | Rational/Irrational | Majority | Probe Acc. [95% CI] | Balanced Acc. [95% CI] | Recall Rational [95% CI] | Recall Irrational [95% CI] |
|---|---|---|---|---|---|---|---|
| InternVL3.5-8B Inst | Model Correct | 157/66 | 70.4 | 64.6 [58.5, 70.5] | 66.5 [60.1, 72.7] | 61.8 [54.1, 69.3] | 71.2 [60.0, 81.8] |
| Model Wrong | 43/134 | 75.7 | 50.3 [43.2, 57.8] | 47.4 [39.0, 56.0] | 41.9 [27.3, 56.5] | 53.0 [44.6, 61.4] | |
| Qwen3.5-9B Think | Model Correct | 175/110 | 61.4 | 70.5 [65.0, 75.9] | 71.4 [65.8, 76.9] | 67.4 [60.5, 74.4] | 75.5 [67.3, 83.3] |
| Model Wrong | 25/90 | 78.3 | 58.3 [49.1, 66.9] | 58.9 [47.7, 69.7] | 60.0 [40.0, 79.3] | 57.8 [47.2, 67.8] | |
| Qwen3.5-27B Think | Model Correct | 188/102 | 64.8 | 80.3 [75.4, 85.0] | 82.4 [77.7, 86.8] | 75.5 [69.3, 81.6] | 89.2 [82.8, 94.9] |
| Model Wrong | 12/98 | 89.1 | 57.3 [48.5, 66.1] | 54.1 [39.1, 69.0] | 50.0 [20.0, 80.0] | 58.2 [48.4, 67.9] |
Interpretation. The balanced accuracy results, which account for class imbalance, support the conclusions in Section 4.2 of our paper.
For Goal-Directed Social Reasoning, the Help/Hinder distinction remains strongly linearly decodable even when the model is wrong. Balanced accuracy on the model-wrong subset ranges from 72.9% to 84.7% across all five models, with all 95% confidence intervals above the 50% chance level.
For Belief-Conditioned Rationality, we observe a different pattern. Across the three Qwen models, balanced accuracy decreases from 71.4–82.4% on model-correct examples to 47.5–58.9% on model-wrong examples. InternVL3.5-8B Inst similarly drops from 66.5% to 47.4%, while LLaVA-OV-2-8B Inst remains close to chance in both subsets. All model-wrong confidence intervals include 50% (Table 13, Appendix A.11.3).
These results reinforce the distinction observed in Section 4.2: for Goal-Directed Social Reasoning, the Help/Hinder distinction can remain strongly linearly decodable even when the model produces the wrong answer, whereas for Belief-Conditioned Rationality, model errors largely coincide with a loss of linear decodability.
These results are now reported in Appendix A.11.3 of the revision (Tables 12 and 13), with the Table 2 caption pointing to them.
Thank authors for the detailed responses. The additional evaluation on naturalistic videos and the revised probe analysis using balanced accuracy address my concerns. I will therefore raise my score.
Thank you for your update. We are glad that our additional experiments and analyses addressed your concerns, and we thank you again for your constructive feedback.
Response to Question 1 (1/2)
Thank you for your constructive feedback. We address each of your questions below.
Q1. Generalization to real-world videos
We thank the reviewer for raising this important concern about generalization to natural videos.
We designed SMORE around simulated and generated videos because they allow precise control over minimal pairs, which is difficult to guarantee with real-world footage. To test generalization to naturalistic settings, we extended SMORE using EPIC-KITCHENS-100 [1], a dataset of unscripted, egocentric recordings of everyday kitchen activities. Using timestamped action annotations, we identified and manually verified clips in which a participant opens a container and retrieves an object. We then specified the agent's goal and belief following Section 2.1, constructing 34 True Belief and 34 False Belief minimal pairs (68 pairs; 136 videos) across 7 distinct kitchens. Both videos within each pair come from the same continuous recording, preserving the participant, kitchen, lighting, and recording session. The goal and stated belief remain fixed while the action changes, preserving the minimal-pair contrast. We focus on these two subclasses because the path-based scenarios require controlled route manipulations that are difficult to obtain from natural footage. Full construction details are provided in Appendix A.13.1.
The videos are available at: https://anonymous.4open.science/r/smore-natural-F48B
Comparing the same 12 models across synthetic (VirtualHome) and natural (EPIC-KITCHENS-100) videos on True Belief and False Belief combined, model rankings based on Joint-Pair Accuracy remain highly consistent (Spearman's rank correlation ρ = 0.93), with Gemini-3.1-Pro and Gemini-3.5-Flash remaining the top two models in both settings.
Table: Rationality results on natural videos (%). Paired Accuracy (P, 50% chance), Single-video Accuracy (S, 50% chance), and Joint-Pair Accuracy (J, 25% chance). Human scores use three independent annotations per question from a pool of 9 annotators. Full results for all 12 models are in Appendix A.13 (Table 16).
These results reproduce several key failure patterns reported in Section 3. We denote Paired Accuracy, Single-video Accuracy, and Joint-Pair Accuracy as P, S, and J, respectively.
Overall, these findings provide evidence that several central failure patterns identified in SMORE persist on real-world videos. Humans remain near ceiling (Joint-Pair Accuracy = 98.5%), while most models show substantially lower performance. While this extension is limited to belief-conditioned rationality, it provides an initial assessment of transfer to real-world settings.
The revision includes this extension in Appendix A.13 (Tables 16 and 17).
References
[1] Damen et al. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. IJCV, 2022.