acceptodds
Under review as a conference paper at ICLR 2027

SMORE: Minimal-Pair Videos for Social, Moral and Rational Reasoning in MLLMs

Abstract

Understanding human behavior involves interpreting actions in relation to agents’ beliefs, goals and social norms. Small differences in behavior can change these interpretations even when the surrounding context remains similar. To test whether multimodal large language models (MLLMs) reliably distinguish such cases, we introduce SMORE, a minimal-pair benchmark for evaluating Social, MOral, and Rational reasoning. It comprises 427 video pairs (854 videos) and 1,281 evaluation questions spanning belief-conditioned rationality, helping and hindering and judgements of intentionality, responsibility and norm violations. Videos within each pair share the same actors, environment, and overall interaction while differing in a single socially-relevant factor that changes the target judgement. This design probes whether model judgements track relevant behavioral differences under closely matched conditions. We evaluate models under two settings. First, we have paired-video evaluation, where both videos from a minimal pair are presented jointly to measure relative discrimination performance. Second, we have single-video evaluation, where each video is evaluated independently to measure absolute performance. From the second setting, we additionally compute Joint-Pair Accuracy, which gives credit only when both videos in a pair are individually answered correctly. On SMORE, humans reach 96% Joint-Pair Accuracy while the best model, Gemini-3.5-Flash, reaches only 63.2%. These results suggest that, despite strong performance on video action understanding, current MLLMs still fall substantially short of humans in reasoning about the subtle differences in human behavior that determine social, moral, and rational judgments.

Primary area:
applications to neuroscience & cognitive science
Keywords:
multimodal large language models, video understanding, social reasoning, moral reasoning, rational reasoning, minimal pairs, social cognition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oQGrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SMORE, a multimodal benchmark containing 427 minimal video pairs, 854 videos, and 1,281 evaluation questions covering belief-conditioned rationality, goal-directed social reasoning, and moral judgments. There are 350 pairs produced by VirtualHome and 77 moral-reasoning pairs generated by Gemini Omni. Evaluation reports paired-video accuracy (P), independent single-video accuracy (S), and Joint-Pair Accuracy (J), which requires both members of a pair to be classified correctly. Human majority-vote consensus achieves 96.0% overall J, compared with 63.2% for the strongest evaluated model. Additional analyses examine agent-favourable prediction asymmetries, the effects of goal achievement and path efficiency on rationality judgments, and the linear decodability of benchmark labels from internal representations.

Critical strengths
  • The controlled behavioral contrasts have clear diagnostic value.Evaluating relative discrimination, absolute judgments, and correctness of both pair members helps determine whether model judgments consistently respond to behavior that changes the label.

  • The evaluation is broad and includes useful reliability checks. Human evaluation covers all questions, with three independent responses per question from a pool of 65 annotators and reported Fleiss' κ values. The experiments include specialized human-behavior models beyond general-purpose models, answer-order checks, and five-fold probe evaluation that keeps pair members together, with inner cross-validation for regularization selection. These choices strengthen the credibility of the benchmark scores and behavioral asymmetries and increase the practical value of the evaluation.

Critical weaknesses
  • Limited domain coverage. All videos are rendered in VirtualHome or generated with Gemini Omni. The VirtualHome portion uses limited scenario templates and scripted actions. This introduces a potential domain gap from naturalistic videos, and transfer remains unverified.

  • Class imbalance in conditional subsets weakens the probe conclusions. Subsets selected by model correctness are not necessarily class-balanced. Thus, high raw accuracy does not establish strong decodability on error cases. Balanced accuracy and per-class recall are needed to reassess the cross-domain conclusions.

Questions for authors
  • Could the authors evaluate a small set (due to time constraints) of real-world videos with comparable reasoning scenarios to assess whether the reported failure patterns generalize beyond the synthetic domains? If consistent failure patterns persist on these videos, I would consider raising my score.

  • Can authors use the existing out-of-fold probe predictions to report class counts, majority-class baselines, balanced accuracy, per-class recall, and uncertainty for each model-correct/model-wrong subset in Table 2? Does the contrast between retained social-label decodability and reduced rationality-label decodability survive under these metrics?

4 comments
AuthorsOpenReview ↗

Response to Question 1 (1/2)

Thank you for your constructive feedback. We address each of your questions below.

Q1. Generalization to real-world videos

We thank the reviewer for raising this important concern about generalization to natural videos.

We designed SMORE around simulated and generated videos because they allow precise control over minimal pairs, which is difficult to guarantee with real-world footage. To test generalization to naturalistic settings, we extended SMORE using EPIC-KITCHENS-100 [1], a dataset of unscripted, egocentric recordings of everyday kitchen activities. Using timestamped action annotations, we identified and manually verified clips in which a participant opens a container and retrieves an object. We then specified the agent's goal and belief following Section 2.1, constructing 34 True Belief and 34 False Belief minimal pairs (68 pairs; 136 videos) across 7 distinct kitchens. Both videos within each pair come from the same continuous recording, preserving the participant, kitchen, lighting, and recording session. The goal and stated belief remain fixed while the action changes, preserving the minimal-pair contrast. We focus on these two subclasses because the path-based scenarios require controlled route manipulations that are difficult to obtain from natural footage. Full construction details are provided in Appendix A.13.1.

The videos are available at: https://anonymous.4open.science/r/smore-natural-F48B

Comparing the same 12 models across synthetic (VirtualHome) and natural (EPIC-KITCHENS-100) videos on True Belief and False Belief combined, model rankings based on Joint-Pair Accuracy remain highly consistent (Spearman's rank correlation ρ = 0.93), with Gemini-3.1-Pro and Gemini-3.5-Flash remaining the top two models in both settings.

Table: Rationality results on natural videos (%). Paired Accuracy (P, 50% chance), Single-video Accuracy (S, 50% chance), and Joint-Pair Accuracy (J, 25% chance). Human scores use three independent annotations per question from a pool of 9 annotators. Full results for all 12 models are in Appendix A.13 (Table 16).

ModelTrue Belief P/S/JFalse Belief P/S/JOverall P/S/J
Human100 / 100 / 100100 / 98.5 / 97.1100 / 99.3 / 98.5
InternVL3.5-8B Inst85.3 / 67.6 / 35.355.9 / 50.0 / 11.870.6 / 58.8 / 23.5
Qwen3.5-9B Inst100 / 80.9 / 61.811.8 / 48.5 / 5.955.9 / 64.7 / 33.8
Qwen3.5-9B Think97.1 / 86.8 / 73.588.2 / 80.9 / 61.892.6 / 83.8 / 67.6
Gemini-3.1-Pro100 / 97.1 / 94.1100 / 95.6 / 94.1100 / 96.3 / 94.1
Gemini-3.5-Flash100 / 98.5 / 97.197.1 / 92.6 / 85.398.5 / 95.6 / 91.2
GPT5.6 Luna100 / 83.8 / 67.682.4 / 77.9 / 55.991.2 / 80.9 / 61.8

These results reproduce several key failure patterns reported in Section 3. We denote Paired Accuracy, Single-video Accuracy, and Joint-Pair Accuracy as P, S, and J, respectively.

  1. Single-video Accuracy masks poor pair-level accuracy (3.1). Humans show only a small gap between S = 99.3% and J = 98.5%. For models, the discrepancy is substantially larger. On False Belief, GPT5.6 Luna drops from S = 77.9% to J = 55.9%, while InternVL3.5-8B Inst achieves S = 50.0% but J = 11.8%, substantially below the 25% chance level. Thus, S alone can obscure systematic failures to consistently classify both members of a minimal pair.
  2. Comparing both videos can make the contrast easier (3.2). Eight of twelve models achieve higher P than S overall. For example, GPT5.6 Luna achieves P = 91.2% versus S = 80.9%, and Qwen3.5-9B Think achieves P = 92.6% versus S = 83.8%. Humans, by contrast, remain near ceiling in both settings (P = 100%, S = 99.3%).
  3. Models show an agent-favourable prediction asymmetry (3.3). All twelve models perform better on Rational than Irrational examples (e.g., InternVL3.5-8B Inst: 94.1% vs. 23.5%); humans are nearly symmetric (100.0% vs. 98.5%). Full results: Appendix A.13 (Table 17).
  4. Goal achievement influences rationality judgments (3.4). All six Qwen variants exhibit lower P on False Belief than True Belief. For example, Qwen3.5-9B Inst drops from 100.0% to 11.8%. Humans, by contrast, achieve P = 100% on both conditions. This drop suggests that the Qwen models partly rely on goal completion when judging rationality.

Overall, these findings provide evidence that several central failure patterns identified in SMORE persist on real-world videos. Humans remain near ceiling (Joint-Pair Accuracy = 98.5%), while most models show substantially lower performance. While this extension is limited to belief-conditioned rationality, it provides an initial assessment of transfer to real-world settings.

The revision includes this extension in Appendix A.13 (Tables 16 and 17).

References

[1] Damen et al. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. IJCV, 2022.

AuthorsOpenReview ↗

Response to Question 2 (2/2)

Q2. Class imbalance in probe analysis

We thank the reviewer for highlighting this concern. We agree that class imbalance within the model-correct and model-wrong subsets can make raw probe accuracy misleading. We therefore re-examined the existing out-of-fold probe predictions and report class counts, majority-class baselines, balanced accuracy, per-class recall, and 95% confidence intervals below.

Evaluation protocol. Following Table 2, we use the final-layer Last Query Token representation with grouped 5-fold cross-validation, ensuring both videos of each minimal pair belong to the same fold. We use the held-out predictions from the same cross-validated probes and partition videos according to whether the model’s own single-video prediction is correct or incorrect. No separate probes are trained on these subsets. We compute 95% percentile confidence intervals using 10,000 bootstrap resamples of minimal pairs. All values are percentages.

The majority-class baselines illustrate the class imbalance within the model-correct and model-wrong subsets. However, the probes are trained on the full balanced dataset, without access to the model's correctness, and are not refitted on these subsets. Balanced accuracy accounts for class imbalance by giving equal weight to both classes, while per-class recall reports performance on each class separately.

1. Goal-Directed Social Reasoning (Help/Hinder). Representative results below; full five-model results are in Appendix A.11.3 (Table 12).

ModelSubsetHelp/HinderMajorityProbe Acc. [95% CI]Balanced Acc. [95% CI]Recall Help [95% CI]Recall Hinder [95% CI]
InternVL3.5-8B InstModel Correct132/2086.879.6 [73.1, 85.6]79.8 [69.6, 88.5]79.5 [72.4, 86.1]80.0 [60.9, 95.7]
Model Wrong18/13087.877.7 [70.8, 84.2]72.9 [60.8, 84.3]66.7 [42.9, 88.0]79.2 [72.1, 86.0]
Qwen3.5-9B ThinkModel Correct125/5270.684.2 [78.7, 89.4]84.9 [79.0, 90.3]83.2 [76.5, 89.5]86.5 [76.6, 95.3]
Model Wrong25/9879.783.7 [77.0, 89.9]83.8 [75.1, 91.3]84.0 [68.2, 96.3]83.7 [76.0, 90.5]
Qwen3.5-27B ThinkModel Correct123/8359.789.3 [84.8, 93.4]89.1 [84.3, 93.4]90.2 [84.7, 95.2]88.0 [80.5, 94.6]
Model Wrong27/6771.383.0 [75.2, 90.0]84.7 [76.9, 91.5]88.9 [75.0, 100.0]80.6 [70.5, 89.4]

2. Belief-Conditioned Rationality (Rational/Irrational). Representative results below; full five-model results are in Appendix A.11.3 (Table 13).

ModelSubsetRational/IrrationalMajorityProbe Acc. [95% CI]Balanced Acc. [95% CI]Recall Rational [95% CI]Recall Irrational [95% CI]
InternVL3.5-8B InstModel Correct157/6670.464.6 [58.5, 70.5]66.5 [60.1, 72.7]61.8 [54.1, 69.3]71.2 [60.0, 81.8]
Model Wrong43/13475.750.3 [43.2, 57.8]47.4 [39.0, 56.0]41.9 [27.3, 56.5]53.0 [44.6, 61.4]
Qwen3.5-9B ThinkModel Correct175/11061.470.5 [65.0, 75.9]71.4 [65.8, 76.9]67.4 [60.5, 74.4]75.5 [67.3, 83.3]
Model Wrong25/9078.358.3 [49.1, 66.9]58.9 [47.7, 69.7]60.0 [40.0, 79.3]57.8 [47.2, 67.8]
Qwen3.5-27B ThinkModel Correct188/10264.880.3 [75.4, 85.0]82.4 [77.7, 86.8]75.5 [69.3, 81.6]89.2 [82.8, 94.9]
Model Wrong12/9889.157.3 [48.5, 66.1]54.1 [39.1, 69.0]50.0 [20.0, 80.0]58.2 [48.4, 67.9]

Interpretation. The balanced accuracy results, which account for class imbalance, support the conclusions in Section 4.2 of our paper.

For Goal-Directed Social Reasoning, the Help/Hinder distinction remains strongly linearly decodable even when the model is wrong. Balanced accuracy on the model-wrong subset ranges from 72.9% to 84.7% across all five models, with all 95% confidence intervals above the 50% chance level.

For Belief-Conditioned Rationality, we observe a different pattern. Across the three Qwen models, balanced accuracy decreases from 71.4–82.4% on model-correct examples to 47.5–58.9% on model-wrong examples. InternVL3.5-8B Inst similarly drops from 66.5% to 47.4%, while LLaVA-OV-2-8B Inst remains close to chance in both subsets. All model-wrong confidence intervals include 50% (Table 13, Appendix A.11.3).

These results reinforce the distinction observed in Section 4.2: for Goal-Directed Social Reasoning, the Help/Hinder distinction can remain strongly linearly decodable even when the model produces the wrong answer, whereas for Belief-Conditioned Rationality, model errors largely coincide with a loss of linear decodability.

These results are now reported in Appendix A.11.3 of the revision (Tables 12 and 13), with the Table 2 caption pointing to them.

Thank authors for the detailed responses. The additional evaluation on naturalistic videos and the revised probe analysis using balanced accuracy address my concerns. I will therefore raise my score.

Thank you for your update. We are glad that our additional experiments and analyses addressed your concerns, and we thank you again for your constructive feedback.

Reviewer_wBQMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SMORE, a benchmark of minimal-pair synthetic videos designed to evaluate whether multimodal LLMs can distinguish controlled behavioral differences in social scenarios. The benchmark contains 427 video pairs (854 videos), covering belief-conditioned rationality, goal-directed social reasoning, and moral reasoning. Based on these pairs, the authors introduce three evaluation metrics: single-video accuracy, paired-video accuracy, and joint-pair accuracy, which requires models to correctly classify both videos independently. Results reveal a substantial human–model gap (96% versus 63.2% joint-pair accuracy for Gemini 3.5 flash). he authors also identify substantial discrepancies between paired-video, single-video, and joint-pair accuracy, suggesting that models struggle to make consistent social judgments. Additionally, they observe systematic biases toward agent-favorable interpretations, and conducted additional linear-probing experiments suggesting that social distinctions could potentially be recovered from internal representations even when models produce incorrect predictions.

Critical strengths
  • the benchmark is well motivated, and the minimal-pair videos provide a controlled setting to isolate the evaluated models' understanding of socially relevant behavioral differences, rather than spurious context cues.
  • the evaluation is extensive, covering multiple open-source and proprietary multimodal LLMs, with comparisons against human baselines. The authors also provided detailed analysis on the model performances, including preliminary probing experiments
Critical weaknesses
  • the dataset is of limited scale, with each sub-class having around only 50 video pairs. Since the videos are synthetic, it's not immediately clear why the dataset couldn't be further expanded
  • the human validation of the generated videos is very limited -- human validation is only done on 13 "representative" pair of videos, out of the total 427 pairs. It is also not clear what the author mean by representative
  • the dataset taxonomy isn't immediately intuitive -- it's not clear why belief-conditioned rationality, goal-directed social reasoning, and moral reasoning consist of a sufficient coverage of human social behavior understanding. It is also not clear how each sub-class is chosen, and why some videos are generated in synthetic environments and some are from general video generation models
Questions for authors
  • could the authors provide a brief justification on why the current dataset scale is sufficient, or why further scaling dataset generation is difficult?
  • could the authors also provide evidence of more comprehensive human validation of the generated videos?
  • could the authors provide further clarification on the dataset taxonomy, as mentioned above?
  • a minor point, but as an interesting ablation experiment to disentangle visual perception from social reasoning, would there still be a gap between single video accuracy and joint-pair accuracy if the models are given only text-transcribed versions of the video pairs, rather than the videos themselves?
7 comments
AuthorsOpenReview ↗

Response to Question 1 (1/4)

We thank the reviewer for the helpful comments. Our responses to each question are below.

Q1. Dataset scale

We thank the reviewer for raising this point. While generating additional candidate videos is relatively straightforward, the main bottleneck is ensuring that each minimal pair is valid. All 427 pairs were manually verified to confirm that the intended behavioral difference was visible and that no other salient differences could affect the judgment. Failing VirtualHome pairs were regenerated, while Gemini Omni pairs were selected from multiple candidates. Furthermore, all 1,281 evaluation questions were answered by three independent annotators (3,843 annotations), providing a human baseline for every item. Expanding the dataset would increase these validation costs, alongside the cost of evaluating multiple proprietary and reasoning models.

Importantly, the current scale is sufficient to support our central findings. Bootstrap standard errors (Figs. 1 and 3; Table 6 in the revision) are at most 4.1 percentage points for domain-level Joint-Pair Accuracy in Belief-Conditioned Rationality and Goal-Directed Social Reasoning, and 7.6 points within subclasses. The corresponding gaps between humans and the best-performing model in each domain are substantially larger: 29.5 and 44.7 percentage points, respectively. Key subclass findings are similarly clear: Qwen3.5-9B-Think achieves only 3.4 ± 2.4% Joint-Pair Accuracy on Find, compared with 25% chance. Since standard errors decrease approximately as 1/√n, doubling the dataset (and its validation effort) would reduce them by only about 29%.

SMORE's 427 minimal pairs are comparable in scale to established contrastive diagnostic benchmarks, including Winoground (400 image–caption sets) and Vinoground (500 video–caption sets). Related social-video benchmarks also use evaluation sets of a few hundred videos: MuMA-ToM, also built using VirtualHome, evaluates on 225 videos, while R³-Bench evaluates on 316 videos.

AuthorsOpenReview ↗

Response to Question 2 (2/4)

Q2. Human validation of minimal pairs

We thank the reviewer for this suggestion and have conducted additional human validation covering the entire benchmark.

Extended human validation. We extended human validation to all 427 video pairs, using 14 annotators. For each pair, annotators first described how the two videos differed, before seeing the benchmark question. They then assessed whether the pair was minimal and whether it contained generation artifacts. Annotators judged 72.8% of pairs as minimal and 20.6% as mostly minimal, totaling 93.4%.

Visual artifacts were reported in 14.1% of pairs (60 of 427). However, model performance was similar on pairs with and without reported artifacts. Averaged across all 13 model configurations, paired accuracy was 72.7% on pairs with reported artifacts and 71.1% on pairs without them.

Clarification of "representative." The original 13 pairs were selected to illustrate the different subclasses: one pair from each VirtualHome subclass and two pairs from each Gemini Omni subclass. These were illustrative examples rather than a statistically representative sample. The extended validation now covers the entire benchmark.

We have updated the Appendix A.6.4 of the revision.

AuthorsOpenReview ↗

Response to Question 3 (3/4)

Q3. Dataset taxonomy

We thank the reviewer for this question. We agree that human social understanding is broader than these three domains, and we do not claim SMORE covers all of it. SMORE focuses on capacities that cognitive science treats as foundational and that can be tested with controlled minimal pairs.

Choice of domains. Developmental research suggests that humans reason about others from very early in life: infants interpret actions as goal-directed [1], expect agents to act rationally and efficiently [2, 3], take others' beliefs into account [4], and evaluate agents who help or hinder others [5]. These capacities are proposed to form an early-emerging core system for the social world [6, 7]. Rationality and social reasoning target this core directly, while the moral subclasses extend it to evaluations of intentions and outcomes: even infants weigh an agent's intention over the outcome of its action [8].

Choice of subclasses. Each subclass is designed so that a single visible change in behavior flips the correct judgment while everything else stays fixed. In belief-conditioned rationality, True Belief and False Belief distinguish rationality from goal achievement: under a true belief the rational agent succeeds, whereas under a false belief the rational agent fails and the irrational agent happens to succeed. Similarly, Believed Safe and Believed Risky distinguish rationality from path efficiency: the shortest path is rational when believed safe, whereas a longer detour is rational when the shorter path is believed risky (Section 3.4). In goal-directed social reasoning, Find, Open/Close, and On/Off capture common ways one agent can help or hinder another during everyday household activities, such as moving an object away from someone who needs it, closing a cabinet they intended to use, or switching off a device they were already using. In moral reasoning, given the breadth of moral cognition, we focus on three aspects of moral evaluation: the intention behind an action, an agent's response to a negative outcome they caused, and the distinction between moral and conventional norm violations [9–11].

Choice of pipelines. Rationality and social reasoning depend on exact object positions, paths taken, and action timing, which VirtualHome controls as explicit parameters; both videos of a pair differ in one judgment-relevant scripted action (Appendix A.6.1). Moral scenarios require behaviors such as accidents, remedying a harm, or norm violations, which fall outside a simulator's action set, so we use Gemini Omni with shared reference images and prompts that differ in one sentence (Appendix A.6.2). Each domain is generated entirely by one pipeline, and all metrics are computed within minimal pairs, so no comparison is made between videos from different pipelines.


References

[1] Woodward (1998). Infants selectively encode the goal object of an actor's reach. Cognition, 69(1), 1–34. [2] Gergely & Csibra (2003). Teleological reasoning in infancy: The naive theory of rational action. Trends in Cognitive Sciences, 7(7), 287–292. [3] Csibra et al. (1999). Goal attribution without agency cues: The perception of 'pure reason' in infancy. Cognition, 72(3), 237–267. [4] Onishi & Baillargeon (2005). Do 15-month-old infants understand false beliefs? Science, 308(5719), 255–258. [5] Hamlin, Wynn & Bloom (2007). Social evaluation by preverbal infants. Nature, 450(7169), 557–559. [6] Spelke & Kinzler (2007). Core knowledge. Developmental Science, 10(1), 89–96. [7] Carey (2009). The Origin of Concepts. Oxford University Press. [8] Hamlin (2013). Failed attempts to help and harm: Intention versus outcome in preverbal infants’ social evaluations. Cognition, 128(3), 451–474. [9] Killen & Rizzo (2014). Morality, intentionality, and intergroup attitudes. Behaviour, 151(2–3), 337–359. [10] Guglielmo (2015). Moral judgment as information processing: An integrative review. Frontiers in Psychology, 6, 1637. [11] White et al. (2017). Neural correlates of conventional and harm/welfare-based moral decision-making. Cognitive, Affective, & Behavioral Neuroscience, 17, 1114–1128.

Reviewer_dKa5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SMORE, a minimal-pair video benchmark for evaluating social, moral, and rational reasoning in MLLMs. The benchmark is composed of 427 video pairs (854 videos) and 1,281 evaluation questions across three reasoning domains: belief-conditioned rationality, goal-directed social reasoning, and moral reasoning. Each pair is designed to preserve the surrounding visual and interaction context while modifying a judgment-relevant behavioral factor.

The authors put forward three complementary evaluation metrics: Paired Accuracy, which measures relative discrimination when both videos are presented together; Single-video Accuracy, which evaluates independent judgments; and Joint-Pair Accuracy, which requires both videos in a pair to be classified correctly. Experiments with a range of open-source and proprietary MLLMs reveal a substantial gap from human performance, with humans achieving 96% Joint-Pair Accuracy compared to 63.2% for the best-performing model. The authors further identify systematic agent-favorable prediction biases and differences between relative and absolute judgment capabilities. Finally, linear probing experiments show that socially relevant distinctions can sometimes be recovered from internal model representations even when the models produce incorrect predictions.

Critical strengths
  1. The primary strength of this paper is its controlled minimal-pair design, which evaluates whether MLLM predictions respond appropriately to socially relevant behavioral differences while holding the surrounding context largely fixed. The benchmark spans belief-conditioned rationality, goal-directed social reasoning, and moral reasoning, providing broader coverage than many existing benchmarks focused on individual aspects of social understanding. Importantly, the authors introduce complementary Paired, Single-video, and Joint-Pair Accuracy metrics that distinguish relative discrimination from independent judgment. The experiments demonstrate that conventional single-video accuracy can substantially overestimate a model's ability to consistently recognize behavioral contrasts. For example, Qwen3.5-9B-Think achieves 51.7% Single-video Accuracy but only 3.4% Joint-Pair Accuracy on the Find task (Section 3.1). This highlights a meaningful evaluation gap and supports the usefulness of the proposed benchmark.

  2. The paper offers several interesting findings about the limitations of current MLLMs. Beyond documenting the substantial human–model performance gap, the authors identify an agent-favorable prediction asymmetry and show that models can perform much better when comparing two videos than when judging either video independently. The additional representation analysis is valuable: for goal-directed social reasoning, linear probes achieve 75.0–83.7% accuracy on examples where the corresponding models make incorrect predictions (Table 2). This suggests that the relevant distinction can remain accessible in model representations even when it is not reflected in the final prediction. The combination of behavioral evaluation, controlled comparisons, and representation-level analysis makes the study more informative than a conventional benchmark paper.

Critical weaknesses
  1. Despite the fact that the benchmark contains 427 minimal pairs, its scenarios are constructed through relatively constrained templates and two substantially different generation pipelines. Belief-conditioned rationality and goal-directed social reasoning rely on VirtualHome simulations, whereas moral reasoning examples are generated using Gemini Omni. The latter subset contains only 77 pairs, with some individual moral reasoning categories containing as few as 25 pairs. While the authors provide similarity analyses and human validation of the minimal-pair construction, the two pipelines offer different degrees of control, and residual generation artifacts may still influence model predictions. Furthermore, the use of simulated and synthetically generated scenarios makes it unclear how well the reported failure patterns generalize to more naturalistic videos involving complex social interactions. These concerns are particularly relevant to the paper's broad conclusions about MLLMs' social and moral reasoning abilities.

  2. The linear probing results are interesting, particularly the finding that social judgments remain decodable when models make incorrect predictions. However, linear decodability does not necessarily imply that models internally represent these distinctions in a way that can be directly used for reasoning or decision-making. The probes are trained with supervision on benchmark labels, and their success may partially reflect task-specific correlations or information that the model does not actually use during inference. Although the paper appropriately acknowledges this distinction, the analysis remains correlational. Consequently, the results do not fully establish whether the observed behavioral failures arise from deficient reasoning, ineffective utilization of existing representations, or limitations in the final prediction process. Additional controlled interventions or stronger probing baselines would help clarify this interpretation.

Questions for authors

1. Generalization beyond the constructed minimal-pair scenarios. How robust are the observed failure patterns to alternations in visual appearance, scene complexity, and the generation method? In particular, could the authors evaluate a small set of independently generated or real-world video pairs to examine whether the large human–model gap and agent-favorable prediction asymmetry persist outside the VirtualHome and Gemini Omni settings? This would help establish the external validity of the benchmark.

2. Interpretation of linear probing results. The results in Table 2 imply that Help/Hinder distinctions remain linearly decodable even when models answer incorrectly. Can the authors provide additional evidence that this information is relevant to the model's decision process rather than merely recoverable through a supervised classifier? For example, would intervention on the identified representations, or comparison against control-task probes, help determine whether the encoded distinctions causally influence model predictions?

3. Robustness of the moral reasoning evaluation. Given the comparatively small size of the moral reasoning subset and its reliance on a different video generation pipeline, how stable are the reported results across independently generated examples and different model families? Could the authors provide uncertainty estimates or additional controls specifically for this subset, including whether models from the same family as the video generator exhibit systematic performance differences? This would strengthen the conclusions regarding moral reasoning capabilities.

4 comments
AuthorsOpenReview ↗

Response to Question 1 (1/3)

Thank you for your thoughtful feedback. We respond to each of your questions individually below.


Q1. Generalization to real-world videos

We thank the reviewer for raising this important concern about external validity.

We designed SMORE around simulated and generated videos because they allow precise control over minimal pairs, which is difficult to guarantee with real-world footage. To test generalization to naturalistic settings, we extended SMORE using EPIC-KITCHENS-100 [1], a dataset of unscripted, egocentric recordings of everyday kitchen activities. Using timestamped action annotations, we identified and manually verified clips in which a participant opens a container and retrieves an object. We then specified the agent's goal and belief following Section 2.1, constructing 34 True Belief and 34 False Belief minimal pairs (68 pairs; 136 videos) across 7 distinct kitchens. Both videos within each pair come from the same continuous recording, preserving the participant, kitchen, lighting, and recording session. The goal and stated belief remain fixed while the action changes, preserving the minimal-pair contrast. We focus on these two subclasses because the path-based scenarios require controlled route manipulations that are difficult to obtain from natural footage. Full construction details are provided in Appendix A.13.1.

The videos are available at: https://anonymous.4open.science/r/smore-natural-F48B

Comparing the same 12 models across synthetic (VirtualHome) and natural (EPIC-KITCHENS-100) videos on True Belief and False Belief combined, model rankings based on Joint-Pair Accuracy remain highly consistent (Spearman's rank correlation ρ = 0.93), with Gemini-3.1-Pro and Gemini-3.5-Flash remaining the top two models in both settings.

Table: Rationality results on natural videos (%). Paired Accuracy (P, 50% chance), Single-video Accuracy (S, 50% chance), and Joint-Pair Accuracy (J, 25% chance). Human scores use three independent annotations per question from a pool of 9 annotators. Full results for all 12 models are in Appendix A.13 (Table 16).

ModelTrue Belief P/S/JFalse Belief P/S/JOverall P/S/J
Human100 / 100 / 100100 / 98.5 / 97.1100 / 99.3 / 98.5
InternVL3.5-8B Inst85.3 / 67.6 / 35.355.9 / 50.0 / 11.870.6 / 58.8 / 23.5
Qwen3.5-9B Inst100 / 80.9 / 61.811.8 / 48.5 / 5.955.9 / 64.7 / 33.8
Qwen3.5-9B Think97.1 / 86.8 / 73.588.2 / 80.9 / 61.892.6 / 83.8 / 67.6
Gemini-3.1-Pro100 / 97.1 / 94.1100 / 95.6 / 94.1100 / 96.3 / 94.1
Gemini-3.5-Flash100 / 98.5 / 97.197.1 / 92.6 / 85.398.5 / 95.6 / 91.2
GPT5.6 Luna100 / 83.8 / 67.682.4 / 77.9 / 55.991.2 / 80.9 / 61.8

These results reproduce several key failure patterns reported in Section 3. We denote Paired Accuracy, Single-video Accuracy, and Joint-Pair Accuracy as P, S, and J, respectively.

  1. Single-video Accuracy masks poor pair-level accuracy (3.1). Humans show only a small gap between S = 99.3% and J = 98.5%. For models, the discrepancy is substantially larger. On False Belief, GPT5.6 Luna drops from S = 77.9% to J = 55.9%, while InternVL3.5-8B Inst achieves S = 50.0% but J = 11.8%, substantially below the 25% chance level. Thus, S alone can obscure systematic failures to consistently classify both members of a minimal pair.
  2. Comparing both videos can make the contrast easier (3.2). Eight of twelve models achieve higher P than S overall. For example, GPT5.6 Luna achieves P = 91.2% versus S = 80.9%, and Qwen3.5-9B Think achieves P = 92.6% versus S = 83.8%. Humans, by contrast, remain near ceiling in both settings (P = 100%, S = 99.3%).
  3. Models show an agent-favourable prediction asymmetry (3.3). All twelve models perform better on Rational than Irrational examples (e.g., InternVL3.5-8B Inst: 94.1% vs. 23.5%); humans are nearly symmetric (100.0% vs. 98.5%). Full results: Appendix A.13 (Table 17).
  4. Goal achievement influences rationality judgments (3.4). All six Qwen variants exhibit lower P on False Belief than True Belief. For example, Qwen3.5-9B Inst drops from 100.0% to 11.8%. Humans, by contrast, achieve P = 100% on both conditions. This drop suggests that the Qwen models partly rely on goal completion when judging rationality.

Overall, these findings provide evidence that several central failure patterns identified in SMORE persist on real-world videos. Humans remain near ceiling (Joint-Pair Accuracy = 98.5%), while most models show substantially lower performance. While this extension is limited to belief-conditioned rationality, it provides an initial assessment of transfer to real-world settings.

The revision includes this extension in Appendix A.13 (Tables 16 and 17).

References

[1] Damen et al. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. IJCV, 2022.

AuthorsOpenReview ↗

Response to Question 2 (2/3)

Q2. Interpretation of linear probing results

We thank the reviewer for highlighting the distinction between linear decodability and causal use. As noted in Section 4.2, a successful probe establishes that a distinction is linearly recoverable from the representation, not that the model relies on it when answering. Our aim was to test whether socially relevant distinctions remain recoverable even when models answer incorrectly, rather than to attribute their decisions causally. To address the reviewer's question, we provide additional controls and an activation-steering intervention. Both are now included in the revision: the label-permutation baseline in Appendix A.11.4 and the activation-steering experiment in Appendix A.11.5 (Table 14).

Task-specific correlations. In SMORE, the apartment, objects, agents, and camera pose are identical within each minimal pair while the label flips, preventing these static properties from predicting the label. Both videos of each pair are assigned to the same cross-validation fold, ensuring evaluation on held-out pairs. Consistent with this, Help/Hinder distinctions are only moderately linearly decodable from Pooled ViT Tokens and become strongly linearly decodable in language-model representations (Figure 5).

Label-permutation baseline. As a sanity check, we retrained the probes after randomly keeping or swapping the Help/Hinder labels within each minimal pair, using the same representations and cross-validation protocol. With permuted labels, accuracy falls to chance for all five models (48.0–51.0%, compared with 78.7–87.3% with the original labels).

Causal intervention. We test whether steering representations along the probe direction influences model predictions, using Qwen3.5-9B on Help/Hinder (300 videos, 150 minimal pairs). This checkpoint supports both instruct and thinking modes. At layer 24 (75% of decoder depth), we add αu to the hidden state from the last query token onward, where α is a fixed steering strength. For each video, u is the normalized weight vector of the probe trained without that video (from the cross-validation fold in which it was held out), and we add or subtract it depending on whether this probe predicts Help or Hinder; the video’s ground-truth label is never used. As controls, we steer along two random directions orthogonalized against u, with the same α.

Table: Activation steering on Qwen3.5-9B in instruct mode (%). S = single-video accuracy; J = joint-pair accuracy. Random steering reports the mean across two random directions. Brackets indicate 95% bootstrap confidence intervals for improvements over the baseline.

Model / ConditionS (%)J (%)
Qwen3.5-9B Inst52.76.0
Qwen3.5-9B Inst + Probe Steering67.3 (+14.6 [11.0, 18.6])34.7 (+28.7 [21.3, 36.7])
Qwen3.5-9B Inst + Random Steering51.33.7

Steering along the probe direction increases J from 6.0% to 34.7% (+28.7 percentage points, 95% CI [21.3, 36.7]), while random steering provides no improvement (J = 3.7%). Probe steering maintains high Help recall while increasing Hinder recall from 6.7% to 38.7%. This shows that interventions along the probe-identified direction can influence the model's predictions, providing intervention-based evidence beyond linear decodability alone. In full thinking-mode generation (with a smaller α, as larger values led to truncated reasoning traces), steering raises J by only 4.7 points.

Together, the minimal-pair design and the label-permutation baseline indicate that the Help/Hinder distinction is linearly decodable from the internal representations, rather than from properties shared within a minimal pair or from arbitrary label assignments. The steering results further show that moving the representation along the probe direction improves the model’s predictions, consistent with one of the possibilities the reviewer raises: ineffective utilization of existing representations. We agree, however, that these experiments do not amount to a complete causal attribution of the model’s decisions, which remains a valuable avenue for future research.

AuthorsOpenReview ↗

Response to Question 3 (3/3)

Q3. Robustness of the moral reasoning evaluation

To test whether our moral reasoning results depend on the video generator, we generated 30 additional moral pairs with an independent generator, PixVerse [1] (10 each for Intentionality, Responsibility Taking, and Moral/Conventional Transgressions), using the same scenario scaffold and acceptance criteria as for Gemini Omni (Appendix A.6.2).

Model rankings are stable across generators. Overall Joint-Pair Accuracy on Omni and PixVerse videos is highly correlated across all 12 models (Spearman ρ = 0.97, pooled over the three subclasses). We also find no evidence of a systematic same-family generation advantage for Gemini models. Both Gemini models achieve comparable overall Joint-Pair Accuracy across generators (Gemini-3.5-Flash: 89.6 → 90.0; Gemini-3.1-Pro: 83.1 → 83.3) and retain the top two positions on PixVerse videos.

Table: Joint-Pair Accuracy (%) for five representative models on Gemini Omni → PixVerse videos. Brackets give the PixVerse estimate ±1 bootstrap standard error (10,000 resamples of minimal pairs). Results for all 12 models are in Appendix A.12 (Table 15).

ModelIntentionalityResponsibilityMoral/Conv.Overall
Gemini-3.5-Flash92.0 → 70.0 [55.6, 84.4]92.6 → 100.0 [100, 100]84.0 → 100.0 [100, 100]89.6 → 90.0 [85.2, 94.8]
Gemini-3.1-Pro84.0 → 70.0 [55.6, 84.4]96.3 → 90.0 [80.4, 99.6]68.0 → 90.0 [80.5, 99.5]83.1 → 83.3 [76.7, 89.9]
GPT-5.6 Luna88.0 → 80.0 [67.3, 92.7]85.2 → 90.0 [80.5, 99.5]48.0 → 70.0 [55.6, 84.4]74.0 → 80.0 [72.9, 87.1]
Qwen3.5-27B Think68.0 → 80.0 [67.4, 92.6]100.0 → 80.0 [67.4, 92.6]44.0 → 60.0 [44.6, 75.4]71.4 → 73.3 [65.5, 81.1]
InternVL3.5-8B Inst20.0 → 10.0 [0.6, 19.4]74.1 → 70.0 [55.6, 84.4]0.0 → 0.0 [0, 0]32.5 → 26.7 [20.9, 32.5]

Together, these results provide evidence that our overall moral reasoning findings and model rankings are not specific to Gemini Omni. While individual subclasses exhibit some variation across generators, the aggregate results remain consistent.

Uncertainty estimates. To further address concerns about the size and statistical reliability of the moral reasoning subset, we have updated Appendix Table 6 to include bootstrap standard errors for all benchmark subclasses, including the three moral reasoning subclasses. Key failure patterns remain clear under these estimates. For example, on Moral/Conventional Transgressions, several models perform below the 25% chance level in Joint-Pair Accuracy even accounting for uncertainty (e.g., Qwen3.5-9B Inst: 8.0 ± 5.5; Keye-VL-1.5-8B Think: 8.0 ± 5.4; InternVL3.5-8B Inst: 0.0 ± 0.0).


References

[1] PixVerse AI. (2026). PixVerse: AI Video Generation Platform. https://pixverse.ai/

AuthorsOpenReview ↗

Follow-up: Summary of Additional Results

We thank the reviewer again for their thoughtful feedback. We have conducted substantial additional experiments addressing all three concerns:

  1. Generalization to real-world videos: We constructed 68 minimal pairs (136 videos) from EPIC-KITCHENS-100 for belief-conditioned rationality. The agent-favourable prediction asymmetry persists: all 12 models perform better on Rational than Irrational examples, while humans are nearly symmetric (100.0% vs. 98.5%). Humans remain near ceiling (98.5% Joint-Pair Accuracy) while most models fall well below, and model rankings are highly consistent across synthetic and natural videos (ρ = 0.93).
  2. Probe interpretation: We conducted activation-steering experiments using probe-derived directions, improving Joint-Pair Accuracy by 28.7 percentage points for Qwen3.5-9B Inst on Help/Hinder.
  3. Moral reasoning robustness: We generated 30 additional minimal pairs using an independent generator, PixVerse, demonstrating highly consistent model rankings across generators (ρ = 0.97), with no evidence of a systematic same-family advantage for Gemini models.

Full details are provided in our individual responses and the revised appendix.

We hope these results address the reviewer’s concerns, and we would be grateful if the reviewer would consider an increase in their rating based on these additional results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. Reviewers found SMORE well motivated (wBQM), with broader coverage of social understanding than many existing benchmarks (dKa5). They noted that its controlled minimal-pair design has clear diagnostic value, isolating whether models respond to socially relevant behavioral differences rather than spurious context cues (dKa5, wBQM, oQGr), and that our complementary metrics expose a meaningful evaluation gap hidden by single-video accuracy (dKa5, oQGr). Reviewers also found the evaluation extensive and credible, with human baselines for every question (wBQM, oQGr), and highlighted the value of combining behavioral evaluation with representation-level analysis (dKa5).

Below we summarize the new experiments added during the discussion; full details are in the individual responses and the revised appendix.

  1. Real-world videos (dKa5, oQGr). A shared concern was generalization to naturalistic scenarios. To assess this, we built 68 True/False Belief minimal pairs from unscripted EPIC-KITCHENS-100 recordings, taking both videos of each pair from the same recording. Humans reach 98.5% Joint-Pair Accuracy (J) while most models fall well below. Model rankings on natural and synthetic videos are highly consistent (Spearman ρ = 0.93), and the failure patterns reported in Sections 3.1–3.4 persist on natural videos.

  2. Probe analysis (oQGr, dKa5). Following oQGr's suggestion, we re-evaluated the probes with metrics that account for class imbalance, such as balanced accuracy and per-class recall. The conclusions of Section 4.2 hold: for Help/Hinder, the distinction remains linearly decodable from model representations even when models answer incorrectly (balanced accuracy 72.9–84.7%), whereas for rationality, model errors coincide with a loss of linear decodability. To address dKa5's concern, we also added a within-pair label-permutation probe baseline and an activation-steering experiment, which shows gains for Qwen3.5-9B Inst on Help/Hinder.

  3. Independent generator for moral reasoning (dKa5). We generated 30 additional moral pairs with PixVerse, using the same scaffold and acceptance criteria as for Gemini Omni. Based on overall Joint-Pair Accuracy, model rankings are consistent across generators (ρ = 0.97), and Gemini models show no same-family advantage.

  4. Text-only ablation (wBQM). We replaced each video with an oracle-style description of the relevant behavior. These differ from ordinary video transcriptions: they were generated with access to the generation metadata and explicitly state the judgment-relevant behavioral cues that models would otherwise have to extract from the video, which makes the task significantly easier. Gemini models reach near-ceiling performance, while several open-source models still show a substantial gap to humans.

  5. Full human validation (wBQM). As suggested, we extended human validation to all 427 pairs, assessing minimality and generation artifacts.

Reviewer oQGr indicated that their concerns have been addressed and raised their score, and Reviewer wBQM indicated no remaining major concerns. Items 1–3 respond to the three questions from Reviewer dKa5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.