- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes SocialPrism, a generative reward framework for evaluating and improving video social reasoning. SocialPrism evaluates model responses along five dimensions, including contextual fact consistency, inference logical consistency, multimodal information utilisation, causal rationality, mental state inference. The authors construct a few datasets, including a 30+k annotated dataset and 300+ sample benchmark, and then train Gen-SocialPrism-7B. The main results include that the reward prediction error decreases compared to existing reward models, the model improves performance over binary reward GRPO.
Critical strengths
- The paper identifies a clear limitation of scalar rewards in that they cannot distinguish failure cases such as perception, multimodal integration and social inference.
- The authors also employ a comprehensive dataset construction pipeline, including diverse response generating models, multi-stage annotation, rubric-compliance verification etc.
- The authors also demonstrate downstream utility by evaluating the reward modelling in RL post-training and test time selection.
Critical weaknesses
- The methodological novelty is unclear. The method seems to be a combination of different components that are already individually established, such as multidimensional reward modelling, generative evaluator etc. There isn't any new or scientific intellectual insight here.
- The work depends heavily on synthetic annotations. The training annotations are produced by Gemini 2.5 Pro as well as the benchmark. This creates a dependence on the teacher model's judgements.
- Ablations are insufficient, in the sense that they do not convincingly establish which components are most necessary. The paper would benefit from stronger ablations on individual dimensions etc.
- The paper also lacks mechanism analysis, to show that the method works as intended.
Questions for authors
Questions to be addressed are stipulated in the weaknesses above.
4 comments
W3: Ablations and justification of the evaluation dimensions.
The paper provides both a theoretical basis for the five dimensions and empirical ablations of the reward model. Section 3 derives the dimensions from a capability hierarchy encompassing perception, multimodal integration, causal and social inference, and logical coherence, drawing on Theory of Mind, the Belief–Desire–Intention model, and social cognition research. Each dimension targets a distinct aspect of response quality.
The original ablations in Table 2 examine both curriculum learning and multidimensional modeling. Removing Stage 1 increases total-score NMAE from 0.2016 to 0.2069, while the total-score-only variant yields an NMAE of 0.2071. These results support the contributions of both components to reward prediction.
To further assess downstream utility, we conducted an additional post-training ablation by replacing Gen-SocialPrism-7B with the total-score-only reward model while keeping the training configuration unchanged. The total-score-only model directly predicts the aggregate score, whereas Gen-SocialPrism-7B explicitly evaluates the five dimensions. The results are:
| Benchmark / subset | + CS + Gen-SocialPrism-7B | + CS + Total-score-only RM | Difference (pp) |
|---|---|---|---|
| R³-Bench-Hard | 33.23 | 31.33 | +1.90 |
| EmoBench-M: SD | 67.80 | 62.20 | +5.60 |
| EmoBench-M: PEA | 75.20 | 69.20 | +6.00 |
| EmoBench-M: MPDER | 51.40 | 51.60 | −0.20 |
| WorldSense | 47.79 | 47.76 | +0.03 |
| OmniVideoBench: (0, 1] min | 49.12 | 47.95 | +1.17 |
| OmniVideoBench: (1, 5] min | 35.59 | 34.12 | +1.47 |
| FutureOmni | 52.42 | 51.55 | +0.87 |
| Mean | 51.57 | 49.46 | +2.11 |
Values are accuracy (%). The mean weights the eight reported metrics equally. Differences are in percentage points.
Under this matched training configuration, Gen-SocialPrism-7B improves seven of the eight metrics, increasing mean accuracy from 49.46% to 51.57% (+2.11 percentage points). The largest gains occur on SD (+5.60 points) and PEA (+6.00 points), with small decreases on MPDER (−0.20 points).
Together with Table 2, this additional experiment demonstrates the benefit of the multidimensional reward formulation over direct total-score prediction in both reward prediction and downstream policy improvement.
W4: Analysis of how the method works.
Our evaluation explicitly tests whether Gen-SocialPrism-7B provides the quality signal required by its intended uses across four complementary settings:
- Reward prediction: On SocialPrism-Bench, Gen-SocialPrism-7B achieves a total-score NMAE of 0.2016 and a Spearman correlation of 0.4247, outperforming the compared reward models on both metrics (Table 2). This tests agreement with human-verified quality judgments.
- Post-training: With the same cold-start initialization, GRPO algorithm, and QA training data, adding the Gen-SocialPrism-7B reward improves all eight reported metrics over binary-reward GRPO (Table 3). The additional total-score-only comparison in W3 further shows the downstream benefit of the multidimensional reward formulation.
- Inference: Figure 3 directly examines whether assigned rewards distinguish response reliability. Accuracy increases from 38.5% to 43.5% to 55.9% across the low-, medium-, and high-reward groups on WorldSense, and from 30.7% to 49.2% to 63.5% on FutureOmni. The high-reward group exceeds the overall accuracy on all five benchmarks.
- Sampling: Figure 4 tests whether the reward signal improves answer selection by increasing the voting weight of high-reward candidates. Under the same sampling budgets, reward-guided voting improves upon or performs competitively with majority voting across the evaluated settings, with particularly clear gains on EmoBench-M, WorldSense, and OmniVideoBench.
These experiments examine agreement with human judgments, discrimination between responses of different reliability, and the use of that discrimination in training and selection. They provide direct evidence that the reward model performs its intended functions. We would welcome clarification of any specific mechanism or failure mode the reviewer considers unaddressed, so that we can provide a targeted response.
Reviewer Response
I thank the authors for their rebuttal. I have decided to retain my score, for the following reasons.
-
I believe the contribution is still unclear. The authors iterate that part of the contribution includes "whether explicitly evaluating different aspects of social reasoning produces a more useful reward signal than evaluating only aggregate quality or answer correctness". My opinion is that this is a rather "obvious" insight that may not be entirely new. Additionally, pitching this as a "benchmark/data" contribution may not help here - benchmark/data contributions typically involve a new axes of evaluation, new problem setting, and more demonstration of the utility of the data.
-
I acknowledge the verification of the synthetic data, and also that of the analysis of how the method works. On the latter part, I think the intellectual insight of this method is still unclear - these appear to be robustness tests, which are welcome, but we aren't really testing any new principles introduced etc.
Overall while I acknowledge the author's efforts, I think the intellectual contribution of this work is still unclear.
I would suggest for the authors, to simplify the entire framework and focus on what is that "one" singular aspect of the methodology that leads to better results. It can't simply be that "one singular aspect" = our multi-dimensional framework. I hope the authors understand my point of view.
Thank you for clarifying your perspective. We understand your emphasis on identifying a specific methodological advance and appreciate your suggestion to sharpen the central contribution.
We respectfully maintain that the structured training and evaluation resources are substantive contributions in their own right. SocialPrism provides explicit scoring criteria, dimension-specific annotations, and human-verified evaluation data for training and assessing video social reasoning verifiers. Their utility is demonstrated through reward prediction, policy improvement, and test-time answer selection.
A reliable verifier is particularly valuable in the domain of video social reasoning: making the cognitive processes underlying social reasoning in the real, open world quantifiable. Resources that enable models to assess these distinctions support further research on social reasoning and verifier-guided improvement (even to support recursive self-improvement in this domain). We therefore believe the contribution should also be assessed through the capabilities these resources enable and their demonstrated downstream utility, alongside the novelty of the individual modeling components.
We thank the reviewer for recognizing the motivation, dataset construction pipeline, and downstream utility of SocialPrism. Below, we clarify the contributions and supporting evidence, including an additional post-training ablation.
W1: Contribution and scientific insight.
Our primary contribution is the structured data and evaluation framework for training and evaluating verifiers and reward models for video social reasoning. Assessing the work solely through the novelty of its individual modeling components overlooks this contribution. SocialPrism-Data provides 37,087 responses annotated with dimension-specific scores and rationales, complemented by VSR-8K with 8,345 QA pairs, SocialPref-8K with 8,251 preference pairs, and SocialPrism-Bench with 339 human-verified evaluation samples. These resources support reward learning, preference learning, policy optimization, and reward-model evaluation within a common framework.
The scientific question is whether explicitly evaluating different aspects of social reasoning produces a more useful reward signal than evaluating only aggregate quality or answer correctness. This distinction matters when a response must reconcile conflicting audiovisual cues and justify an interpretation of hidden mental states: reaching the correct option does not establish that the supporting reasoning is sound. SocialPrism makes these aspects measurable through explicit rubrics and structured supervision. Table 2 and the additional experiment in W3 show benefits in both reward prediction and downstream policy performance.
The resulting data resources also provide a basis for future work on verifier-guided iterative self-improvement. Their practical utility is demonstrated here through post-training and test-time selection. The contribution therefore comprises reusable supervision resources, a domain-specific evaluation protocol, and empirical evidence that the structured reward formulation improves these applications.
W2: Dependence on synthetic annotations.
Gemini 2.5 Pro serves as the initial annotator, but the final scoring data undergo additional verification before use. Annotation quality should therefore be assessed through the full construction and validation procedure, which includes:
These stages separately address response coverage, video understanding, rubric compliance, and unsupported claims. Acceptance into the training corpus is thus subject to checks beyond the initial evaluator's judgment.
Crucially, the final reference annotations in SocialPrism-Bench are human-verified. Two annotators independently review the scores and rationales across all five dimensions. Samples are retained when both annotators accept the annotations or agree on the same correction. Section 4.4 reports 92.64% agreement between automatic annotations and human judgments, alongside 89.65% inter-annotator agreement.
The reported agreement with human judgments provides empirical evidence of annotation reliability. Moreover, reward prediction is evaluated against the final human-verified reference annotations, rather than unreviewed outputs from Gemini 2.5 Pro.