- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper Summary The paper extends the Social Proactive Intelligence (SPI) task of RobotEQ (Fang et al., 2026) from images to short videos. Given a robot-viewpoint clip and a set of candidate robot actions, a model labels each action "should do" or "should not do".
A four-level world-state taxonomy (6 domains, 20 dimensions, 142 level-1 and 816 level-2 attributes) drives two things:
- the generation of 1,218 synthetic videos with Seedance;
- the selection of 928 real clips from existing egocentric and robot datasets.
Candidate actions are produced from captions and counterfactually edited captions. Each is labelled by majority vote of four annotators drawn from a pool of eight, and low-consensus items are removed, leaving 17,544 labels.
Forty-eight VLMs are evaluated with the RobotEQ metrics. The best model (A-Avg 79.5) trails the human average (88.8). Supporting analyses include:
- a human study on the value of video (Fig. 7);
- adding prompt-derived visual cues to the model input (Fig. 8);
- augmenting VLMs with an image-to-video "world model" that renders each candidate action (Fig. 9).
Critical strengths
-
A useful, well-documented resource for a timely problem.
- Video-based judgment of socially appropriate robot actions is under-served.
- The benchmark offers 2,146 real and synthetic videos, 17,544 labelled actions and a 48-model leaderboard.
- The full taxonomy and all generation, filtering and evaluation prompts are disclosed (App. A–C, I).
- Why it matters: this resource value is the main reason I lean towards acceptance.
-
A reusable construction recipe and informative diagnostics.
- Taxonomy-conditioned storyboards plus counterfactual caption edits produce closely related candidate actions. Humans cannot judge these from the action text alone (51.0% action-only, Fig. 7).
- Per-class precision/recall exposes systematic response biases. Most closed models over-reject appropriate actions (e.g., GPT-5.6-Luna: proper recall 74.0 vs improper recall 85.2), which matters for deploying proactive agents.
- Why it matters: the benchmark can reveal behaviour that aggregate accuracy hides.
Critical weaknesses
-
The validity of the main evaluation (the leaderboard and the human–model gap) is not yet established.
- Protocol mismatch. The evaluation prompt (App. I, l. ~1273) tells models to judge "using only the action-option text below" and gives no robot role. Yet l. 305 says models receive the video, and annotators answered role-specific questions with a "Not sure" option (Fig. 10). Frames/fps, audio (claimed as a cue, l. 79) and decoding settings are not reported.
- No model-side controls. There are no text-only, single-frame or shuffled-video baselines. The "video-centric" claim therefore rests on a human study (Fig. 7). Its Set A has no image condition despite l. 405, and its set sizes and CIs are not reported.
- Labels and human baseline.
- The consensus rule, discard rate and agreement statistics are missing.
- If annotators are scored against majorities that include their own votes, on consensus-retained items, the 88.8 human score is inflated. The Fig. 4d per-annotator mean is 87.8, while Table 2 reports 90.3/92.7.
- PA and CW correlate with Acc at r = 1.00 (Fig. 5), so A-Avg mostly re-counts accuracy.
- These determine whether Table 2 measures video-grounded social judgment at all. My acceptance leaning depends on the authors clarifying the protocol and adding model-side controls.
-
The two advances claimed beyond the benchmark itself (the world-model solution and the taxonomy's coverage/necessity) are not supported.
- World-model results conflict with Table 2.
- Qwen3-VL-30B scores 60.2 as the Fig. 9b baseline but 76.2 in Table 2 (action-weighted over both subsets). The latter equals its world-model-augmented score.
- For Qwen3-VL-8B and 4B, the world-model scores are within one point of Table 2 (71.5 vs 70.5; 65.2 vs 66.1).
- The reported gains may therefore come from a different baseline setup. Controls are also missing, such as a continuation for a mismatched action or text-only consequence reasoning.
- Taxonomy claims are not supported.
- "Ensures comprehensive coverage" is not measured: there is no realised-coverage statistic and no comparison with free-form generation, and the Limitations concede gaps (ll. 483–484).
- The derivation of the taxonomy is not described. Its Event categories closely resemble EgoNormia's norm categories, which should be cited.
- No per-dimension results are reported.
- Fig. 8b feeds the model cues taken from the generation prompts. It therefore does not show that the taxonomy is "necessary" (l. 436).
- Both are stated contributions in the abstract and conclusion. Unless they are corrected or toned down, the paper overclaims. Together with Weakness 1, this keeps my score at weak rather than clear acceptance.
- World-model results conflict with Table 2.
Questions for authors
Clarifications that could change my assessment:
- Was the App. I prompt used for all 48 models? How were the video, the robot role/place and audio provided to each model (frames, fps, API mode)?
- What are the text-only and single-frame results for a few representative models?
- On labels and the human baseline:
- What is the consensus rule (4–0 vs 3–1, and how are "Not sure" votes handled)?
- How many candidates were discarded, and what are κ/α?
- How is the human baseline computed (leave-one-out)?
- Please break down "100K+ annotations" (17,544 × 4 = 70,176 retained judgments).
- Which prompt, metric, checkpoint and subset produced the Fig. 9b baselines, and why do they differ from Table 2?
- How was the "Chinese roles" restriction (ll. 264–265) enforced for clips from JRDB, Ego4D and Charades-Ego? What is the per-source breakdown of the 928 real clips?
- How are the robot's embodiment and capabilities conveyed to annotators and models? Several actions assume an arm.
Suggestions (not required for my score):
- Report per-dimension Macro-F1 and realised coverage for both subsets, and add a taxonomy-vs-free-form comparison.
- Add bootstrap CIs over videos.
- Soften two claims:
- "GPT-5.6-Luna is the most suitable model" (l. 476). Its lead is 0.2 points, and it is 4th on the synthetic subset.
- The training-data claim (l. 478), which is untested.
- Cite the source datasets (JRDB, Ego4D, Ego-Exo4D, Charades-Ego) and Wan2.2, and recheck the "not video-centric" mark for EgoNormia in Table 1.
Minor (no response needed):
- "40+" (l. 304) vs 48 models.
- Fig. 1 pairs an orchard clip with a school-hall role.
- Some Fig. 10 options read like scene descriptions.
- The three Keye-VL rows are nearly identical.
- The Kimi-VL-A3B-Thinking link points to the Instruct checkpoint.
9 comments
Response to Weakness 1.3
Response to Weakness 1.3. Labels and human baseline:
1.3.1 The consensus rule, discard rate and agreement statistics
Thank you for your valuable comment.
(1) Regarding the Consensus Rule, each candidate action was independently evaluated by four annotators. An action was retained only if at least 75% of the annotators agreed on the same judgment, corresponding to at least three out of four annotators. Actions with insufficient agreement or any "Not Sure" judgments were excluded from the final dataset to reduce ambiguity in the ground-truth labels.
(2) Regarding the Discard Rate, among the 27,053 initially annotated candidate actions, 17,544 were retained and 9,509 were discarded, corresponding to a retention rate of 64.85% and a discard rate of 35.15%. This filtering procedure was intended to reduce annotation ambiguity and improve the reliability of the retained ground-truth labels.
(3) Regarding Agreement Statistics, we calculated pairwise percentage agreement among the eight annotators based on their overlapping annotations of retained candidate actions. The mean pairwise agreement was 83.61% . The detailed results are presented below.
Table 3. Pairwise agreement (%) among eight annotators on retained samples.
| Annotator | A1 | A2 | A3 | A4 | A5 | A6 | A7 | A8 |
|---|---|---|---|---|---|---|---|---|
| A1 | — | 81.7 | 82.6 | 85.8 | 84.7 | 86.0 | 89.5 | 93.1 |
| A2 | 81.7 | — | 74.7 | 78.6 | 84.3 | 84.3 | 79.0 | 80.8 |
| A3 | 82.6 | 74.7 | — | 83.2 | 78.4 | 81.7 | 83.4 | 82.7 |
| A4 | 85.8 | 78.6 | 83.2 | — | 82.8 | 83.4 | 87.4 | 86.2 |
| A5 | 84.7 | 84.3 | 78.4 | 82.8 | — | 83.9 | 81.5 | 83.6 |
| A6 | 86.0 | 84.3 | 81.7 | 83.4 | 83.9 | — | 84.5 | 84.8 |
| A7 | 89.5 | 79.0 | 83.4 | 87.4 | 81.5 | 84.5 | — | 88.5 |
| A8 | 93.1 | 80.8 | 82.7 | 86.2 | 83.6 | 84.8 | 88.5 | — |
These results provide additional evidence of consistency among annotators on the retained samples. We have included the consensus rule and discard statistics in the appendix of the revised manuscript.
1.3.2 Clarification of the Human Score
Thank you for pointing this out. We apologize for the confusion caused by Figure 4d, which was mistakenly included in the original submission. We have corrected this error by replacing Figure 4d with a pairwise annotator agreement matrix in the revised manuscript.
The average pairwise agreement rate is 83.61%, indicating generally high consistency among annotators. These results provide additional evidence supporting the reliability of our human annotations and the resulting ground-truth labels. The detailed pairwise agreement matrix is provided in Table 3.
We would also like to clarify that the human baseline results reported in Table 2 were obtained from newly recruited participants who were independent of the original annotation process. Their judgments were not involved in constructing the ground-truth labels.
1.3.3 PA and CW Correlations with Acc
Thank you for this insightful observation. We would like to further clarify the motivation behind introducing PA and CW.
PA (Probability of Agreement) measures the agreement between model predictions and individual human annotators, rather than relying solely on the final ground-truth labels. CW (Consensus-Weighted Agreement) further incorporates the strength of human consensus, assigning greater importance to samples with higher agreement among annotators.
While Accuracy and Macro-F1 evaluate model predictions against majority-vote ground-truth labels, Social Proactive Intelligence inherently involves subjective judgments of social appropriateness, where individual opinions may differ even when a majority consensus exists. Therefore, we introduce PA and CW to account for individual human perspectives, including minority judgments that differ from the majority. This allows our evaluation to consider not only the majority consensus but also the diversity of human opinions, which is particularly important for socially situated decision-making.
Responses to Weakness 2.1 and Weakness 2.2
Response to Weakness 2.1 World-Model Results:
Thank you for raising this question. We would like to clarify the discrepancy between Figure 9b and Table 2.
Due to the substantial computational cost of video generation using Wan2.2-I2V-A14B, we selected a subset of challenging samples from the original benchmark, focusing on cases that models frequently misclassified in the Table 2 evaluation. As Figure 9b follows the same model configurations and evaluation protocol as Table 2, its lower baseline performance reflects the greater difficulty of the selected samples rather than differences in evaluation settings.
Furthermore, the baseline and world-model-augmented results in Figure 9b are evaluated on the same subset under identical settings, with world-model augmentation being the only difference.
Regarding Qwen3-VL-8B and Qwen3-VL-4B, the numerical similarity between their world-model-augmented scores in Figure 9b and their results in Table 2 is purely coincidental. Therefore, these results have no direct relationship.
Response to Weakness 2.2 Taxonomy Claims:
2.2.1 Evidence for Comprehensive Coverage
Thank you for your feedback. We would like to provide additional evidence demonstrating the effectiveness of our world-state taxonomy in constructing diverse scenarios.
First, we conducted an additional human evaluation to verify the actual coverage of our taxonomy. Four annotators independently examined the presence of the 20 dimensions and their Level-1 and Level-2 attributes in the 1,218 generated videos. An attribute was considered present when at least three of the four annotators agreed. The results confirm 100% coverage of all 142 Level-1 and 816 Level-2 attributes across the generated dataset. The detailed prevalence of the 20 dimensions is shown below.
Table 4. Prevalence of the 20 Dimensions.
| Dimension | Count | Percentage |
|---|---|---|
| Scene | 1,205 | 98.93% |
| Time | 1,046 | 85.88% |
| Event | 1,085 | 89.08% |
| Action | 1,134 | 93.10% |
| Gaze | 1,182 | 97.04% |
| Expression | 1,172 | 96.22% |
| Gesture | 407 | 33.42% |
| Linguistic content | 606 | 49.75% |
| Age | 1,182 | 97.04% |
| Gender | 1,176 | 96.55% |
| Occupation | 832 | 68.31% |
| Object category | 945 | 77.59% |
| Object material | 1,215 | 99.75% |
| Object risk | 478 | 39.24% |
| Human-human relation | 641 | 52.63% |
| Human-object relation | 964 | 79.15% |
| Object-object relation | 1,119 | 91.87% |
| Spatial relation | 1,218 | 100.00% |
| Human state | 1,180 | 96.88% |
| Object state | 1,062 | 87.19% |
As for unconstrained video generation, we observed that it tends to produce common and simple scenarios. For example, when given an "object tipping" event, the model repeatedly generated similar restaurant scenes involving spilled soup or coffee. We encountered such repetitive cases during our video generation process, which further highlights the importance of using our taxonomy to guide video generation and improve scenario diversity.
2.2.2 Taxonomy Derivation
Thank you for the insightful observation.
Our taxonomy draws on established research on contextual understanding. Specifically, Dey and Abowd [1] identify Identity, Location, Activity, and Time as fundamental context types, corresponding to our Human, Place, Event, and Time domains. Augusto [2] further emphasizes the importance of world states, events, and temporal context in intelligent systems. Tong et al. [3] highlight human-object relationships in intention understanding, motivating our Object and Relationship domains.
Building upon these perspectives and RobotEQ's definition of Social Proactive Intelligence (SPI), we developed a four-level taxonomy comprising 6 domains, 20 dimensions, 142 level-1 attributes, and 816 level-2 attributes, aiming to provide a structured representation of the contextual factors influencing SPI judgments.
We also acknowledge that EgoNormia [4] inspired some social norm-related categories within our Event domain. However, our taxonomy extends beyond social norms to characterize broader contextual factors. We have clarified these design motivations and added the relevant citations in the revised paper.
References
[1] Dey and Abowd. Towards a Better Understanding of Context and Context-Awareness. 1999.
[2] Augusto. Contexts and Context-awareness Revisited from an Intelligent Environments Perspective. 2022.
[3] Tong et al. Human Intention Recognition Using Context Relationships in Complex Scenes. 2025.
[4] Rezaei et al. EgoNormia: Benchmarking Physical-Social Norm Understanding. 2025.
Responses to Weaknesses 1.1 and Weaknesses 1.2
Response to Weakness 1.1. Protocol mismatch:
Thank you for raising these important concerns. We appreciate the opportunity to clarify our annotation and evaluation protocols.
(1) Regarding the Evaluation Prompt, we sincerely apologize for the wording error in Appendix I. All evaluated models received visual information from the corresponding videos rather than relying solely on action-option text. This was a documentation error that did not affect the actual evaluation procedure. We have corrected the prompt in the revised manuscript. The corrected prompt is provided below:
Evaluation Prompt:
Based on the following video input:
{video_input}Determine whether each candidate action is appropriate in the current scene.
Candidate Actions:
{options}Carefully consider each candidate action before making a judgment.
Rules:
<Judgment>below is a placeholder and must be replaced with the selected label.Strictly follow the format below, outputting one line per option without omitting any options:
(2) Regarding robot roles, explicit role descriptions were provided to human annotators to facilitate their understanding of the scenarios and improve annotation efficiency. To assess whether these descriptions influenced annotation judgments, we conducted a spot check on 200 videos, comparing judgments with and without explicit role descriptions. The results showed 97.5% agreement between the two settings, suggesting that the videos themselves provide sufficient contextual cues to infer the robot's role.
(3) Regarding the "Not Sure" Option, we introduced this option during human annotation to handle actions that annotators could not confidently judge. Any action receiving a "Not Sure" annotation was excluded before the ground-truth label selection process and therefore did not enter the final benchmark. This ensures that all retained actions have clear binary labels consistent with the model evaluation setting.
(4) Regarding Input and Inference Configurations, models supporting native video input received the original MP4 files, while models requiring frame-based inputs received eight uniformly sampled frames from each video. Audio was provided only to models with native audiovisual capabilities, such as Qwen3-Omni and Qwen2.5-Omni. Detailed input processing, inference parameters, and decoding configurations are available in our anonymized repository.
Response to Weakness 1.2. No model-side controls:
Thank you for this question. To investigate the contribution of video information, we conducted additional controlled experiments comparing Action-only (text-only), Image+Action (single-frame), and Video+Action settings across representative models on both synthetic and real-world subsets.
Importantly, we used identical samples and models across the three input settings within each subset to ensure a fair comparison. The results are presented below.
Table 1. Accuracy (%) on the synthetic subset.
Table 2. Accuracy (%) on the real subset.
Across both subsets, Video+Action consistently achieves higher Accuracy than Image+Action and Action-only across all eight model configurations. These findings provide additional model-side evidence that video inputs offer useful contextual information beyond action text and single-frame images, supporting the importance of video information in Social Proactive Intelligence (SPI) evaluation.