acceptodds
Under review as a conference paper at ICLR 2027

RobotEQ-Video: A Video-Centric Benchmark for Social Proactive Intelligence with World-State Taxonomy

Abstract

Social Proactive Intelligence (SPI) extends proactive assistance beyond task completion to consider social appropriateness in diverse embodied scenarios. However, prior SPI research faces two key limitations. First, existing work focuses on static images, whereas dynamic videos provide crucial cues for inferring human states and needs, offering richer information than isolated images. Second, prior work often relies on free-form data collection pipelines, which fail to guarantee comprehensive coverage of diverse contextual factors. To address these gaps, we introduce **RobotEQ-Video**, shifting the focus from *image-centric* to *video-centric* analysis. To ensure coverage of diverse contextual factors, we construct a *hierarchical world-state taxonomy* organized into a four-level coarse-to-fine structure, comprising 6 domains, 20 dimensions, 142 level-1 attributes, and 816 level-2 attributes. The resulting benchmark comprises 2K+ videos with 100K+ human annotations and 17K+ labels for assessing action appropriateness. Benchmark evaluation reveals that current systems still fall short of human performance. We further explore how world models can help tackle this task. *This work advances SPI research from static images to dynamic videos and ensures more comprehensive coverage of diverse contextual factors through our world-state taxonomy.*

Primary area:
datasets and benchmarks
Keywords:
Social Proactive Intelligence, Video-Centric Benchmark, Fine-grained Taxonomy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_889fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper Summary The paper extends the Social Proactive Intelligence (SPI) task of RobotEQ (Fang et al., 2026) from images to short videos. Given a robot-viewpoint clip and a set of candidate robot actions, a model labels each action "should do" or "should not do".

A four-level world-state taxonomy (6 domains, 20 dimensions, 142 level-1 and 816 level-2 attributes) drives two things:

  1. the generation of 1,218 synthetic videos with Seedance;
  2. the selection of 928 real clips from existing egocentric and robot datasets.

Candidate actions are produced from captions and counterfactually edited captions. Each is labelled by majority vote of four annotators drawn from a pool of eight, and low-consensus items are removed, leaving 17,544 labels.

Forty-eight VLMs are evaluated with the RobotEQ metrics. The best model (A-Avg 79.5) trails the human average (88.8). Supporting analyses include:

  1. a human study on the value of video (Fig. 7);
  2. adding prompt-derived visual cues to the model input (Fig. 8);
  3. augmenting VLMs with an image-to-video "world model" that renders each candidate action (Fig. 9).
Critical strengths
  1. A useful, well-documented resource for a timely problem.

    • Video-based judgment of socially appropriate robot actions is under-served.
    • The benchmark offers 2,146 real and synthetic videos, 17,544 labelled actions and a 48-model leaderboard.
    • The full taxonomy and all generation, filtering and evaluation prompts are disclosed (App. A–C, I).
    • Why it matters: this resource value is the main reason I lean towards acceptance.
  2. A reusable construction recipe and informative diagnostics.

    • Taxonomy-conditioned storyboards plus counterfactual caption edits produce closely related candidate actions. Humans cannot judge these from the action text alone (51.0% action-only, Fig. 7).
    • Per-class precision/recall exposes systematic response biases. Most closed models over-reject appropriate actions (e.g., GPT-5.6-Luna: proper recall 74.0 vs improper recall 85.2), which matters for deploying proactive agents.
    • Why it matters: the benchmark can reveal behaviour that aggregate accuracy hides.
Critical weaknesses
  1. The validity of the main evaluation (the leaderboard and the human–model gap) is not yet established.

    • Protocol mismatch. The evaluation prompt (App. I, l. ~1273) tells models to judge "using only the action-option text below" and gives no robot role. Yet l. 305 says models receive the video, and annotators answered role-specific questions with a "Not sure" option (Fig. 10). Frames/fps, audio (claimed as a cue, l. 79) and decoding settings are not reported.
    • No model-side controls. There are no text-only, single-frame or shuffled-video baselines. The "video-centric" claim therefore rests on a human study (Fig. 7). Its Set A has no image condition despite l. 405, and its set sizes and CIs are not reported.
    • Labels and human baseline.
      • The consensus rule, discard rate and agreement statistics are missing.
      • If annotators are scored against majorities that include their own votes, on consensus-retained items, the 88.8 human score is inflated. The Fig. 4d per-annotator mean is 87.8, while Table 2 reports 90.3/92.7.
      • PA and CW correlate with Acc at r = 1.00 (Fig. 5), so A-Avg mostly re-counts accuracy.
    • These determine whether Table 2 measures video-grounded social judgment at all. My acceptance leaning depends on the authors clarifying the protocol and adding model-side controls.
  2. The two advances claimed beyond the benchmark itself (the world-model solution and the taxonomy's coverage/necessity) are not supported.

    • World-model results conflict with Table 2.
      • Qwen3-VL-30B scores 60.2 as the Fig. 9b baseline but 76.2 in Table 2 (action-weighted over both subsets). The latter equals its world-model-augmented score.
      • For Qwen3-VL-8B and 4B, the world-model scores are within one point of Table 2 (71.5 vs 70.5; 65.2 vs 66.1).
      • The reported gains may therefore come from a different baseline setup. Controls are also missing, such as a continuation for a mismatched action or text-only consequence reasoning.
    • Taxonomy claims are not supported.
      • "Ensures comprehensive coverage" is not measured: there is no realised-coverage statistic and no comparison with free-form generation, and the Limitations concede gaps (ll. 483–484).
      • The derivation of the taxonomy is not described. Its Event categories closely resemble EgoNormia's norm categories, which should be cited.
      • No per-dimension results are reported.
    • Fig. 8b feeds the model cues taken from the generation prompts. It therefore does not show that the taxonomy is "necessary" (l. 436).
    • Both are stated contributions in the abstract and conclusion. Unless they are corrected or toned down, the paper overclaims. Together with Weakness 1, this keeps my score at weak rather than clear acceptance.
Questions for authors

Clarifications that could change my assessment:

  1. Was the App. I prompt used for all 48 models? How were the video, the robot role/place and audio provided to each model (frames, fps, API mode)?
  2. What are the text-only and single-frame results for a few representative models?
  3. On labels and the human baseline:
    • What is the consensus rule (4–0 vs 3–1, and how are "Not sure" votes handled)?
    • How many candidates were discarded, and what are κ/α?
    • How is the human baseline computed (leave-one-out)?
    • Please break down "100K+ annotations" (17,544 × 4 = 70,176 retained judgments).
  4. Which prompt, metric, checkpoint and subset produced the Fig. 9b baselines, and why do they differ from Table 2?
  5. How was the "Chinese roles" restriction (ll. 264–265) enforced for clips from JRDB, Ego4D and Charades-Ego? What is the per-source breakdown of the 928 real clips?
  6. How are the robot's embodiment and capabilities conveyed to annotators and models? Several actions assume an arm.

Suggestions (not required for my score):

  1. Report per-dimension Macro-F1 and realised coverage for both subsets, and add a taxonomy-vs-free-form comparison.
  2. Add bootstrap CIs over videos.
  3. Soften two claims:
    • "GPT-5.6-Luna is the most suitable model" (l. 476). Its lead is 0.2 points, and it is 4th on the synthetic subset.
    • The training-data claim (l. 478), which is untested.
  4. Cite the source datasets (JRDB, Ego4D, Ego-Exo4D, Charades-Ego) and Wan2.2, and recheck the "not video-centric" mark for EgoNormia in Table 1.

Minor (no response needed):

  1. "40+" (l. 304) vs 48 models.
  2. Fig. 1 pairs an orchard clip with a school-hall role.
  3. Some Fig. 10 options read like scene descriptions.
  4. The three Keye-VL rows are nearly identical.
  5. The Kimi-VL-A3B-Thinking link points to the Instruct checkpoint.
9 comments
AuthorsOpenReview ↗

Responses to Weaknesses 1.1 and Weaknesses 1.2

Response to Weakness 1.1. Protocol mismatch:

Thank you for raising these important concerns. We appreciate the opportunity to clarify our annotation and evaluation protocols.

(1) Regarding the Evaluation Prompt, we sincerely apologize for the wording error in Appendix I. All evaluated models received visual information from the corresponding videos rather than relying solely on action-option text. This was a documentation error that did not affect the actual evaluation procedure. We have corrected the prompt in the revised manuscript. The corrected prompt is provided below:

Evaluation Prompt:

Based on the following video input:

{video_input}

Determine whether each candidate action is appropriate in the current scene.

Candidate Actions:

{options}

Carefully consider each candidate action before making a judgment.

Rules:

  • For each option, output exactly one label: "Should Do" or "Should Not Do".
  • Do not output placeholder text such as "Should Do/Should Not Do".
  • The <Judgment> below is a placeholder and must be replaced with the selected label.

Strictly follow the format below, outputting one line per option without omitting any options:

opt1: <Judgment>
opt2: <Judgment>
...

(2) Regarding robot roles, explicit role descriptions were provided to human annotators to facilitate their understanding of the scenarios and improve annotation efficiency. To assess whether these descriptions influenced annotation judgments, we conducted a spot check on 200 videos, comparing judgments with and without explicit role descriptions. The results showed 97.5% agreement between the two settings, suggesting that the videos themselves provide sufficient contextual cues to infer the robot's role.

(3) Regarding the "Not Sure" Option, we introduced this option during human annotation to handle actions that annotators could not confidently judge. Any action receiving a "Not Sure" annotation was excluded before the ground-truth label selection process and therefore did not enter the final benchmark. This ensures that all retained actions have clear binary labels consistent with the model evaluation setting.

(4) Regarding Input and Inference Configurations, models supporting native video input received the original MP4 files, while models requiring frame-based inputs received eight uniformly sampled frames from each video. Audio was provided only to models with native audiovisual capabilities, such as Qwen3-Omni and Qwen2.5-Omni. Detailed input processing, inference parameters, and decoding configurations are available in our anonymized repository.

Response to Weakness 1.2. No model-side controls:

Thank you for this question. To investigate the contribution of video information, we conducted additional controlled experiments comparing Action-only (text-only), Image+Action (single-frame), and Video+Action settings across representative models on both synthetic and real-world subsets.

Importantly, we used identical samples and models across the three input settings within each subset to ensure a fair comparison. The results are presented below.

Table 1. Accuracy (%) on the synthetic subset.

ModelAction-onlyImage+ActionVideo+Action
GLM-4.6V-Flash66.0667.2271.27
Keye-VL-1.5-8B66.1668.3769.60
Qwen3.6-27B-Instruct73.3175.5076.69
Qwen3-VL-8B-Thinking72.4473.5074.73

Table 2. Accuracy (%) on the real subset.

ModelAction-onlyImage+ActionVideo+Action
Keye-VL-1.5-8B67.6771.7773.08
InternVL3.5-8B-Instruct65.9467.1969.69
InternVL3.5-4B-Instruct61.9263.3565.58
Qwen3-VL-4B-Instruct65.8667.0868.25

Across both subsets, Video+Action consistently achieves higher Accuracy than Image+Action and Action-only across all eight model configurations. These findings provide additional model-side evidence that video inputs offer useful contextual information beyond action text and single-frame images, supporting the importance of video information in Social Proactive Intelligence (SPI) evaluation.

AuthorsOpenReview ↗

Response to Weakness 1.3

Response to Weakness 1.3. Labels and human baseline:

1.3.1 The consensus rule, discard rate and agreement statistics

Thank you for your valuable comment.

(1) Regarding the Consensus Rule, each candidate action was independently evaluated by four annotators. An action was retained only if at least 75% of the annotators agreed on the same judgment, corresponding to at least three out of four annotators. Actions with insufficient agreement or any "Not Sure" judgments were excluded from the final dataset to reduce ambiguity in the ground-truth labels.

(2) Regarding the Discard Rate, among the 27,053 initially annotated candidate actions, 17,544 were retained and 9,509 were discarded, corresponding to a retention rate of 64.85% and a discard rate of 35.15%. This filtering procedure was intended to reduce annotation ambiguity and improve the reliability of the retained ground-truth labels.

(3) Regarding Agreement Statistics, we calculated pairwise percentage agreement among the eight annotators based on their overlapping annotations of retained candidate actions. The mean pairwise agreement was 83.61% . The detailed results are presented below.

Table 3. Pairwise agreement (%) among eight annotators on retained samples.

AnnotatorA1A2A3A4A5A6A7A8
A1—81.782.685.884.786.089.593.1
A281.7—74.778.684.384.379.080.8
A382.674.7—83.278.481.783.482.7
A485.878.683.2—82.883.487.486.2
A584.784.378.482.8—83.981.583.6
A686.084.381.783.483.9—84.584.8
A789.579.083.487.481.584.5—88.5
A893.180.882.786.283.684.888.5—

These results provide additional evidence of consistency among annotators on the retained samples. We have included the consensus rule and discard statistics in the appendix of the revised manuscript.

1.3.2 Clarification of the Human Score

Thank you for pointing this out. We apologize for the confusion caused by Figure 4d, which was mistakenly included in the original submission. We have corrected this error by replacing Figure 4d with a pairwise annotator agreement matrix in the revised manuscript.

The average pairwise agreement rate is 83.61%, indicating generally high consistency among annotators. These results provide additional evidence supporting the reliability of our human annotations and the resulting ground-truth labels. The detailed pairwise agreement matrix is provided in Table 3.

We would also like to clarify that the human baseline results reported in Table 2 were obtained from newly recruited participants who were independent of the original annotation process. Their judgments were not involved in constructing the ground-truth labels.

1.3.3 PA and CW Correlations with Acc

Thank you for this insightful observation. We would like to further clarify the motivation behind introducing PA and CW.

PA (Probability of Agreement) measures the agreement between model predictions and individual human annotators, rather than relying solely on the final ground-truth labels. CW (Consensus-Weighted Agreement) further incorporates the strength of human consensus, assigning greater importance to samples with higher agreement among annotators.

While Accuracy and Macro-F1 evaluate model predictions against majority-vote ground-truth labels, Social Proactive Intelligence inherently involves subjective judgments of social appropriateness, where individual opinions may differ even when a majority consensus exists. Therefore, we introduce PA and CW to account for individual human perspectives, including minority judgments that differ from the majority. This allows our evaluation to consider not only the majority consensus but also the diversity of human opinions, which is particularly important for socially situated decision-making.

AuthorsOpenReview ↗

Responses to Weakness 2.1 and Weakness 2.2

Response to Weakness 2.1 World-Model Results:

Thank you for raising this question. We would like to clarify the discrepancy between Figure 9b and Table 2.

Due to the substantial computational cost of video generation using Wan2.2-I2V-A14B, we selected a subset of challenging samples from the original benchmark, focusing on cases that models frequently misclassified in the Table 2 evaluation. As Figure 9b follows the same model configurations and evaluation protocol as Table 2, its lower baseline performance reflects the greater difficulty of the selected samples rather than differences in evaluation settings.

Furthermore, the baseline and world-model-augmented results in Figure 9b are evaluated on the same subset under identical settings, with world-model augmentation being the only difference.

Regarding Qwen3-VL-8B and Qwen3-VL-4B, the numerical similarity between their world-model-augmented scores in Figure 9b and their results in Table 2 is purely coincidental. Therefore, these results have no direct relationship.


Response to Weakness 2.2 Taxonomy Claims:

2.2.1 Evidence for Comprehensive Coverage

Thank you for your feedback. We would like to provide additional evidence demonstrating the effectiveness of our world-state taxonomy in constructing diverse scenarios.

First, we conducted an additional human evaluation to verify the actual coverage of our taxonomy. Four annotators independently examined the presence of the 20 dimensions and their Level-1 and Level-2 attributes in the 1,218 generated videos. An attribute was considered present when at least three of the four annotators agreed. The results confirm 100% coverage of all 142 Level-1 and 816 Level-2 attributes across the generated dataset. The detailed prevalence of the 20 dimensions is shown below.

Table 4. Prevalence of the 20 Dimensions.

DimensionCountPercentage
Scene1,20598.93%
Time1,04685.88%
Event1,08589.08%
Action1,13493.10%
Gaze1,18297.04%
Expression1,17296.22%
Gesture40733.42%
Linguistic content60649.75%
Age1,18297.04%
Gender1,17696.55%
Occupation83268.31%
Object category94577.59%
Object material1,21599.75%
Object risk47839.24%
Human-human relation64152.63%
Human-object relation96479.15%
Object-object relation1,11991.87%
Spatial relation1,218100.00%
Human state1,18096.88%
Object state1,06287.19%

As for unconstrained video generation, we observed that it tends to produce common and simple scenarios. For example, when given an "object tipping" event, the model repeatedly generated similar restaurant scenes involving spilled soup or coffee. We encountered such repetitive cases during our video generation process, which further highlights the importance of using our taxonomy to guide video generation and improve scenario diversity.

2.2.2 Taxonomy Derivation

Thank you for the insightful observation.

Our taxonomy draws on established research on contextual understanding. Specifically, Dey and Abowd [1] identify Identity, Location, Activity, and Time as fundamental context types, corresponding to our Human, Place, Event, and Time domains. Augusto [2] further emphasizes the importance of world states, events, and temporal context in intelligent systems. Tong et al. [3] highlight human-object relationships in intention understanding, motivating our Object and Relationship domains.

Building upon these perspectives and RobotEQ's definition of Social Proactive Intelligence (SPI), we developed a four-level taxonomy comprising 6 domains, 20 dimensions, 142 level-1 attributes, and 816 level-2 attributes, aiming to provide a structured representation of the contextual factors influencing SPI judgments.

We also acknowledge that EgoNormia [4] inspired some social norm-related categories within our Event domain. However, our taxonomy extends beyond social norms to characterize broader contextual factors. We have clarified these design motivations and added the relevant citations in the revised paper.

References

[1] Dey and Abowd. Towards a Better Understanding of Context and Context-Awareness. 1999.

[2] Augusto. Contexts and Context-awareness Revisited from an Intelligent Environments Perspective. 2022.

[3] Tong et al. Human Intention Recognition Using Context Relationships in Complex Scenes. 2025.

[4] Rezaei et al. EgoNormia: Benchmarking Physical-Social Norm Understanding. 2025.

Reviewer_KNj5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RobotEQ-Video, a video-centric benchmark for Social Proactive Intelligence (SPI), where a model judges whether candidate robot actions are socially appropriate in an embodied scene, extending the image-based RobotEQ setting to videos. Data construction is guided by a four-level world-state taxonomy (6 domains, 20 dimensions, 142 level-1 and 816 level-2 attributes) used for Seedance-based synthetic video generation and for selecting real egocentric clips, resulting in 2,146 videos (1,218 synthetic / 928 real) and 17,544 binary action labels from majority voting over four annotators.

Critical strengths
  1. Extending SPI from single images to short videos is a well-motivated step, and the taxonomy-guided generation gives explicit control over which contextual factors appear in each scene. The human study in Figure 7 shows a clear gain from video over a single frame, so the benchmark targets information that image-based SPI benchmarks cannot capture. This is the main reason I lean towards acceptance.

  2. The evaluation is broad, covering 48 models on both synthetic and real subsets. Together with the dataset size, this makes the benchmark a useful reference for selecting VLMs for socially aware embodied agents.

Critical weaknesses
  1. It is unclear whether Table 2 measures video-grounded reasoning. The evaluation prompt (Appendix I) asks models to judge "using only the action-option text below", the video/audio input setup is not described, and the necessity of video is only verified for humans (Figure 7). Text-only, single-frame, and majority-class (about 60% "proper") baselines for the models are needed to interpret the model-human gap.

  2. The taxonomy is a core contribution but is barely used in the evaluation: Table 2 reports only aggregate scores, and Figure 8b covers a single model and four dimensions. A per-dimension breakdown across models, and an analysis of whether the counterfactual actions (Section 3.3) are harder, would make the benchmark much more informative.

Questions for authors
  1. (Clarification) The baselines in Figure 9b are much lower than Table 2 for the same models (e.g., Qwen3-VL-30B: 60.2 vs. 77.2/74.4), while the WM-augmented results (e.g., 76.2) are close to the Table 2 numbers. What explains this protocol difference, and how large is the WM gain relative to the Table 2 setting?

  2. (Clarification) Is the Appendix I prompt the one used for Table 2, and how are video frames and audio provided to each model?

  3. (Clarification) The three Keye-VL-1.5-8B variants give nearly identical results, and Kimi-VL-A3B-Thinking links to the Instruct checkpoint. Were the thinking modes actually enabled?

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 2

Weakness 1 and Question 2:

Thank you for your constructive comment. We address your concerns below.

(1) Clarification of the Evaluation Prompt and Input Configuration

We apologize for the wording error in Appendix I, which incorrectly suggested that judgments relied only on action-option text. In fact, all evaluated models received visual information from the corresponding videos. We have corrected the prompt to explicitly include video input:

Evaluation Prompt:

Based on the following video input:

{video_input}

Determine whether each candidate action is appropriate in the current scene.

Candidate Actions:

{options}

Carefully consider each candidate action before making a judgment.

Rules:

  • For each option, output exactly one label: "Should Do" or "Should Not Do".
  • Do not output placeholder text such as "Should Do/Should Not Do".
  • The <Judgment> below is a placeholder and must be replaced with the selected label.

Strictly follow the format below, outputting one line per option without omitting any options:

opt1: <Judgment>
opt2: <Judgment>
...

Regarding the visual input configuration, we follow each model's native input interface. For models that support direct video input, such as Qwen3-VL, Qwen2.5-VL, and GLM-4.5V, we provide the original MP4 files. For models without direct video input support, such as LLaVA-Video, Video-LLaVA, and VideoLLaMA3, we uniformly sample eight frames at evenly spaced temporal intervals throughout each video and provide them as image inputs.

Audio is provided only to models with native audiovisual capabilities. Specifically, for Qwen3-Omni and Qwen2.5-Omni, we extract the audio tracks from the videos and provide them alongside the visual inputs through the models' official audiovisual interfaces. Other vision-language models, such as Qwen3-VL and InternVL3, receive visual inputs only.

Detailed inference parameters, input and output configurations for each model are available in our anonymized repository.

(2) Comparison of Input Modalities

To assess the contribution of video information, we compared identical samples and models under three settings: Action-only, Image+Action, and Video+Action. Image+Action uses each video's first frame. We report Accuracy and Macro-F1, with gains relative to Action-only in parentheses.

Table 1. Results on the synthetic subset.

ModelMetricAction-onlyImage+ActionVideo+Action
GLM-4.6V-FlashAcc66.0667.22 (+1.15)71.27 (+5.21)
Macro-F154.5559.43 (+4.88)65.72 (+11.16)
Keye-VL-1.5-8BAcc66.1668.37 (+2.20)69.60 (+3.43)
Macro-F165.2068.10 (+2.89)69.39 (+4.19)
Qwen3.6-27B-InstructAcc73.3175.50 (+2.19)76.69 (+3.38)
Macro-F173.2875.39 (+2.11)76.60 (+3.31)
Qwen3-VL-8B-ThinkingAcc72.4473.50 (+1.06)74.73 (+2.29)
Macro-F171.7773.48 (+1.71)74.67 (+2.90)

Table 2. Results on the real-world subset.

ModelMetricAction-onlyImage+ActionVideo+Action
Keye-VL-1.5-8BAcc67.6771.77 (+4.10)73.08 (+5.41)
Macro-F166.2971.15 (+4.86)72.47 (+6.18)
InternVL3.5-8B-InstructAcc65.9467.19 (+1.25)69.69 (+3.75)
Macro-F157.7665.64 (+7.88)66.31 (+8.55)
InternVL3.5-4B-InstructAcc61.9263.35 (+1.42)65.58 (+3.66)
Macro-F156.5963.35 (+6.76)64.40 (+7.81)
Qwen3-VL-4B-InstructAcc65.8667.08 (+1.22)68.25 (+2.39)
Macro-F155.1263.37 (+8.26)60.98 (+5.86)

Across both subsets, Video+Action achieves the highest Accuracy for every model.

These results complement Figure 7 and show that video inputs provide useful context beyond a single frame for social appropriateness judgments.

(3) Majority-Class Baseline and Class Imbalance

For the majority-class baseline, we evaluate a constant predictor that always predicts "Should Do", the dataset's majority class.

Table 3. Majority-class baseline on the synthetic and real-world subsets.

DatasetProperImproperAccuracy (%)Macro-F1 (%)
Synthetic6,7424,48360.0637.52
Real3,7272,59258.9837.10
Overall10,4697,07559.6737.37

The baseline reaches 60.06% Accuracy and 37.52% Macro-F1 on synthetic videos and 58.98% Accuracy and 37.10% Macro-F1 on real videos.

We report Accuracy and Macro-F1 in the main Table 2 and controlled experiments. Macro-F1 weights both classes equally, lessening imbalance bias. Notably, all models in the main Table 2 exceed the majority-class Macro-F1 baseline of 37.37%, showing their performance is not solely due to majority-class predictions. Together, these metrics offer a balanced evaluation.

AuthorsOpenReview ↗

Responses to Weakness 2 and Question 1

Response to Weakness 2:

Thank you very much for your valuable suggestion.

(1) Performance Analysis Across Taxonomy Dimensions.

First, we would like to clarify that Figure 8b is intended to investigate the importance of fine-grained contextual information in SPI judgments, rather than directly compare model performance across taxonomy dimensions. Specifically, we examine whether providing additional contextual cues associated with selected dimensions can improve social appropriateness judgments.

Following your suggestion, we further evaluated two representative models, Qwen3-VL-8B and Gemma-4-12B-IT, across all 20 taxonomy dimensions. The results are presented below, and the corresponding radar charts have been included in the appendix of the revised paper.

Table 4. Accuracy (%) of representative models across 20 contextual dimensions.

DimensionQwen3-VL-8BGemma-4-12B-IT
Scene69.274.1
Time68.876.1
Event69.274.1
Action68.373.5
Gaze69.674.8
Expression69.674.0
Gesture68.673.6
Linguistic content69.174.6
Age73.375.0
Gender75.376.4
Occupation69.877.1
Object category69.074.1
Object material63.874.6
Object risk69.275.9
Human-human relation68.772.7
Human-object relation70.375.6
Object-object relation74.876.1
Spatial relation69.773.9
Human state69.674.6
Object state68.673.1

The results reveal distinct performance patterns across contextual dimensions. For example, Qwen3-VL-8B achieves relatively lower accuracy on Object Material (63.8%), while Gemma-4-12B-IT exhibits stronger performance on this dimension (74.6%). These findings demonstrate that our taxonomy supports fine-grained diagnostic analysis of model capabilities and limitations beyond aggregate performance metrics.

(2) Difficulty Analysis of Counterfactual Actions.

We also agree that analyzing the difficulty of counterfactual actions would provide additional insights into the benchmark. In our dataset, the ratio of counterfactual to non-counterfactual actions is approximately 1:5. To investigate their relative difficulty, we compared the performance of three representative models on these two categories.

Table 5. Accuracy (%) on counterfactual (CF) and non-counterfactual (Non-CF) actions.

ModelOverall AccCF AccNon-CF AccNon-CF − CF
Qwen3-VL-8B64.7160.5765.54+4.97
Gemma-4-12B-IT74.0071.3074.51+3.21
Keye-VL-1.5-8B71.2868.4471.85+3.41

All three models achieve lower accuracy on counterfactual actions. These results suggest that counterfactual actions introduce additional challenges for current VLMs and support their inclusion in the benchmark to evaluate context-sensitive social appropriateness judgments.

Overall, these additional analyses demonstrate how our taxonomy can facilitate more detailed evaluation of model capabilities and reveal the additional difficulty associated with counterfactual actions. We have included the corresponding analyses and visualizations in the revised manuscript.

Response to Question 1:

Thank you for raising this important question. We appreciate the opportunity to clarify the discrepancy between Figure 9b and Table 2.

Due to the substantial computational cost of video generation using Wan2.2-I2V-A14B, we selected a subset of challenging samples from the original benchmark, focusing on cases that models frequently misclassified in the Table 2 evaluation. As Figure 9b follows the same model configurations and evaluation protocol as Table 2, its lower baseline performance reflects the greater difficulty of the selected samples rather than differences in evaluation settings.

Furthermore, the baseline and world-model-augmented results in Figure 9b are evaluated on the same subset under identical settings, with world-model augmentation being the only difference.

AuthorsOpenReview ↗

Response to Question 3

Response to Question 3:

Thank you for carefully examining our results and pointing out these issues. We sincerely apologize for the reporting errors in the Keye-VL-1.5-8B results and the incorrect checkpoint link for Kimi-VL-A3B-Thinking. We appreciate the opportunity to clarify and correct these issues.

(1) Clarification of Keye-VL-1.5-8B Thinking Modes.

The three Keye-VL-1.5-8B variants use the same model checkpoint but differ in their inference configurations:

  • No-Think (referred to as Instruct in Table 2): The /no_think instruction is appended to the prompt to disable explicit reasoning.
  • Auto-Thinking: No explicit thinking control instruction is added, allowing the model to determine whether to engage in reasoning.
  • Thinking: The /think instruction is appended to the prompt to explicitly enable reasoning.

We acknowledge that the Auto-Thinking and Thinking results in the original Table 2 contained reporting errors. We have corrected these results in the revised manuscript, as shown below.

Table 6. Corrected Keye-VL-1.5-8B results on the synthetic subset (%).

ModePro PPro RImpro PImpro RAccMacPACWAvg
No-Think77.8067.5656.5868.6967.9967.1865.0175.7268.97
Auto-Thinking78.1456.9351.3374.0463.4363.2561.4771.8565.00
Thinking81.2932.5544.5087.8453.6252.7853.5863.4655.86

Table 7. Corrected Keye-VL-1.5-8B results on the real-world subset (%).

ModePro PPro RImpro PImpro RAccMacPACWAvg
No-Think77.0774.7059.9262.9970.3068.6467.9176.4670.83
Auto-Thinking78.6566.2555.4570.0367.6666.9165.7674.1768.62
Thinking82.9942.7647.0685.3158.6658.5557.6665.8260.17

We have updated these results in the revised PDF.

(2) Clarification of the Kimi-VL-A3B-Thinking Checkpoint.

We sincerely appreciate your careful observation regarding the incorrect checkpoint link for Kimi-VL-A3B-Thinking. We confirm that the dedicated Kimi-VL-A3B-Thinking-2506 checkpoint (https://huggingface.co/moonshotai/Kimi-VL-A3B-Thinking-2506) was correctly used during evaluation, with its thinking mode enabled. The issue was limited to an incorrect link to the Instruct checkpoint in the appendix and did not affect the evaluation results. We have corrected this link in the revised manuscript.

Furthermore, we have carefully rechecked the inference configurations and checkpoint references for all other Thinking and Instruct variants and confirmed their correctness.

All of my concerns have been addressed. I will keep my positive score.

Reviewer_ptz3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces RobotEQ-Video, a video-centric benchmark for Social Proactive Intelligence (SPI), which evaluates the social appropriateness of robot actions in diverse embodied scenarios. The authors propose a hierarchical world-state taxonomy to guide dataset construction, resulting in 2K+ real and synthetic videos with over 17K annotated actions. The paper evaluates 48 models and investigates the use of world models to improve action judgment.

Critical strengths
  1. Relevant research problem. Evaluating socially appropriate proactive behavior is important for developing intelligent embodied agents. Extending SPI evaluation from static images to videos is a meaningful direction.

  2. Structured dataset construction. The hierarchical taxonomy provides a systematic framework for organizing diverse contextual factors, while the inclusion of both real and synthetic videos increases scenario diversity.

  3. Extensive benchmarking. The evaluation covers a broad range of multimodal models and provides useful observations about their limitations in socially appropriate decision-making.

Critical weaknesses
  1. Unclear novelty and contributions. The contributions are not sufficiently distinguished from existing benchmarks, particularly RobotEQ. Extending an existing benchmark from images to videos and introducing a taxonomy do not, by themselves, establish substantial novelty. The paper should clearly explain what is fundamentally new compared with prior work and why these differences matter. Moreover, the listed contributions mainly describe what has been implemented rather than identifying the underlying technical or scientific advances. Experimental results should be presented as findings rather than contributions.

  2. Insufficient validation of the world-state taxonomy. Although the taxonomy contains 816 fine-grained attributes, its effectiveness is not convincingly demonstrated. The paper does not sufficiently establish whether the taxonomy improves dataset coverage, diversity, or evaluation quality compared with simpler or existing categorization approaches. The quantitative analysis in Figure 8 does not directly validate the proposed taxonomy.

  3. Limited evidence for the necessity of video. Figure 7 compares video, image, and action-only inputs, but the experiments are conducted on different subsets and primarily involve human evaluation. More controlled comparisons using identical samples and representative models are needed to establish the benefits of temporal information over static images.

  4. Concerns regarding annotation reliability and generalizability. The benchmark relies on eight annotators, with each video evaluated by four individuals. Given the subjective nature of social appropriateness, stronger validation of annotation consistency is needed. Furthermore, the restriction to Chinese cultural contexts limits the generalizability of the benchmark (this is somehow undertandable though).

  5. Insufficient experimental analysis. Despite evaluating numerous models, the paper mainly reports aggregate performance without sufficiently investigating failure cases or the influence of different contextual factors. The world-model augmentation experiment is also preliminary, lacking analysis of the performance difference.

  6. Presentation quality. Several figures, particularly Figures 1–3 and 8, contain excessively small text, making them difficult to interpret. The overall presentation would benefit from clearer figures and more focused explanations of the main contributions and experimental findings.

Overall, the paper addresses a relevant problem and presents a reasonably extensive benchmark. However, its novelty beyond existing work remains insufficiently established, and several central claims, particularly the effectiveness of the world-state taxonomy and the necessity of video-centric evaluation, lack convincing experimental validation. Concerns regarding annotation reliability, evaluation methodology, and limited analytical depth further weaken the contribution.

Questions for authors
  1. What are the main novel contributions beyond RobotEQ (except from image to video) and other existing video-based proactive assistance benchmarks (Table 1 only provides a binary comparison for each criterion) ?

  2. How can the authors demonstrate that the proposed world-state taxonomy provides measurable advantages over simpler or existing taxonomies?

  3. Can the authors provide controlled image-versus-video comparisons using the same evaluation samples and models to quantify the importance of temporal information?

  4. How is annotation reliability established, particularly for socially ambiguous scenarios? Can the authors also clarify the evaluation prompt in Appendix I and whether all models receive the video input?

2 comments
AuthorsOpenReview ↗

Responses to Weaknesses 1–3 and Questions 1–3

Response to Weakness 1 and Question 1:

Thank you for your thoughtful comment. We would like to further clarify the motivation and contributions of our work.

First, we use videos rather than static images as in RobotEQ because videos capture temporal cues about evolving human states and needs, offering richer context than isolated images.

Furthermore, unconstrained video generation may favor common scenarios and overlook important contextual factors. We therefore introduce a world-state taxonomy to guide more systematic and diverse video generation. The resulting benchmark dataset also serves as a valuable resource for SPI research.

Additionally, we explore world-model-based augmentation, which generates potential future visual outcomes of candidate actions to support social appropriateness judgments, offering a promising direction for improving SPI capabilities.

Finally, by organizing contextual factors into six domains (Event, Time, Place, Relationship, Human, and Object), we further operationalize the broad concept of SPI through a systematic, fine-grained evaluation framework for embodied intelligence.

Regarding our experimental contributions, our work goes beyond extensive evaluations. In addition to investigating the limitations of current VLMs in SPI, we explore world-model-based augmentation as a potential solution for improving socially appropriate action judgments.

Response to Weakness 2 and Question 2:

Thank you for your feedback. We would like to provide additional evidence demonstrating the effectiveness of our world-state taxonomy in constructing diverse scenarios.

First, we conducted an additional human evaluation to verify taxonomy coverage. Four annotators independently examined the 20 dimensions and their Level-1 and Level-2 attributes in 1,218 generated videos. An attribute was considered present when at least three of four annotators agreed. The results confirm 100% coverage of all 142 Level-1 and 816 Level-2 attributes across the dataset. Compared with EgoNormia [1], which focuses on seven social norm categories, our taxonomy considers a broader range of contextual factors from multiple perspectives, better capturing the complexity and multifaceted nature of SPI. The detailed prevalence of each contextual dimension is presented in Table 1.

Table 1. Prevalence of the 20 Contextual Dimensions.

DimensionCountPercentage
Scene1,20598.93%
Time1,04685.88%
Event1,08589.08%
Action1,13493.10%
Gaze1,18297.04%
Expression1,17296.22%
Gesture40733.42%
Linguistic content60649.75%
Age1,18297.04%
Gender1,17696.55%
Occupation83268.31%
Object category94577.59%
Object material1,21599.75%
Object risk47839.24%
Human-human relation64152.63%
Human-object relation96479.15%
Object-object relation1,11991.87%
Spatial relation1,218100.00%
Human state1,18096.88%
Object state1,06287.19%

Regarding unconstrained video generation, we observed that it tends to produce common and simple scenarios. For example, given an "object tipping" event, the model repeatedly generated similar restaurant scenes involving spilled soup or coffee. These repetitive cases highlight the importance of using our taxonomy to guide video generation and improve scenario diversity.

Response to Weakness 3 and Question 3:

Thank you for your suggestion.

To address your concern about the limited evidence for video inputs, we conducted additional experiments across three settings: Action-only, Image+Action, and Video+Action. Image+Action uses the first frame of each video. For each model, all settings used identical samples, differing only in input modality. Results for representative VLMs on both subsets are presented below.

Table 2. Accuracy (%) on the synthetic subset.

ModelAction-onlyImage+ActionVideo+Action
GLM-4.6V-Flash66.0667.2271.27
Keye-VL-1.5-8B66.1668.3769.60
Qwen3.6-27B-Instruct73.3175.5076.69
Qwen3-VL-8B-Thinking72.4473.5074.73

Table 3. Accuracy (%) on the real subset.

ModelAction-onlyImage+ActionVideo+Action
Keye-VL-1.5-8B67.6771.7773.08
InternVL3.5-8B-Instruct65.9467.1969.69
InternVL3.5-4B-Instruct61.9263.3565.58
Qwen3-VL-4B-Instruct65.8667.0868.25

Across all evaluated models on both subsets, Video+Action consistently achieves the highest accuracy, followed by Image+Action and Action-only. These comparisons provide additional evidence that videos offer valuable contextual information beyond a single static frame for socially appropriate action judgments, complementing the findings from our human study in Figure 7.

AuthorsOpenReview ↗

Responses to Weaknesses 4–6 and Question 4

Response to Weakness 4:

Thank you for your valuable comment. All eight annotators were carefully selected from undergraduate and master's students and received training on social norms and embodied intelligence to ensure a consistent understanding of SPI and its evaluation criteria.

To further assess annotation reliability, we calculated pairwise agreement on overlapping annotations, as shown below. The average pairwise agreement rate is 83.61%, indicating generally high consistency and supporting the reliability of our annotations and ground-truth labels.

Table 4. Pairwise Agreement (%) Among Eight Annotators

Annotator12345678
1—81.782.685.884.786.089.593.1
281.7—74.778.684.384.379.080.8
382.674.7—83.278.481.783.482.7
485.878.683.2—82.883.487.486.2
584.784.378.482.8—83.981.583.6
686.084.381.783.483.9—84.584.8
789.579.083.487.481.584.5—88.5
893.180.882.786.283.684.888.5—

Response to Weakness 5:

To address your concern, we conducted additional case studies to examine current VLM limitations and the potential benefits of world-model-based augmentation.

1. Failure Case Analysis. We examined six representative models and identified three common failure patterns:

  • Insufficient Awareness of Human Agency and Intervention Timing. Models sometimes favor premature interventions, such as adjusting a ladder while someone is using it or turning a page before the reader has finished, overlooking ongoing human activities and appropriate intervention timing.
  • Overconservative Physical Intervention. Models sometimes reject appropriate physical assistance, such as extending a robotic arm to catch a falling phone, while preferring verbal responses.
  • Failure to Respect Social Interaction Boundaries. Models sometimes endorse unsolicited interventions, such as interrupting conversations or commenting on someone laughing at their phone, without adequately considering social boundaries.

2. World Model Augmentation Analysis. We observed two representative patterns:

  • Recognizing Appropriate Assistance. Generated video continuations can help models reconsider previously rejected but appropriate actions, such as delivering medical equipment or handing records to healthcare workers.
  • Identifying Inappropriate Interventions. Generated video continuations can also help models recognize inappropriate actions, such as intrusive recording or obstructing others.

These findings provide further insights into the limitations of current VLMs in SPI and illustrate how world-model-based augmentation may improve social appropriateness judgments. We have included these additional case studies in the revised manuscript's appendix.

Response to Weakness 6:

Thank you for your suggestion. To improve readability, we included enlarged versions of figures with small text in the appendix of the revised paper.

Response to Question 4:

Regarding annotation reliability, as clarified in Weakness 4, all annotators received training on social norms and embodied intelligence to ensure consistent understanding of SPI and its evaluation criteria. Furthermore, ground-truth labels were established through majority voting, requiring at least three out of four annotators to agree on the same judgment. Actions with insufficient agreement or any "Not Sure" judgments were excluded. These measures improve annotation reliability, as supported by the agreement analysis in our response to Weakness 4.

Regarding the evaluation prompt in Appendix I, we apologize for the wording error that may have caused confusion. We would like to clarify that all evaluated models received visual information from the corresponding videos along with the candidate actions. The error was limited to the prompt description and did not affect the evaluation procedure. The corrected prompt in the revised PDF is provided below:

Evaluation Prompt:

Based on the following video input:

{video_input}

Determine whether each candidate action is appropriate in the current scene.

Candidate Actions:

{options}

Carefully consider each candidate action before making a judgment.

Rules:

  • For each option, output exactly one label: "Should Do" or "Should Not Do".
  • Do not output placeholder text such as "Should Do/Should Not Do".
  • The <Judgment> below is a placeholder and must be replaced with the selected label.

Strictly follow the format below, outputting one line per option without omitting any options:

opt1: <Judgment>
opt2: <Judgment>
...

[1] Rezaei M H, Fu Y, Cuvin P, et al. EgoNormia: Benchmarking physical-social norm understanding. ACL 2025. 2025: 19256–19283.

Comments on the paper

1 comment

Dear reviewers

Please read the authors' rebuttal to see whether or not your concerns have been resolved, and post your opinions.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.