- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RobotEQ, a benchmark to evaluate Social Proactive Intelligence (SPI) in embodied agents: the ability to act appropriately in complex social scenarios without being explicitly prompted to do so. The dataset comprises 1,812 synthetic scenes and 270 real-world scenarios, spanning seven different social aspects. Evaluation focuses on behavior judgment, assessing whether candidate actions are appropriate, and spatial grounding, evaluating whether an agent can select appropriate regions in a scene. The benchmark includes general-purpose and task-specific VLMs, revealing a substantial gap relative to human performance.
Critical strengths
-
Importance and motivation of the problem. The benchmark addresses a well-motivated problem. As generalist models and increasingly capable VLMs are integrated into embodied AI, assessing their judgment and social understanding becomes important for deploying robots in socially complex real-world scenarios. I consider this a strong motivation for acceptance: embodied agents should recognize appropriate behavior and understand social expectations, beyond completing assigned tasks.
-
Benchmark design and data collection effort. The benchmark is carefully constructed around complex social scenarios in which robot deployment is increasingly plausible. Its 1,812 synthetic and 270 real-world scenarios, supported by more than 25,000 human annotations, represent a substantial collection and curation effort. This breadth and human validation make it a potentially valuable resource for the community and are a major positive factor in my assessment.
Critical weaknesses
-
The spatial grounding experiment is inconsistently described. The main paper and Appendix F describe selecting appropriate people, regions, or movement paths in physical social scenarios. Appendix J instead describes identifying the bounding box of a target UI element and mentions metrics that differ from those reported in Table 6. The authors should clarify which dataset and evaluation procedure produced Figure 5 and Table 6, and whether this discrepancy is a description error or reflects a deliberately different experiment. If the latter, its relevance to SPI should be explained.
-
Some conclusions about model differences lack statistical support. The discussion at L317–320 interprets small score differences as evidence of model superiority or regression between versions. For example, Opus 4.6 and Sonnet 4.6 differ by only 0.05 percentage points in average Macro-F1. Observed rankings can be reported, but broader conclusions such as “newer versions do not always bring improvements on SPI,” illustrated by the comparison between Opus 4.7 and Opus 4.6, require uncertainty estimates. Paired confidence intervals for the performance differences, obtained by resampling scenarios, would help establish whether these conclusions are supported.
-
The evaluated checkpoints need clarification. Several links in Table 4 point to different model sizes from those reported, including the Qwen, Gemma, LLaVA-OneVision, and Aya-Vision models. These may simply be incorrect links, but the authors should confirm the exact evaluated checkpoints to make the comparisons reproducible.
Minor weaknesses
- Figure 1 contains “The clint” instead of “The client.”
- Figure 9 is difficult to read at its current size. Arranging the six plots in two rows would improve readability.
- Figure 10 has a similar readability issue and would benefit from a 2×2 layout with larger images and text.
- The related work discussion could be broadened: At L099, the VLN discussion would benefit from recent work such as NaVILA: Legged Robot Vision-Language-Action Model for Navigation [1], alongside the foundational references, to provide a more current picture of embodied vision-language models. The discussion of social intelligence could also acknowledge approaches that infer human social dynamics from robot observations, such as Following the Human Thread in Social Navigation [2], and approaches involving robot-initiated interaction, such as Robots Ask the Way: Communication-Enabled Social Navigation [3]. These directions remain distinct from RobotEQ’s scope, which I find well motivated, but discussing them would help clarify its position within the broader literature.
Suggested references [1] Cheng et al. NaVILA: Legged Robot Vision-Language-Action Model for Navigation. RSS, 2025. https://www.roboticsproceedings.org/rss21/p018.html [2] Scofano et al. Following the Human Thread in Social Navigation. ICLR, 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/76b010c4d21f97b7b113383ce7160cbc-Abstract-Conference.html [3] Sacco et al. Robots Ask the Way: Communication-Enabled Social Navigation, IROS, 2026. https://arxiv.org/abs/2607.01044
Questions for authors
- How does the experiment in Appendix J relates to SPI?
- At L044, the statement that existing research is “confined to limited scenarios and primarily emphasizes task execution” needs supporting citations and concrete examples. Which approaches and limitations are the authors referring to?
- Could the authors clarify whether the reported human scores come from independent raters or from annotators whose judgments contributed to the reference labels?
- Equation 2 implies that the Consensus-Weighted Probability of Agreement (CWPA) should be greater than or equal to the Probability of Agreement (PA). However, Table 1 contains cases where CWPA is lower than PA. Could the authors explain this discrepancy?
I currently lean towards weak rejection. I consider the problem important and the benchmark valuable, but these concerns prevent me from supporting acceptance at this stage. I would be happy to reconsider my score if the authors satisfactorily address them.
2 comments
Response to Reviewer kVkq
Dear Reviewer kVkq,
The author-reviewer discussion period will soon be closed. We kindly ask that you review our responses, and we welcome further discussion.
Thanks for your time and reconsideration.
Best regards,
Authors of Submission 5674
Response to Reviewer
Response to Weakness 1 and Question 1 (Inconsistent Description of Spatial Grounding Experiment)
We sincerely thank the reviewer for carefully cross-checking the descriptions across different sections. Appendix J previously contained some incorrect expressions. We have updated the corresponding content in Appendix J in our revised manuscript, with the following main changes:
(1) Expression Correction. We have replaced the phrase "the bounding box of a target UI element" with "regions and areas" to ensure consistency with the main text.
(2) Metric Correction. We adopted the same evaluation metrics as those used in the main text and updated the corresponding experimental results.
Response to Weakness 2
We appreciate the reviewer’s comment on the discussion at L317–320 and would like to offer a broader perspective on the relationship between general model improvements and social proactive intelligence (SPI).
Official evidence of Opus 4.7’s general improvements: According to the official Anthropic System Card, Opus 4.7 is explicitly described as “stronger across the board” compared to 4.6 in core capabilities such as software engineering, real-world professional tasks, long-horizon agentic tasks, and high-resolution vision. Quantitative evidence includes a 13% increase in resolution rate on an internal 93-task coding benchmark, approximately 3× higher visual input resolution, and internal visual accuracy rising from 54.5% to 98.5%.
Observation on SPI vs. general capabilities: Given such substantial gains, one would naturally expect a synchronous increase in RobotEQ scores if SPI evolved in the same direction. However, our measurements show that 4.7 actually scored slightly lower than 4.6 (a 1.65-point gap, which is not statistically significant). We interpret this not as regression, but as an indication that significant improvements in general coding, reasoning, and vision capabilities do not automatically transfer to the SPI level. This observation directly echoes our core argument—SPI needs to be modeled, evaluated, and optimized separately, rather than assuming that advances in general reasoning will incidentally bring improvements in social capabilities.
Revision plan: In the revised version, we will rewrite the relevant paragraph (L317–320) to frame this as a discussion of the expectation gap, and clearly mark it as an observational conclusion rather than a statistical assertion.
Response to Weakness 3
We thank the reviewer for the comment and have updated the correct links for each model accordingly.
Response to Minor weaknesses
We thank the reviewer for the comment and have updated the corresponding parts in the revised manuscript. The revisions include correcting spelling mistakes, enlarging figures, and incorporating additional related works.
Response to Question 3
We thank the reviewer for raising this important question, which prompts us to clarify the construction process of the human baseline and the quality control mechanisms of our annotation.
1. Complete Independence of Human Baseline Participants
The human baseline scores reported in Table 1 were obtained from a group of participants who are completely independent of the original annotation pipeline. Specifically:
Response to Q4 (Discrepancy between CWPA and PA)
We apologize for the confusion. After careful verification, we confirmed that during manual entry of model metrics, the PA values for two models on the Synthetic subset were incorrectly registered.
All other entries in Table 1 remain correct and unchanged. We sincerely apologize for this clerical error. In the revised version, we will open-source all raw inference logs and evaluation scripts in our code repository to ensure full transparency and reproducibility. The PDF will be updated accordingly to reflect these corrections.