acceptodds
Under review as a conference paper at ICLR 2027

RobotEQ: Towards Social Proactive Intelligence in Embodied Agents

Abstract

Embodied agents represent a prominent research focus across both academia and industry. The prevailing paradigm has gradually shifted from *reactive assistance*, which requires explicit user queries, to *proactive assistance*, capable of recognizing human needs and offering support without explicit instructions. Nevertheless, existing studies on *proactive assistance* remain confined to narrow scenarios and primarily emphasize task completeness, whereas real-world agents must operate in open-domain environments while adhering to social expectations. To bridge this gap, we extend the concept of *proactive assistance* to **Social Proactive Intelligence (SPI)**, characterized by diverse scenarios, social understanding, and robot-centric behaviors. We further introduce **RobotEQ**, a dedicated benchmark for SPI. We first define two tasks: *behavior judgment*, emphasizing global contextual understanding, and *spatial grounding*, focusing on local perceptual details. Building on these tasks, we construct **RobotEQ-Data**, a dataset comprising 1,812 synthetic and 270 real-world scenarios, 7 social facets, 25K+ human annotations, 3K+ behavior judgment questions, and 3K+ spatial grounding questions. Furthermore, we establish **RobotEQ-Bench** to evaluate the performance of representative models. Experimental results demonstrate that current models fall short of achieving reliable SPI. Further analysis reveals that incorporating external social knowledge yields consistent improvements. This work aims to advance the development of socially desirable embodied agents in open-domain environments.

Primary area:
datasets and benchmarks
Keywords:
Social Proactive Intelligence, dataset, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kVkqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RobotEQ, a benchmark to evaluate Social Proactive Intelligence (SPI) in embodied agents: the ability to act appropriately in complex social scenarios without being explicitly prompted to do so. The dataset comprises 1,812 synthetic scenes and 270 real-world scenarios, spanning seven different social aspects. Evaluation focuses on behavior judgment, assessing whether candidate actions are appropriate, and spatial grounding, evaluating whether an agent can select appropriate regions in a scene. The benchmark includes general-purpose and task-specific VLMs, revealing a substantial gap relative to human performance.

Critical strengths
  • Importance and motivation of the problem. The benchmark addresses a well-motivated problem. As generalist models and increasingly capable VLMs are integrated into embodied AI, assessing their judgment and social understanding becomes important for deploying robots in socially complex real-world scenarios. I consider this a strong motivation for acceptance: embodied agents should recognize appropriate behavior and understand social expectations, beyond completing assigned tasks.

  • Benchmark design and data collection effort. The benchmark is carefully constructed around complex social scenarios in which robot deployment is increasingly plausible. Its 1,812 synthetic and 270 real-world scenarios, supported by more than 25,000 human annotations, represent a substantial collection and curation effort. This breadth and human validation make it a potentially valuable resource for the community and are a major positive factor in my assessment.

Critical weaknesses
  • The spatial grounding experiment is inconsistently described. The main paper and Appendix F describe selecting appropriate people, regions, or movement paths in physical social scenarios. Appendix J instead describes identifying the bounding box of a target UI element and mentions metrics that differ from those reported in Table 6. The authors should clarify which dataset and evaluation procedure produced Figure 5 and Table 6, and whether this discrepancy is a description error or reflects a deliberately different experiment. If the latter, its relevance to SPI should be explained.

  • Some conclusions about model differences lack statistical support. The discussion at L317–320 interprets small score differences as evidence of model superiority or regression between versions. For example, Opus 4.6 and Sonnet 4.6 differ by only 0.05 percentage points in average Macro-F1. Observed rankings can be reported, but broader conclusions such as “newer versions do not always bring improvements on SPI,” illustrated by the comparison between Opus 4.7 and Opus 4.6, require uncertainty estimates. Paired confidence intervals for the performance differences, obtained by resampling scenarios, would help establish whether these conclusions are supported.

  • The evaluated checkpoints need clarification. Several links in Table 4 point to different model sizes from those reported, including the Qwen, Gemma, LLaVA-OneVision, and Aya-Vision models. These may simply be incorrect links, but the authors should confirm the exact evaluated checkpoints to make the comparisons reproducible.

Minor weaknesses

  • Figure 1 contains “The clint” instead of “The client.”
  • Figure 9 is difficult to read at its current size. Arranging the six plots in two rows would improve readability.
  • Figure 10 has a similar readability issue and would benefit from a 2×2 layout with larger images and text.
  • The related work discussion could be broadened: At L099, the VLN discussion would benefit from recent work such as NaVILA: Legged Robot Vision-Language-Action Model for Navigation [1], alongside the foundational references, to provide a more current picture of embodied vision-language models. The discussion of social intelligence could also acknowledge approaches that infer human social dynamics from robot observations, such as Following the Human Thread in Social Navigation [2], and approaches involving robot-initiated interaction, such as Robots Ask the Way: Communication-Enabled Social Navigation [3]. These directions remain distinct from RobotEQ’s scope, which I find well motivated, but discussing them would help clarify its position within the broader literature.

Suggested references [1] Cheng et al. NaVILA: Legged Robot Vision-Language-Action Model for Navigation. RSS, 2025. https://www.roboticsproceedings.org/rss21/p018.html [2] Scofano et al. Following the Human Thread in Social Navigation. ICLR, 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/76b010c4d21f97b7b113383ce7160cbc-Abstract-Conference.html [3] Sacco et al. Robots Ask the Way: Communication-Enabled Social Navigation, IROS, 2026. https://arxiv.org/abs/2607.01044

Questions for authors
  • How does the experiment in Appendix J relates to SPI?
  • At L044, the statement that existing research is “confined to limited scenarios and primarily emphasizes task execution” needs supporting citations and concrete examples. Which approaches and limitations are the authors referring to?
  • Could the authors clarify whether the reported human scores come from independent raters or from annotators whose judgments contributed to the reference labels?
  • Equation 2 implies that the Consensus-Weighted Probability of Agreement (CWPA) should be greater than or equal to the Probability of Agreement (PA). However, Table 1 contains cases where CWPA is lower than PA. Could the authors explain this discrepancy?

I currently lean towards weak rejection. I consider the problem important and the benchmark valuable, but these concerns prevent me from supporting acceptance at this stage. I would be happy to reconsider my score if the authors satisfactorily address them.

2 comments
AuthorsOpenReview ↗

Response to Reviewer

Response to Weakness 1 and Question 1 (Inconsistent Description of Spatial Grounding Experiment)

We sincerely thank the reviewer for carefully cross-checking the descriptions across different sections. Appendix J previously contained some incorrect expressions. We have updated the corresponding content in Appendix J in our revised manuscript, with the following main changes:

(1) Expression Correction. We have replaced the phrase "the bounding box of a target UI element" with "regions and areas" to ensure consistency with the main text.

(2) Metric Correction. We adopted the same evaluation metrics as those used in the main text and updated the corresponding experimental results.

Response to Weakness 2

We appreciate the reviewer’s comment on the discussion at L317–320 and would like to offer a broader perspective on the relationship between general model improvements and social proactive intelligence (SPI).

  • Official evidence of Opus 4.7’s general improvements: According to the official Anthropic System Card, Opus 4.7 is explicitly described as “stronger across the board” compared to 4.6 in core capabilities such as software engineering, real-world professional tasks, long-horizon agentic tasks, and high-resolution vision. Quantitative evidence includes a 13% increase in resolution rate on an internal 93-task coding benchmark, approximately 3× higher visual input resolution, and internal visual accuracy rising from 54.5% to 98.5%.

  • Observation on SPI vs. general capabilities: Given such substantial gains, one would naturally expect a synchronous increase in RobotEQ scores if SPI evolved in the same direction. However, our measurements show that 4.7 actually scored slightly lower than 4.6 (a 1.65-point gap, which is not statistically significant). We interpret this not as regression, but as an indication that significant improvements in general coding, reasoning, and vision capabilities do not automatically transfer to the SPI level. This observation directly echoes our core argument—SPI needs to be modeled, evaluated, and optimized separately, rather than assuming that advances in general reasoning will incidentally bring improvements in social capabilities.

  • Revision plan: In the revised version, we will rewrite the relevant paragraph (L317–320) to frame this as a discussion of the expectation gap, and clearly mark it as an observational conclusion rather than a statistical assertion.

Response to Weakness 3

We thank the reviewer for the comment and have updated the correct links for each model accordingly.

Response to Minor weaknesses

We thank the reviewer for the comment and have updated the corresponding parts in the revised manuscript. The revisions include correcting spelling mistakes, enlarging figures, and incorporating additional related works.

Response to Question 3

We thank the reviewer for raising this important question, which prompts us to clarify the construction process of the human baseline and the quality control mechanisms of our annotation.

1. Complete Independence of Human Baseline Participants

The human baseline scores reported in Table 1 were obtained from a group of participants who are completely independent of the original annotation pipeline. Specifically:

  • The annotators who established the ground-truth (GT) labels are undergraduate, master's, and PhD students with relevant background knowledge, rigorously screened for the task.
  • The evaluators for the human baseline were 12 independent annotators randomly selected from a larger candidate pool. They had never participated in the GT annotation process and had no prior knowledge of the "correct answers."
  • The PA/CWPA scores for the human baseline are the average performance of these 12 individuals, with zero information leakage. The two groups are fully isolated in terms of identity, recruitment channel, and task phase, complying with standard protocols.

Response to Q4 (Discrepancy between CWPA and PA)

We apologize for the confusion. After careful verification, we confirmed that during manual entry of model metrics, the PA values for two models on the Synthetic subset were incorrectly registered.

  • Nanonets-OCR-s (Synthetic): CWPA = 46.64, PA = 45.62
  • Qwen-VL-Plus (Synthetic): CWPA = 48.31, PA = 47.28

All other entries in Table 1 remain correct and unchanged. We sincerely apologize for this clerical error. In the revised version, we will open-source all raw inference logs and evaluation scripts in our code repository to ensure full transparency and reproducibility. The PDF will be updated accordingly to reflect these corrections.

AuthorsOpenReview ↗

Response to Reviewer kVkq

Dear Reviewer kVkq,

The author-reviewer discussion period will soon be closed. We kindly ask that you review our responses, and we welcome further discussion.

Thanks for your time and reconsideration.

Best regards,

Authors of Submission 5674

Reviewer_kj75OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Social Proactive Intelligence (SPI), which asks embodied agents to recognize needs, complete tasks, and follow social expectations in open-domain settings. The accompanying RobotEQ benchmark has 2,082 robot-view scenarios across 10 categories and 56 subcategories. Of these, 1,812 are generated with Gemini and reviewed automatically and by humans; the other 270 are real images from JRDB-Social and filtered web searches. RobotEQ has two tasks. For behavior judgment (3,278 questions), three annotators label LLM-generated candidate behaviors as proper or improper. For spatial grounding (3,300 questions), A–D candidate regions or paths are overlaid on an image, and the model selects the socially appropriate ones. Each candidate is treated as an independent binary unit. Evaluation uses Macro/Micro-F1 against majority labels and PA/CWPA to account for annotator disagreement. Behavior judgment covers 30 VLMs (31 configurations, with Gemma-3 at 4B and 12B); GPT-5.5 performs best with an average Macro-F1 of 73.7, compared with 90.5 for humans. Spatial grounding covers 14 of these models, which score 43.8–60.5 Macro-F1, compared with 84.4 for humans. The paper also studies image necessity, synthetic-to-real SFT, visual perturbations, per-facet performance, error types, and RAG.

Critical strengths

S1. Broad coverage and a fairly careful annotation process. RobotEQ spans 10 categories and 56 subcategories through two complementary tasks. Each item is labeled by three screened annotators, and invalid items are removed. Fig. 7a is also a useful check: without the image, human Macro-F1 falls from 90.7 to about 35–68, suggesting that most items need visual evidence rather than text-only common sense.

S2. A large evaluation of recent models. For behavior judgment, the paper tests 30 VLMs, including the latest closed-source models from 2026, and reports per-facet results, error types, visual perturbations, and a RAG analysis. The best model remains about 17 Macro-F1 points behind humans, with sensitivity as the hardest facet. This is a useful snapshot for work on social reasoning in embodied agents.

Critical weaknesses

W1. The relation to prior work and the construct validity need more clarification. EgoNormia (Rezaei et al., ACL 2025 Findings) already evaluates VLMs on physical-social norm understanding using real egocentric videos. Five of its seven categories directly overlap with the facets here, and it explicitly includes proactivity. It also reports a retrieval-based improvement similar to the RAG finding in this paper, but I could not find a citation or comparison.

Also, every item states the robot's role and gives candidate behaviors to judge one by one. The model does not need to notice a need on its own or decide whether to intervene. From my reading, this means that "Proactive Execution," the first property in the SPI definition, is not tested.

For embodiment, the input is a single static image and the platform's form and capabilities are unspecified; many real images are human eye-level web POV photos, yet whether actions such as tapping a shoulder or closing a door with an arm are appropriate depends on the platform, and the paper does not show that RobotEQ scores relate to a robot's actual behavior.

I suggest clarifying what RobotEQ adds beyond EgoNormia and limiting the "proactive" and "embodied" claims to what the benchmark measures.

W2. Spatial grounding may have positional bias, and the human reference is unclear. I may be reading Fig. 4a incorrectly, but it seems to show that A, B, C, and D are labeled correct in 86.5%, 48.6%, 25.8%, and 17.3% of cases. If so, a rule that always selects only A, without looking at the image, gets a Macro-F1 of 71.0 on the paper's primary metric, higher than every model in Table 6 (best 60.52). Under the usual two-class definition, its Micro-F1 equals accuracy, 73.7, which is also above the best 70.12. Results such as "closed-source models have no clear advantage" and "spatial grounding is harder" may therefore reflect calibration to option position more than social-spatial reasoning.

I am also unsure how to reconcile the 92.8–99.1% pairwise annotator agreement in Fig. 4d with a human PA of only 81–84%. The paper does not say who provides the human performance, which items they evaluated, how the score is computed, or whether these are the same annotators who produced the majority labels. I suggest confirming the meaning of Fig. 4a, reporting position-prior and random baselines, re-evaluating spatial grounding with balanced or randomized option positions, and explaining how the human reference is computed.

Questions for authors

Q1 (W2): Do the A/B/C/D numbers in Fig. 4a show how often each option is labeled correct? If so, what does the "always select only A" rule score on the Table 6 metrics, and how do model results change after shuffling the option order?

Q2 (W2): Who provides the human performance, on which items, and how is it computed? Did these people help produce the majority-vote labels? How do the 93–99% pairwise agreements in Fig. 4d correspond to a human PA of 81–84%? Could you report Fleiss' kappa?

Q3 (W1): What does RobotEQ measure that EgoNormia does not, and do models rank similarly on the two benchmarks?

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 (Relation to EgoNormia, Proactive Execution, and Embodiment Claims)

Response to Weakness 1 (Relation to EgoNormia, Proactive Execution, and Embodiment Claims)

1. Fundamental Differences with EgoNormia

Thanks for your comments. These two benchmarks differ fundamentally in task definition. Specifically, EgoNormia centers on human-human interactions, while RobotEQ focuses on human-robot interaction. They have totally different perception and action spaces. More details are illustrated in Figure 2 in the revised manuscript. For example, for the action space, humans can eat food, but robots cannot eat humans' food generally. This basic difference shifts them into different tasks. If embodied agents come into our daily life, this work can help to define social boundaries for the embodied agents under diverse scenarios.

2. Clarification of Proactive Execution

The concept of "Proactive" is drawn from prior works of "Proactive Assistance", which aims to actively recognize human needs and execute according to changes in the scenario. The counterpart of "Proactive Assistance" is "Reactive Assistance", which aims to passively execute the instructions issued by humans. In RobotEQ, the concept of "Proactive" is aligned with "Proactive Assistance", aiming to determine the proper and improper actions under specific scenarios, without human interfaces. Using candidate behavior judgment to evaluate proactive intelligence is a common paradigm in the field, such as Smart Help [1], Attentive Support [2], and ProVox [3]. This is a widely accepted evaluation paradigm.

References [1] Cao, Z., et al. "Smart Help: Strategic Opponent Modeling for Proactive and Adaptive Robot Assistance in Households." CVPR, 2024.

[2] To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactions. ACM Transactions on Human-Robot Interaction (THRI), 2020.

[3] Grannen, J., et al. "ProVox: Personalization and Proactive Planning for Situated Human-Robot Collaboration." IEEE Robotics and Automation Letters (RA-L), 2025.

3. Clarification of Embodiment

Robot-Centric Viewpoints. First, we emphasize that the majority of images adopt a robot-centric viewpoint. For synthetic images, we apply a strict filtering process to ensure robot-centric perspectives, which constitute the largest portion of our dataset. For real images, due to acquisition challenges, we collect data from two sources: robot-captured and web-searched images. Robot-captured images are selected from JRDB-Social, a widely used robotics dataset. For web-searched images, we construct a large candidate pool from multiple image retrieval platforms, designing specific search queries for each subcategory to ensure diversity. A multi-stage filtering pipeline is then applied to remove images that do not reflect a robot-centric perspective or are unsuitable for SPI evaluation.

Robot Platform Unrestricted. RobotEQ focuses on high-level semantic planning, decoupled from specific hardware constraints. For instance, we provide semantic-level instructions rather than hardware-dependent atomic actions tailored to a particular robot platform. As illustrated in Figure 1, our focus lies in the planning component, the "brain" of embodied agents. Whether an agent can physically execute the corresponding actions depends on downstream platforms, which falls outside the scope of this work and will be addressed in future research.

AuthorsOpenReview ↗

Response to Weakness 2 (Position Bias and Human Baseline Explanation for Spatial Grounding Task)

Response to Weakness 2 (Position Bias and Human Baseline Explanation for Spatial Grounding Task)

We fully understand the reviewer's concern about our unbalanced label distribution. However, due to our metric design, even with such a high proportion of A, the scenario the reviewer worries about---``always selecting A outperforms all models on the main metric''---does not occur. Our primary metric, Macro-F1, effectively resists this positional bias. We elaborate on this through experiments and metric explanations below.

1. Metric Calculation Rules (Addressing the reviewer's question on F1 aggregation)

The spatial grounding task consists of 825 questions (the 2-of-3 agreement subset). We decompose the A–D options of each question into independent binary classification units (aligned with the action-level computation in behavior judgment), where each unit determines whether the model correctly classifies the option as "should be selected" or "should not be selected."

Here are definitions of Macro-F1 and Micro-F1:

Micro-F1: aggregate TP, FP, FN across all four options, then compute F1 \begin{equation} \mbox{Micro-F1} = \frac{2 \sum_{c \in {A,B,C,D}} TP_c}{2 \sum_{c \in {A,B,C,D}} TP_c + \sum_{c \in {A,B,C,D}} FP_c + \sum_{c \in {A,B,C,D}} FN_c} \end{equation}

Macro-F1 (Primary Metric): compute F1 for each option independently, then take the arithmetic mean \begin{equation} \mbox{Macro-F1 (Primary Metric)} = \frac{1}{4} \sum_{c \in {A,B,C,D}} \frac{2 TP_c}{2 TP_c + FP_c + FN_c} \end{equation}

For Macro-F1, the F1 score is first computed independently for each of the four options (A, B, C, D), and the arithmetic mean is then taken across them. This formulation inherently mitigates class imbalance, as it assigns equal weight to each option regardless of category frequency, thereby naturally resisting positional bias.

2. Heuristic Baseline Experiments

We supplemented two heuristic baseline experiments:

  • Majority Class: As shown in Figure 4(a), option A is the most likely to be the proper actions. Therefore, we assume the model selects only A.
  • Random Guessing: Randomly assigning A/B/C/D using a fixed random seed (seed 0); and the average across 20 seeds.

Experimental results are provided below. On the primary metric Macro-F1, "Majority Class" scores only 23.20, far below all models. This is because Macro-F1 requires correct identification of options B, C, and D as well. "Majority Class" completely ignores these options, dragging down the overall score. Therefore, our metric design (with Macro-F1 as the primary metric) ensures that even with imbalanced GT distribution, "Majority Class" strategy cannot surpass models that perform genuine reasoning. In the revised manuscript, we will add the performance of these heuristic baselines.

StrategyMicro-F1Macro-F1
Majority Class62.2023.20
Random Guessing (seed 0)32.5829.16
Random Guessing (20-seed avg)32.0029.37
Qwen3-VL-8B68.6960.52
Nanonets-OCR270.1257.23
InternVL3-8B66.4955.02
DeepSeek-VL269.6749.79

3. Conclusion: Metric Design Effectively Resists Positional Bias

The results clearly show:

  • On the primary metric Macro-F1, Always-A scores only 23.20, far below all models (the lowest is DeepSeek-VL2 at 49.79). This is because Macro-F1 requires correct identification of options B, C, and D as well; Always-A completely ignores these options, dragging down the overall score.
  • On Micro-F1, Always-A scores 62.20, also below all models (the highest is Nanonets-OCR2 at 70.12).

Therefore, our metric design (with Macro-F1 as the primary metric) ensures that even with imbalanced GT distribution, the "always select A" strategy cannot surpass models that perform genuine reasoning on the primary metric. This demonstrates the validity and reliability of the dataset.

AuthorsOpenReview ↗

Response to Weakness 2 (Position Bias and Human Baseline Explanation for Spatial Grounding Task)

Response to Weakness 2 (Position Bias and Human Baseline Explanation for Spatial Grounding Task)

2. On the Calculation and Independence of the Human Baseline

The reviewer expressed confusion about the discrepancy between the pairwise annotator agreement (92.8–99.1%) in Figure 4d and the human PA (81–84%), and inquired about the calculation method of the human baseline.

We clarify as follows:

  • The participants in the human baseline are completely independent of the GT annotation pipeline. The annotators who established the GT labels are strictly screened undergraduates, master's, and PhD students; while the evaluators for the human baseline are 12 independent annotators randomly selected from a larger candidate pool. They have never participated in the GT annotation process and have no prior knowledge of the "correct answers." The PA/CWPA scores of the human baseline are the average performance of these 12 individuals, with no information leakage.

  • Handling of divergent questions: We only retained questions with a majority vote exceeding 3/4 (≥75% agreement) for the final dataset, and filtered out the rest low-consensus questions. All annotation results, especially borderline cases with initial disagreements, underwent final review and arbitration by domain experts to ensure the absolute authority of the GT.

  • Consistency metrics: Previously, we reported pairwise agreement (92.8–99.1%). Now we supplement a more rigorous multi-annotator agreement metric Fleiss' κ = 0.93, which indicates "almost perfect agreement," confirming the reliability of the annotations. The human PA (81–84%) is the average performance of independent evaluators on the retained questions, which is calculated on a different object from the inter-annotator agreement (pairwise agreement), so the numerical difference is reasonable.

We will detail the calculation process of Fleiss' κ and the expert review protocol in the Appendix of the revised version, and open-source all raw annotation logs, model reasoning traces, and evaluation scripts in our GitHub repository to ensure full transparency.

In summary, we hope the above explanations and experiments can eliminate the reviewer's concerns. We thank the reviewer for helping us more clearly demonstrate the robustness of our metric design and the rigor of the annotation process.

Correction of Ethic Statement

We sincerely apologize for the inaccurate description in the Ethical Statement section. We confirm that our experiments did utilize real-world datasets. However, all datasets used in this paper are either open-source or have been properly authorized for research purposes. They can be downloaded or cited via the references and related links provided in the paper. We will correct the Ethical Statement in the revised version to accurately reflect this information and ensure full compliance with ethical guidelines.

AuthorsOpenReview ↗

Response to Reviewer kj75

Dear Reviewer kj75,

The author-reviewer discussion period will soon be closed. We kindly ask that you review our responses, and we welcome further discussion.

Thanks for your time and reconsideration.

Best regards,

Authors of Submission 5674

Reviewer_RpFGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RobotEQ propose a benchmark focusing Social Proactive Intelligence: a robot’s ability to recognize human needs and respond appropriately in a social setting. The dataset contains 1812 synthetic and 270 real-world scenarios covering seven social dimensions. Models are evaluated on two tasks: judging whether a proposed behavior is appropriate and selecting suitable regions or paths in an annotated image. The paper also studies the role of visual information, transfer from synthetic training data to real images, and the use of retrieved social guidelines.

Critical strengths
  1. The problem is worth studying. Robots operating around people need to consider social context. Evaluating these aspects across several application settings is a useful goal.

  2. The data construction involves meaningful human judgment. The authors remove model-proposed labels before annotation and use human judgments to establish the answers. Image review, independent annotation, and expert checks are reasonable steps toward building a reliable dataset.

  3. The paper includes relevant validation experiments. The image/no-image comparison shows that visual information contributes to behavior judgment. Training on synthetic examples and evaluating on real images also gives encouraging evidence that the data could help improve models.

Critical weaknesses
  1. The metrics need clarification: Equation (2) requires CWPA ≥ PA, but Table 1 violates this for Nanonets-OCR-s and Qwen-VL-Plus. The authors need to reconcile the definition with the reported results.

  2. Answer-position bias is not controlled: Option A is correct in 86.5% of cases. If Figure 4a describes the evaluation set and Section 5 uses flattened binary F1, always selecting A would outperform every model in Table 6. Fixed-choice baselines and randomized option labels are needed.

  3. The human reference needs stronger validation: It is unclear whether the human baseline is independent of the annotators who established the labels. Some error examples also allow reasonable disagreement, making raw votes and independent reassessment useful.

Questions for authors
  1. How do you explain the CWPA < PA entries in Table 1? Could you provide the evaluation implementation and reconcile the equations with the reported results?

  2. What are the exact F1 aggregation rules? How do fixed-choice baselines perform, and do the model rankings hold when the A–D assignments are randomized?

  3. How was the human baseline obtained? Were its participants independent of the ground-truth annotation process?

3 comments
AuthorsOpenReview ↗

Response to Question 1,2 and Weakness 1,2

We sincerely thank the reviewer for the valuable feedback. We address each of the raised concerns and provide clarifications below.

Response to Question 1 and Weakness 1

We apologize for the confusion. After careful verification, we confirmed that during manual entry of model metrics, the PA values for two models on the Synthetic subset were incorrectly registered.

  • Nanonets-OCR-s (Synthetic): CWPA = 46.64, PA = 45.62
  • Qwen-VL-Plus (Synthetic): CWPA = 48.31 , PA = 47.28

All other entries in Table 1 remain correct and unchanged. We sincerely apologize for this clerical error. In the revised version, we will open-source all raw inference logs and evaluation scripts in our code repository to ensure full transparency and reproducibility. The PDF will be updated accordingly. We will open-source all inference results and experimental details in our code repository to ensure full transparency and reproducibility. The PDF will be updated accordingly to reflect these corrections.

Response to Weakness 2 / Question 1 (Position Bias and Metric Explanation for Spatial Grounding Task)

We fully understand the reviewer's concern about our unbalanced label distribution. However, due to our metric design, even with such a high proportion of A, the scenario the reviewer worries about---``always selecting A outperforms all models on the main metric''---does not occur. Our primary metric, Macro-F1, effectively resists this positional bias. We elaborate on this through experiments and metric explanations below.

1. Metric Calculation Rules (Addressing the reviewer's question on F1 aggregation)

The spatial grounding task consists of 825 questions (the 2-of-3 agreement subset). We decompose the A–D options of each question into independent binary classification units (aligned with the action-level computation in behavior judgment), where each unit determines whether the model correctly classifies the option as "should be selected" or "should not be selected."

Here are definitions of Macro-F1 and Micro-F1:

Micro-F1: aggregate TP, FP, FN across all four options, then compute F1 \begin{equation} \mbox{Micro-F1} = \frac{2 \sum_{c \in {A,B,C,D}} TP_c}{2 \sum_{c \in {A,B,C,D}} TP_c + \sum_{c \in {A,B,C,D}} FP_c + \sum_{c \in {A,B,C,D}} FN_c} \end{equation}

Macro-F1 (Primary Metric): compute F1 for each option independently, then take the arithmetic mean \begin{equation} \mbox{Macro-F1 (Primary Metric)} = \frac{1}{4} \sum_{c \in {A,B,C,D}} \frac{2 TP_c}{2 TP_c + FP_c + FN_c} \end{equation}

For Macro-F1, the F1 score is first computed independently for each of the four options (A, B, C, D), and the arithmetic mean is then taken across them. This formulation inherently mitigates class imbalance, as it assigns equal weight to each option regardless of category frequency, thereby naturally resisting positional bias.

2. Hierarchical Baseline Experiments

We supplemented two hierarchical baseline experiments:

  • Majority Class: As shown in Figure 4(a), option A is the most likely to be the proper actions. Therefore, we assume the model selects only A.
  • Random Guessing: Randomly assigning A/B/C/D using a fixed random seed (seed 0); and the average across 20 seeds.

Experimental results are provided below. On the primary metric Macro-F1, "Majority Class" scores only 23.20, far below all models. This is because Macro-F1 requires correct identification of options B, C, and D as well. "Majority Class" completely ignores these options, dragging down the overall score. Therefore, our metric design (with Macro-F1 as the primary metric) ensures that even with imbalanced GT distribution, "Majority Class" strategy cannot surpass models that perform genuine reasoning. In the revised manuscript, we will add the performance of these hierarchical baselines.

StrategyMicro-F1Macro-F1
Majority Class62.2023.20
Random Guessing (seed 0)32.5829.16
Random Guessing (20-seed avg)32.0029.37
Qwen3-VL-8B68.6960.52
Nanonets-OCR270.1257.23
InternVL3-8B66.4955.02
DeepSeek-VL269.6749.79

3. Conclusion: Metric Design Effectively Resists Positional Bias

The results clearly show:

  • On the primary metric Macro-F1, Always-A scores only 23.20, far below all models (the lowest is DeepSeek-VL2 at 49.79). This is because Macro-F1 requires correct identification of options B, C, and D as well; Always-A completely ignores these options, dragging down the overall score.
  • On Micro-F1, Always-A scores 62.20, also below all models (the highest is Nanonets-OCR2 at 70.12).

Therefore, our metric design (with Macro-F1 as the primary metric) ensures that even with imbalanced GT distribution, the "always select A" strategy cannot surpass models that perform genuine reasoning on the primary metric. This demonstrates the validity and reliability of the dataset.

AuthorsOpenReview ↗

Response to Weakness 3 / Question 2 (Human Baseline Independence and Validation)

Response to Weakness 3 / Question 2 (Human Baseline Independence and Validation)

We thank the reviewer for raising this important question, which prompts us to clarify the construction process of the human baseline and the quality control mechanisms of our annotation.

1. Complete Independence of Human Baseline Participants

The human baseline scores reported in Table 1 were obtained from a group of participants who are completely independent of the original annotation pipeline. Specifically:

  • The annotators who established the ground-truth (GT) labels are undergraduate, master's, and PhD students with relevant background knowledge, rigorously screened for the task.
  • The evaluators for the human baseline were 12 independent annotators randomly selected from a larger candidate pool. They had never participated in the GT annotation process and had no prior knowledge of the "correct answers."
  • The PA/CWPA scores for the human baseline are the average performance of these 12 individuals, with zero information leakage. The two groups are fully isolated in terms of identity, recruitment channel, and task phase, complying with standard protocols.

2. Handling of Disagreements and Expert Review

For cases with annotator disagreements, we enforced a strict filtering and arbitration mechanism:

  • All questions were independently annotated by at least 4 annotators. We only retained items where the majority vote exceeded 3/4 (i.e., ≥75% agreement), filtering out low-consensus items to ensure the GT itself has high consensus.
  • Furthermore, as stated in the paper, all annotation results—especially those borderline cases with initial disagreements—underwent final review and arbitration by domain experts to guarantee the absolute authority and correctness of the GT.

3. Correction of Ethic Statement

We sincerely apologize for the inaccurate description in the Ethical Statement section. We confirm that our experiments did utilize real-world datasets. However, all datasets used in this paper are either open-source or have been properly authorized for research purposes. They can be downloaded or cited via the references and related links provided in the paper. We will correct the Ethical Statement in the revised version to accurately reflect this information and ensure full compliance with ethical guidelines.

AuthorsOpenReview ↗

Response to Reviewer RpFG

Dear Reviewer RpFG,

The author-reviewer discussion period will soon be closed. We kindly ask that you review our responses, and we welcome further discussion.

Thanks for your time and reconsideration.

Best regards,

Authors of Submission 5674

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewer kVkq

Dear Reviewer kVkq,

The author-reviewer discussion period will soon be closed. We kindly ask that you review our responses, and we welcome further discussion.

Thanks for your time and reconsideration.

Best regards,

Authors of Submission 5674

AuthorsOpenReview ↗

Overall Reply

We sincerely thank all three reviewers for their valuable feedback and insightful questions. Below, we provide a unified response to the common concerns.

(1) Correlation with Prior Work

RobotEQ differs from prior works by targeting human-robot interaction rather than human-human social norm understanding. They have totally different perception and action spaces. For example, for the action space, humans can eat food, but robots generally cannot eat humans' food. This basic difference shifts them into different tasks. If embodied agents come into our daily life, this work can help to define social boundaries for the embodied agents under diverse scenarios.

(2) Position Bias and Metric Explanation for Spatial Grounding Task

We detail our metric calculation process and add additional heuristic baselines, addressing the concern the reviewer worries about---"always selecting A outperforms all models on the main metric". These results verify the rationality of our metric selection and reliability of experimental results.

(3) Human Baseline Explanation

We add additional explanations to human baselines and address the concerns regarding the difference between pairwise annotator agreement and human performance, fully addressing the reviewers' concerns.

(4) Additional details and proofreading

We add additional details regarding our sample filtering process, correct the content in Appendix J, update the correct link in the Appendix, correct spelling mistakes, enlarge figures, and incorporate additional related works.

We would like to further emphasize the core contribution of our work. We introduce the concept of Social Proactive Intelligence (SPI) and introduce seven characteristics to define this new task. Unlike existing benchmarks such as EgoNormia, which focus on human-human interactions, RobotEQ is the first work centered on the socialness of human-robot interactions, covering more scenarios and social dimensions than existing works. Meanwhile, we construct the first benchmark and dataset for this field. Our experiments reveal a key finding: despite rapid advances in general coding, reasoning, and vision capabilities of large models, these improvements do not automatically transfer to SPI, underscoring the need for dedicated modeling, evaluation, and optimization of this field.

Since during the rebuttal process, no reviewer responded to our responses, considering that all concerns have been adequately addressed and most of them lie in the description strictness, which does not harm the core contribution of our work. We hope the reviewers will take care of all our responses and reconsider this work. We will release all raw annotation logs, model reasoning traces, and evaluation scripts in our GitHub repository upon acceptance.

Sincerely,

Authors of Submission 5674

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.