acceptodds
Withdrawn from ICLR 2027

RobotEQ 3.0: Towards Personalized Social Proactive Intelligence in Embodied Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Social Proactive Intelligence (SPI) is an emerging research area, aiming to shift embodied agents from reactive assistance toward proactively understanding human needs and executing socially desirable actions. Prior work has largely centered on the *average user*. However, human expectations are inherently diverse, and prior work overlooks *individual nuances*. To bridge this gap, we introduce **RobotEQ 3.0**, a benchmark for Personalized SPI. **(Dataset)** We first profile participants via a structured questionnaire covering factors that are correlated with human expectations of embodied agents, such as basic demographics and personality traits. Participants then select their preferred actions from a set of candidates. Unlike prior SPI benchmarks that focus on assessing behavioral appropriateness, our task centers on predicting the actions preferred by a specific user, thereby capturing human subjectivity. The resulting dataset establishes explicit links between individual traits and behavioral preferences. **(Solution)** We observe substantial inter-annotator variance, confirming that user preferences over actions are highly individualized. This motivates our exploration of Personalized SPI, in which user traits serve as additional inputs to predict individual preferences. Experimental results show that incorporating user traits can aid personalized prediction. *This work aims to shift the research paradigm from developing agents suited for the average user to designing systems tailored to specific individuals.*

Primary area:
datasets and benchmarks
Keywords:
Social Proactive Intelligence, Personalized SPI, Dataset, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rYUcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RobotEQ 3.0 studies whether user profiles help predict individuals' preferred proactive robot actions. Ten participants, selected from twenty respondents for divergent profiles, each annotate 1,800 image-based scenarios by selecting one or more of five candidate actions. The dataset combines these 18,000 annotations with a 78-item questionnaire covering demographics, robot experience and trust, personality, autonomy, and empathy. The paper evaluates eighteen VLMs with different profile inputs, introduces PAS as an average of set F-score and hit rate, and explores iterative induction of participant-specific textual guidance from 100 labeled support scenarios. Its main claims are that profiles improve preference prediction and that induced experience further improves performance on disjoint scenarios.

Critical strengths
  1. The dataset retains meaningful individual variation. All ten participants evaluate the same 1,800 scenarios and candidate sets (Sections 3.3-3.5). This paired design makes differences between participants observable while holding the scenario constant. Allowing multiple equally preferred actions also avoids forcing an artificial single consensus label.

  2. The supplementary material supports detailed inspection. Appendix B documents all questionnaire items and scoring; Appendix D defines repeated-annotation reliability and aggregation; and Tables 10-19 provide every participant's scores under individual and full-profile conditions. These details make the benchmark's assumptions and the variability behind Table 1 substantially more inspectable than a leaderboard alone.

Critical weaknesses
  1. The central profile-benefit and method comparisons do not isolate an applicable personalization procedure. Section 5.1, Table 1, and Appendix F define “Best Profile” as the maximum over nine individual dimensions plus the full profile, separately for each participant-model pair. No disjoint set for selecting that condition is described. As written, this is an oracle-style upper envelope over profile settings, rather than evidence for a profile-selection policy that works on unseen scenarios. The complete appendix results are useful but do not remove this selection issue. Please identify the selection split, or select the profile on support/validation scenarios and report performance only on untouched scenarios, alongside a fixed full-profile condition. The related attribution problem persists in Table 2: the proposed method receives 100 participant-specific labels and iterative expert feedback, whereas the reported Best Profile comparator does not use those labels to induce preferences (Section 5.5; Appendix I.2). Its gains therefore do not establish the contribution of profile-aware induction over learning preferences from the support examples themselves. A summary/induction baseline using the same support labels without profiles, plus an otherwise identical shuffled-profile control, would make that claim assessable.

  2. PAS requires simple coverage controls before its absolute scores and gains can be interpreted as personalized prediction. Equation (1) gives half the score to hitting any preferred action, while Figure 11 allows up to three selected actions out of five. Even a uniformly random three-action prediction has expected PAS of 0.69 when two actions are preferred, and 0.80 when three are preferred: its expected F-scores are 0.48 and 0.60, and its hit rates are 0.90 and 1.00, respectively. These are conditional analytic examples, not estimates of the dataset's aggregate baseline. They show why a relatively high PAS can arise from broad coverage without identifying a user. Figure 3 reports a mean of 2.24 preferred actions, but neither the full cardinality distribution nor cardinality-controlled baselines are reported. Please evaluate permitted random-subset baselines on the actual labels using the same participant weights, report predicted and true set-size distributions, and compare profile and no-profile predictions at matched output cardinalities. Reporting exact-set accuracy or action-level precision and recall alongside PAS would further reveal whether improvements reflect better preference discrimination. I am not asserting that these controls would erase the reported gains; their absence prevents that alternative explanation from being assessed.

Questions for authors
  1. Which scenarios and labels select each participant-model “Best Profile” in Table 1? Are those scenarios excluded from its reported evaluation, and are the twenty few-shot demonstrations in Appendix H excluded from Figure 4b's evaluation?
  2. With the same 100 support labels, how much of Table 2's improvement remains when profiles are omitted or shuffled, and when preferences are summarized without the proposed iterative profile-aware procedure? Please also reconcile the stopping rule in Section 5.5 with the fixed five rounds in Appendix I.2.
  3. Figure 10 asks annotators for the “most appropriate” action and includes “There is little difference between options” as an additional checkbox. Were participants explicitly instructed to express their own preferences, rather than a judgment of general appropriateness? How was that additional response mapped to the five-action preference set and evaluation?
Reviewer_qEhKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper describes the development of the RobotEQ 3.0 benchmark for personalized social proactive intelligence (SPI). They collected 1,800 scenarios from 10 participants and used 18 state-of-the-art large language models (LLMs) to demonstrate the effectiveness of individual profiles.

Critical strengths
  1. Personalizing agents for users appears to be extremely useful.
  2. Publishing benchmarks and demonstrating their current effectiveness appear to be important for the field.
Critical weaknesses
  1. The paper states that 20 subjects were recruited and 10 were selected based on their profiles. However, sampling 10 subjects randomly multiple times and averaging the results would be more appropriate.
  2. Selecting the “best profile” for each pair of model and subject is problematic. How would one select a model for a completely unknown user?
  3. Furthermore, having completely unknown users fill out the same questionnaire would be far too time-consuming for practical application.
Questions for authors
  1. The paper reports that 20 participants were recruited and 10 were selected for the experiment. What criteria were used to select those 10? Could you repeat the analysis on multiple randomly sampled subsets of 10 participants to show how sensitive the results are to participant selection?

  2. The best profile is selected separately for each model–participant pair. For a new user with no prior interaction data, how would the system select a model and profile? Can you evaluate this selection procedure on participants who were held out from profile selection?

  3. Would every new user need to complete the same questionnaire before using the system? If so, how much time does it take, and how does performance change if users complete a shorter questionnaire or skip it entirely?

Reviewer_o3ZpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is a benchmark paper that introduces RobotEQ 3.0, a benchmark for personalised social proactive intelligence. The key question is to investigate whether embodied agents can predict robot actions preferred by individual users. The main results is that the profile conditioning improves the best-case performance across models, with gains of up to 16+%.

Critical strengths
  • The paper presents a shift toward more personalised social intelligence, which is interesting, as it supposes a modelling of personalised preferences.
  • The benchmark introduced here is pretty decent, with about 1800 scenario and 18k participant scenario annotations.
  • The evaluation covers 18 models, which is pretty comprehensive, and these are from different model families. It also provides a breakdown of the individual PAS etc.
Critical weaknesses
  • I think the key weakness here is the limited participant diversity. There are only about 10 participants and this can limit the level of generalization of the dataset, as well as the contribution.
  • The best profile results may potentially inflate the results as it is selected from multiple profile configurations based on the achieved performance. It begs the question whether the same results are achievable in real world settings where target preferences and the "test set" may be missing.
  • While the ablation and added experiments study impact of scenario images, few-shot prompting etc., it does not robustly tests whether the trait-preference relationships are meaningful in terms of achieving the improved performance. For instance, more analysis on how the semantic content of the profiles relate to the performance improvements would be useful.
  • The PAS evaluation metric may be unecessary complicated, and it rewards predicting at least one preferred action. This can underweight the precision performance of the model, and the real world applicability.
Questions for authors

The major questions are above, and stated in the weaknesses. I think that the underlying idea is interesting, but the execution raises some major concerns, which is why I have am putting reject score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.