- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RobotEQ 3.0 studies whether user profiles help predict individuals' preferred proactive robot actions. Ten participants, selected from twenty respondents for divergent profiles, each annotate 1,800 image-based scenarios by selecting one or more of five candidate actions. The dataset combines these 18,000 annotations with a 78-item questionnaire covering demographics, robot experience and trust, personality, autonomy, and empathy. The paper evaluates eighteen VLMs with different profile inputs, introduces PAS as an average of set F-score and hit rate, and explores iterative induction of participant-specific textual guidance from 100 labeled support scenarios. Its main claims are that profiles improve preference prediction and that induced experience further improves performance on disjoint scenarios.
Critical strengths
-
The dataset retains meaningful individual variation. All ten participants evaluate the same 1,800 scenarios and candidate sets (Sections 3.3-3.5). This paired design makes differences between participants observable while holding the scenario constant. Allowing multiple equally preferred actions also avoids forcing an artificial single consensus label.
-
The supplementary material supports detailed inspection. Appendix B documents all questionnaire items and scoring; Appendix D defines repeated-annotation reliability and aggregation; and Tables 10-19 provide every participant's scores under individual and full-profile conditions. These details make the benchmark's assumptions and the variability behind Table 1 substantially more inspectable than a leaderboard alone.
Critical weaknesses
-
The central profile-benefit and method comparisons do not isolate an applicable personalization procedure. Section 5.1, Table 1, and Appendix F define “Best Profile” as the maximum over nine individual dimensions plus the full profile, separately for each participant-model pair. No disjoint set for selecting that condition is described. As written, this is an oracle-style upper envelope over profile settings, rather than evidence for a profile-selection policy that works on unseen scenarios. The complete appendix results are useful but do not remove this selection issue. Please identify the selection split, or select the profile on support/validation scenarios and report performance only on untouched scenarios, alongside a fixed full-profile condition. The related attribution problem persists in Table 2: the proposed method receives 100 participant-specific labels and iterative expert feedback, whereas the reported Best Profile comparator does not use those labels to induce preferences (Section 5.5; Appendix I.2). Its gains therefore do not establish the contribution of profile-aware induction over learning preferences from the support examples themselves. A summary/induction baseline using the same support labels without profiles, plus an otherwise identical shuffled-profile control, would make that claim assessable.
-
PAS requires simple coverage controls before its absolute scores and gains can be interpreted as personalized prediction. Equation (1) gives half the score to hitting any preferred action, while Figure 11 allows up to three selected actions out of five. Even a uniformly random three-action prediction has expected PAS of 0.69 when two actions are preferred, and 0.80 when three are preferred: its expected F-scores are 0.48 and 0.60, and its hit rates are 0.90 and 1.00, respectively. These are conditional analytic examples, not estimates of the dataset's aggregate baseline. They show why a relatively high PAS can arise from broad coverage without identifying a user. Figure 3 reports a mean of 2.24 preferred actions, but neither the full cardinality distribution nor cardinality-controlled baselines are reported. Please evaluate permitted random-subset baselines on the actual labels using the same participant weights, report predicted and true set-size distributions, and compare profile and no-profile predictions at matched output cardinalities. Reporting exact-set accuracy or action-level precision and recall alongside PAS would further reveal whether improvements reflect better preference discrimination. I am not asserting that these controls would erase the reported gains; their absence prevents that alternative explanation from being assessed.
Questions for authors
- Which scenarios and labels select each participant-model “Best Profile” in Table 1? Are those scenarios excluded from its reported evaluation, and are the twenty few-shot demonstrations in Appendix H excluded from Figure 4b's evaluation?
- With the same 100 support labels, how much of Table 2's improvement remains when profiles are omitted or shuffled, and when preferences are summarized without the proposed iterative profile-aware procedure? Please also reconcile the stopping rule in Section 5.5 with the fixed five rounds in Appendix I.2.
- Figure 10 asks annotators for the “most appropriate” action and includes “There is little difference between options” as an additional checkbox. Were participants explicitly instructed to express their own preferences, rather than a judgment of general appropriateness? How was that additional response mapped to the five-action preference set and evaluation?