acceptodds
Under review as a conference paper at ICLR 2027

ProactiveXBench: Benchmarking When and How to Intervene in the Physical World

Abstract

Physical AI increasingly requires intelligent systems to operate autonomously in evolving real-world environments, where proactive assistance is a fundamental capability. However, existing benchmarks predominantly adopt reactive settings with explicit requests or goals, leaving proactive physical intelligence without systematic evaluation. A benchmark for this capability should assess whether a system can infer implicit intervention needs from ongoing observations, determine when intervention is appropriate, and sustain coherent communication and physical actions until the intended objective is achieved. To this end, we introduce ProactiveXBench, a benchmark for proactive physical intelligence comprising 1,000 real-world physical tasks with 166K fine-grained second-level annotations across diverse scenarios involving proactive communication and physical actions. ProactiveXBench evaluates the full intervention trajectory through three complementary dimensions: Timing, which measures alignment with the reference intervention time; Progress, which characterizes post-initiation execution duration; and Outcome, which measures event-level completion under entity-matching and ordering constraints. Evaluations of leading AI systems including GPT-6-Astra reveal substantial limitations in proactive physical intelligence: strong response or completion performance does not necessarily translate into accurate recognition of when intervention is needed, and timely initiation does not guarantee successful completion. These findings expose complementary challenges in intervention discovery and sustained execution, and establish ProactiveXBench as a unified framework for studying and advancing proactive physical intelligence.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Proactive Physical Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HHCvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work presents ProactiveXBench. It evaluates when and how agents intervene in physical situations without explicit requests. The manuscript reports 1,000 tasks spanning 166k seconds, with role/capability conditioning and hidden timing/event annotations. Reported evaluations compare multimodal baselines with a controller-worker Proactive-Engine and include some robot case-study scenarios.

Critical strengths

S1. Withholding task-specific triggers makes role-conditioned intervention discovery central to the evaluation. Separating initiation timing, execution duration, and event completion helps distinguish responsiveness from successful assistance. Also, requiring execution evidence avoids crediting unsupported verbal claims as physical actions.

S2. The reported 1,000 tasks cover three functional domains. 800 video-derived tasks draw on six activity datasets. The documented human-review stages and task-conversion prompts make the intended construction process inspectable and provide a concrete basis for further evaluation.

Critical weaknesses

W1. Table 2 explicitly uses illustrative synthetic data, yet Sec. 4.2 treats it as empirical robot results. GPT-6-Astra's Outcome scores are 0.97/0.96/0.95 in the table but 0.78/0.74/0.64 in the text, reversing the claimed comparison with Proactive-Engine. These discrepancies undermine the central model-performance conclusions.

W2. The Timing Model's backbone and training overlap with evaluation sources, inference cadence/latency accounting, and event-matching implementation are not specified. Tables 2-3 also omit per-model eligible counts despite the text invoking them. Consequently, architectural gains cannot be separated reliably from training, runtime, or sample-selection effects.

W3. Retained tasks must contain meaningful intervention needs, but no dedicated no-intervention evaluation or false-alarm rate is reported. Waiting before a trigger within a positive task does not establish appropriate abstention throughout a genuinely negative scenario, leaving a central proactive-decision failure mode unmeasured.

W4. The central paired comparison has no reported results. Section 4.3 describes an implicit-versus-explicit prompt comparison but supplies no quantitative findings. The intended separation of intervention-need inference from trigger recognition is therefore not empirically demonstrated. Its contribution to the observed failures remains unclear.

Questions for authors

Q1. Which Table 2 entries, if any, come from actual runs? Please reconcile the conflicting Outcome values and rankings with run counts and trace-level evidence.

Q2. What are the Timing Model's backbone and training-source separation, the observation cadence and timestamp convention, and the event-matching procedure? Please also provide per-model eligible/missing counts for the reported metrics.

Q3. Were genuinely no-intervention scenarios evaluated, and with what false-alarm rates? What independent human agreement supports the intervention-need, timestamp, and event-order annotations, particularly for mandatory motions in the reminder task?

1 comment

Response to W1 and Q1:

Table 2 reports physical-robot runs, but its caption incorrectly describes illustrative synthetic data. Section 4.2 contains a transcription error. The verified GPT-6-Astra Outcome scores are 0.97, 0.96, and 0.95. We will correct the caption and Section 4.2 and align the abstract and related performance and ranking statements.

Response to W2 and Q2:

  1. We revised the manuscript to clarify the Timing Model, data separation, inference, event matching, run counts, trajectories, and eligible samples.

The Timing Model uses Qwen3.5-4B, fine-tuned on 4,561 video-derived decision sequences. Evaluation tasks come from COIN, Ego-Exo4D, HACS, HoloAssist, YouCook2, QEVD-FIT-COACH, and separately generated interactive scenarios. Training and evaluation instances have no overlap at the video, clip, scenario, prompt, or annotation level. Reference intervention times, target events, and completion annotations remain hidden during inference.

Observations update every 200 ms. The model outputs \texttt{<silence>} or \texttt{<response>}, mapped to \texttt{WAIT}/\texttt{INITIATE} when inactive and \texttt{STOP}/\texttt{CONTINUE} when active. The reference intervention time is when intervention becomes warranted. The trigger time is the first valid model-initiated timestamp. Timing error is their absolute difference. Progress spans initiation to completion of the required verbal or physical response. Runtime latency is measured separately.

Events are matched only when content, associated person or object, and ordering agree with the reference. Physical actions require an execution trajectory or equivalent action record. Textual claims or function-call acknowledgments alone receive no credit.

The physical-scenario evaluation has 20 cases per domain, with one run and trajectory per case. The real-robot evaluation has 200 tasks per model, comprising 63 Guidance, 107 Warning, and 30 Interaction tasks, with one trajectory per task. All instances in Tables 2 and 3 yielded valid Timing, Progress, and Outcome measurements. Eligible counts are 20/20 per physical-scenario model-domain group and 63/63, 107/107, or 30/30 for the corresponding synthetic-evaluation domains. Missing counts are zero, and captions now report these values. Observable inputs, roles, capabilities, timestamps, and scoring rules are fixed across systems. Because heterogeneous baselines retain native interfaces and runtimes, the results are integrated-system comparisons. Claims assigning differences solely to controller-worker architecture have been narrowed.

Response to W3 and Q3:

  1. The benchmark contains intervention-positive episodes conditional on a verified intervention need. Ordinary no-need candidates were excluded, so no complete intervention-negative set is available. \texttt{WAIT} periods precede intervention within tasks that eventually require it and are not true negatives. A false-positive rate is therefore neither reported nor estimable, and its absence does not imply a zero rate.

Annotators reviewed model-assisted proposals for observability, temporal grounding, intervention necessity, role, and capability. The study performed no independent double annotation or agreement analysis for intervention-need labels, reference timestamps, ordered event labels, or mandatory reminder actions. In lost-phone examples, forward motion, left turning, antenna movement, and the phone-and-location reminder are ordered events scored against the trajectory. Outcome follows the matching rubric but does not validate annotation consistency.

The benchmark supports conditional evaluation of initiation timing, post-initiation execution, and event completion. Matched no-intervention scenarios and an independent annotation audit are needed to estimate false-positive rates and annotation agreement.

Response to W4:

  1. Section 4.3 described implicit-versus-explicit prompting without paired results, overstating separation of intervention-need inference from recognition of a revealed intervention condition. We removed this claim. The revision presents identity-driven prompting as a construction choice and notes that no quantitative paired ablation is included. Timing, Progress, and Outcome measure the combined proactive trajectory, so they cannot isolate implicit need inference in failures.
Reviewer_jXZUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces ProactiveXBench, a benchmark for proactive physical assistance. An agent receives a role, its responsibilities, and its capability constraints, but no explicit trigger, and must decide from streaming observations whether and when to intervene and then carry the intervention through to completion. Tasks are derived mainly from existing human-activity video datasets plus interactive scenarios, built with a model-assisted pipeline and human review, and grouped into Guidance, Warning, and Interaction. Three metrics are proposed: Timing (deviation from a reference intervention time), Progress (post-initiation execution duration), and Outcome (completion of entity-matched, correctly ordered events). The paper reports a real-robot results table (Table 2), a physical-scenario evaluation of three systems (Table 3), and several single-recording real-robot case studies. The task formulation is a clear conceptual contribution, but the only results table that includes the frontier models is explicitly labeled as synthetic, and these models appear otherwise only in single-recording case studies, so the headline empirical findings are not supported by the presented evidence.

Critical strengths

A task formulation and Outcome metric that evaluate the full intervention, not just response relevance. The agent receives only its role, responsibilities, and capabilities, while the trigger, reference time, and expected events are withheld (L49–51, L135–136), so it must decide by itself whether and when to intervene. Completion is scored on observable, entity-matched, correctly ordered events, and a physical action counts only if it is verified in the execution trajectory rather than merely stated (L70–71, L156–161, L320–321). This separates recognizing the need, initiating at the right time, and actually completing the assistance, which is the paper's main conceptual contribution.

Critical weaknesses
  1. The main multi-model results are labeled as synthetic, and the text does not match them. Table 2, the only table that includes the frontier models, is captioned as "Illustrative … results … using synthetic data", with Outcome values described as "illustrative synthetic scores" (L344–346). Yet the text discusses it as a real-robot evaluation (L363–369), quotes Outcome values for GPT-6-Astra that differ from those in the table (L365–366), and states that Proactive-Engine achieves higher Outcome scores in all three domains (L373–374), whereas in the table GPT-6-Astra has the higher Outcome in every domain. The only other quantitative evaluation (Table 3) covers only Proactive-Engine, Omni-3B, and Qwen3.5-4B; GPT-6-Astra, Gemini 3, Claude Fable 5, and Qwen3.5-27B (L307–310) do not appear in it. The abstract's central claim about the limitations of leading AI systems therefore rests on synthetic numbers and single-recording case studies. For a benchmark paper, this issue is decisive.
  2. The benchmark's measurement validity is not established. The Timing metric depends on a single annotated reference time whose placement requires annotator judgment (L223–224), yet no inter-annotator agreement is reported (Section 3.3, L227–230; Appendix A.3, L698–700), and the paper does not explain how Outcome matching (Eq. 5, L262–264) is decided at scale or how physical actions are credited for the recorded-video tasks (L72–73; Appendix A.2, L635–638). Without these, differences between systems cannot be attributed to capability rather than annotation or scoring noise.
Questions for authors
  1. (Weakness 1, clarification) Were the numbers in Table 2 obtained from actual model runs? If so, why is the table captioned as synthetic, and which values are correct, the table or the text (L365–366, L373–374)? Please report real results for all evaluated systems on the full benchmark.
  2. (Weakness 2, validation) What is the inter-annotator agreement on reference intervention times and evaluation events, how is Outcome matching performed at scale, and how are physical actions credited for recorded-video tasks? Real, complete results for Q1 together with the reliability analysis in Q2 are the minimum needed to reassess this submission.
2 comments

Response W1, Q1 and D1:

1.Thank you to the reviewers for their valuable feedback. We agree that Table 3 does not encompass all the models discussed in the paper. While Table 2 presents experimental results for frontier models in real-world robotic environments, the evaluation of physical scenarios in Table 3 is limited to three specific models: Proactive-Engine, Omni-3B, and Qwen3.5-4B. We acknowledge that the scope of the evaluation was not clearly defined previously. We have revised the table captions and Section 4.2 to explicitly state that Table 3 covers only these three models. Additionally, we have included a "model-protocol" coverage table that details the applicability of each model and provides statistics on missing data, while removing any statements implying that all frontier models were covered. In the revised abstract and main text, all discussions concerning frontier models are now based exclusively on the corrected real-world robotic experimental results presented in Table 2. Upon verification, the "Outcome" scores for GPT-6-Astra are 0.97, 0.96, and 0.95, respectively. The previously reported values of 0.78, 0.74, and 0.64 were recording errors. Furthermore, we have corrected our description of the rankings: Proactive-Engine performs better in terms of "Timing" errors, whereas GPT-6-Astra achieves higher "Outcome" scores across all three domains.

Table 3 provides a quantitative comparison of only Proactive-Engine, Omni-3B, and Qwen3.5-4B, and should not be interpreted as an exhaustive quantitative ranking of all systems listed in Lines 307–310. Our central conclusion does not rely on Table 3 as a complete cross-model evaluation. Instead, it is based on complementary evidence from the controlled quantitative comparison, synthetic-data experiments, and single-record evaluations. Table 3 quantifies performance for the directly comparable systems, whereas the additional evaluations examine the observed limitations under other task and data conditions. Accordingly, the statement in the Abstract is intended to describe limitations observed under the benchmark settings examined in this study. It does not claim that Table 3 quantitatively evaluates GPT-6-Astra, Gemini 3, Claude Fable 5, or Qwen3.5-27B, nor that the results establish a universal ranking of all leading systems. We acknowledge that broader model coverage would further strengthen the benchmark, but we do not agree that the restricted scope of Table 3 invalidates the central conclusion. The conclusion is bounded by the evaluated systems, datasets, and protocols.

Response W2 and Q2:

2.Thank you for raising this concern. We have clarified the annotation and outcome-scoring procedures in Sections 3.3, 3.4 and Appendix A.3. The reference intervention time is determined by human annotators based on the assigned role and capability requirements, marking the earliest observable moment at which intervention is warranted. Qwen3.8-27B is used as an auxiliary consistency checker for temporal localization, event recognition, entity association, and event ordering, while human annotators make the final decisions.

Outcome matching follows a unified event-level protocol. For each execution trajectory, evaluators check event content, associated persons or objects, observable evidence, and ordering constraints, with each event counted at most once. For physical-action events, annotators define key observable actions as sufficient completion criteria. An event receives credit only when these actions appear in the evaluated execution record and follow the expected human-motion sequence. Actions visible only in raw videos, verbal claims, or text-only intentions receive no execution credit.

Reponse to D2: 3. We thank the reviewer for identifying this omission. We agree that the original manuscript did not make the ethical basis of the human-participant recordings sufficiently clear. The real-robot case studies were staged demonstrations with 2 student volunteers. The completed protocol was Harbin Institution of Technology. Participants provided verbal informed consent covering participation, audio-video recording, and publication of de-identified material. The fall-assistance case was fully simulated, with no actual fall or robot-participant physical contact. We added an Ethics Statement before the references and clarified participant recruitment, consent, privacy, and safety procedures in Section 4.1 and Appendix B.3.1. We also revised the fall-assistance caption to identify the scenario as simulated.

Thanks for the detailed rebuttal. The replies partially solved my concern. However, the full paper need further revision and improvements. I will raise the score but still hope to see a revised version.

Reviewer_97dhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors introduce ProactiveXBench, designed to evaluate when and how agents should intervene in physical environments without explicit requests. The benchmark comprises 1k tasks with annotated observations across three domains (Guidance, Warning, Interaction), combining recorded videos and interactive scenarios. Authors also propose multiple evaluation dimensions, i.e. timing, progress, and outcome, along with a controller–worker proactive system and case studies on robotic systems.

Critical strengths
  • Gap identification and corresponding contribution. The paper aims to jointly evaluate implicit intervention discovery & sustained execution -- a relatively unique blend of capabilities beyond proactive response timing alone.

  • Role-conditioned task formulation. Providing responsibilities & capability constraints without explicit triggers makes the setting more relevant to autonomous assistance.

  • Complementary evaluation dimensions. Separating initiation timing, execution duration, and event completion allows researchers to see where the gap is clearly -- where many literature in proactive focus primarily focus on timely response as successful assistance.

  • Broad task coverage with physical grounding. The benchmark draws from six activity datasets, while the specific case studies on the robotic systems illustrate why relevant language alone does not establish action completion.

Critical weaknesses
  • The subjective nature of the task doesn't seem to be taken into account well in the evaluation design -- due to the nature of proactive assistance, the same visible user context may need a different outcome depending on personalization signals, whereas the paper assumes ground-truth interruptions. Measuring precision and recall (outcome) on this will need more careful design.

*Some of the evaluation design choices also seem rather arbitary -- e.g. a single reference timestamp with symmetric early/late penalties, zero outcome score for any ordering violation.

  • Evaluation results are all over -- Table 2 describes GPT 6 Astra outcome scores of 0.97 / 0.96 / 0.95, whereas the text in SEction 4.2 notes .78, 0.74, and 0.64. The discrepancy in descriptions (e.g. real robot scenarios vs. synthetic data) seem misleading and need clarifications.

  • Missing some critical relevant literature on proactive assistance, e.g. below

[1] Zhang et al., "Proactive assistant dialogue generation from streaming egocentric videos", 2025. [2] Bhattacharyya et al., "Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?", 2025. [3] Kundu et al., "Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance", 2026.

Questions for authors

See above

1 comment

Response W1:

  1. We thank the reviewer for raising this important concern. We agree that proactive assistance is context-dependent and that the same visible situation may support different interventions for users with different preferences. In ProactiveXBench, the reference intervention times and ordered response events are obtained through human annotation and quality-control procedures. They are conditioned on each task's role, responsibilities, capability constraints, observable context, and required response events. Thus, the annotations are intended as operational reference targets for a controlled benchmark, rather than as universally correct interruptions for every user or personalization setting. We will clarify this distinction throughout the revised manuscript and avoid describing these annotations as absolute ground truth. We agree that these design choices need clearer justification. We use a single reference timestamp to provide a consistent point of comparison across systems. Annotators mark this time as the point at which the task-specific evidence is sufficient to warrant intervention, rather than as a universally correct moment for every user. The symmetric absolute timing error provides a simple and reproducible baseline, while our annotation and agreement analyses characterize the uncertainty in these references. Besides, We enforce event ordering because many interventions contain necessary prerequisites. For example, in the fall-assistance task, the robot should first approach and stop near the person before checking their condition or waiting for a response. If it asks a question without first approaching, the response may still be relevant, but the required intervention sequence has not been completed. We therefore assign a zero strict Outcome score when a prerequisite order is violated. This is a conservative completion criterion and does not mean that a partially completed response has no practical value. In the revision, we will explain this rationale, report the annotation, agreement, and adjudication procedures, and add event-level coverage to show partial completion alongside the strict Outcome score.

Response W2:

2.We thank the reviewers for catching this inconsistency. The entries in Table 2 were obtained from actual model runs. The caption describing them as 'illustrative synthetic scores is incorrect', and the conflicting values in Section 4.2 resulted from an erroneous transcription. We have verified the results against the original run logs. The correct "Outcome" values for GPT-6-Astra on the physical robot are 0.97, 0.96, and 0.95. We will correct the caption, Section 4.2, the abstract, and all statements about model ranking.

Response W3:

3.We thank the reviewer for the reminder. We have added citations for these three references in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.