- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work presents ProactiveXBench. It evaluates when and how agents intervene in physical situations without explicit requests. The manuscript reports 1,000 tasks spanning 166k seconds, with role/capability conditioning and hidden timing/event annotations. Reported evaluations compare multimodal baselines with a controller-worker Proactive-Engine and include some robot case-study scenarios.
Critical strengths
S1. Withholding task-specific triggers makes role-conditioned intervention discovery central to the evaluation. Separating initiation timing, execution duration, and event completion helps distinguish responsiveness from successful assistance. Also, requiring execution evidence avoids crediting unsupported verbal claims as physical actions.
S2. The reported 1,000 tasks cover three functional domains. 800 video-derived tasks draw on six activity datasets. The documented human-review stages and task-conversion prompts make the intended construction process inspectable and provide a concrete basis for further evaluation.
Critical weaknesses
W1. Table 2 explicitly uses illustrative synthetic data, yet Sec. 4.2 treats it as empirical robot results. GPT-6-Astra's Outcome scores are 0.97/0.96/0.95 in the table but 0.78/0.74/0.64 in the text, reversing the claimed comparison with Proactive-Engine. These discrepancies undermine the central model-performance conclusions.
W2. The Timing Model's backbone and training overlap with evaluation sources, inference cadence/latency accounting, and event-matching implementation are not specified. Tables 2-3 also omit per-model eligible counts despite the text invoking them. Consequently, architectural gains cannot be separated reliably from training, runtime, or sample-selection effects.
W3. Retained tasks must contain meaningful intervention needs, but no dedicated no-intervention evaluation or false-alarm rate is reported. Waiting before a trigger within a positive task does not establish appropriate abstention throughout a genuinely negative scenario, leaving a central proactive-decision failure mode unmeasured.
W4. The central paired comparison has no reported results. Section 4.3 describes an implicit-versus-explicit prompt comparison but supplies no quantitative findings. The intended separation of intervention-need inference from trigger recognition is therefore not empirically demonstrated. Its contribution to the observed failures remains unclear.
Questions for authors
Q1. Which Table 2 entries, if any, come from actual runs? Please reconcile the conflicting Outcome values and rankings with run counts and trace-level evidence.
Q2. What are the Timing Model's backbone and training-source separation, the observation cadence and timestamp convention, and the event-matching procedure? Please also provide per-model eligible/missing counts for the reported metrics.
Q3. Were genuinely no-intervention scenarios evaluated, and with what false-alarm rates? What independent human agreement supports the intervention-need, timestamp, and event-order annotations, particularly for mandatory motions in the reminder task?
Response to W1 and Q1:
Table 2 reports physical-robot runs, but its caption incorrectly describes illustrative synthetic data. Section 4.2 contains a transcription error. The verified GPT-6-Astra Outcome scores are 0.97, 0.96, and 0.95. We will correct the caption and Section 4.2 and align the abstract and related performance and ranking statements.
Response to W2 and Q2:
The Timing Model uses Qwen3.5-4B, fine-tuned on 4,561 video-derived decision sequences. Evaluation tasks come from COIN, Ego-Exo4D, HACS, HoloAssist, YouCook2, QEVD-FIT-COACH, and separately generated interactive scenarios. Training and evaluation instances have no overlap at the video, clip, scenario, prompt, or annotation level. Reference intervention times, target events, and completion annotations remain hidden during inference.
Observations update every 200 ms. The model outputs \texttt{<silence>} or \texttt{<response>}, mapped to \texttt{WAIT}/\texttt{INITIATE} when inactive and \texttt{STOP}/\texttt{CONTINUE} when active. The reference intervention time is when intervention becomes warranted. The trigger time is the first valid model-initiated timestamp. Timing error is their absolute difference. Progress spans initiation to completion of the required verbal or physical response. Runtime latency is measured separately.
Events are matched only when content, associated person or object, and ordering agree with the reference. Physical actions require an execution trajectory or equivalent action record. Textual claims or function-call acknowledgments alone receive no credit.
The physical-scenario evaluation has 20 cases per domain, with one run and trajectory per case. The real-robot evaluation has 200 tasks per model, comprising 63 Guidance, 107 Warning, and 30 Interaction tasks, with one trajectory per task. All instances in Tables 2 and 3 yielded valid Timing, Progress, and Outcome measurements. Eligible counts are 20/20 per physical-scenario model-domain group and 63/63, 107/107, or 30/30 for the corresponding synthetic-evaluation domains. Missing counts are zero, and captions now report these values. Observable inputs, roles, capabilities, timestamps, and scoring rules are fixed across systems. Because heterogeneous baselines retain native interfaces and runtimes, the results are integrated-system comparisons. Claims assigning differences solely to controller-worker architecture have been narrowed.
Response to W3 and Q3:
Annotators reviewed model-assisted proposals for observability, temporal grounding, intervention necessity, role, and capability. The study performed no independent double annotation or agreement analysis for intervention-need labels, reference timestamps, ordered event labels, or mandatory reminder actions. In lost-phone examples, forward motion, left turning, antenna movement, and the phone-and-location reminder are ordered events scored against the trajectory. Outcome follows the matching rubric but does not validate annotation consistency.
The benchmark supports conditional evaluation of initiation timing, post-initiation execution, and event completion. Matched no-intervention scenarios and an independent annotation audit are needed to estimate false-positive rates and annotation agreement.
Response to W4: