acceptodds
Under review as a conference paper at ICLR 2027

ProPAI: Proactive Physical AI for Perceiving, Reasoning, and Acting in the Open World

Abstract

Physical AI is commonly formulated around explicit tasks: a system receives an instruction, perceives its environment, and acts. In everyday settings, however, needs often emerge before anyone formulates a request. We study this setting as Proactive Physical AI, where a system continuously observes its environment, identifies needs under standing responsibilities, and initiates appropriate interventions. We introduce ProAgent, an end-to-end system that separates whether and when to intervene from how to respond. A Proactive Timing Model monitors the evolving scene, while an asynchronous Response Model generates communication or invokes available device capabilities, allowing observation to continue during response generation and execution. An automated data pipeline converts videos into causal supervision for learning intervention timing, and complementary inference optimizations reduce the cost of repeated decisions. Across streaming video evaluations, ProAgent achieves stronger intervention timing and time-aware response quality than the evaluated standalone multimodal models and streaming assistants under both explicit objectives and standing roles. Real-robot evaluations further demonstrate improved task completion across guidance, warning, and interaction scenarios. These results support a practical approach to physical AI that connects continuous observation with timely assistance and physical execution.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Proactive Physical AI, Embodied Intelligence, Multimodal Agents, Streaming Video Understanding, Intervention Timing, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3CeqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies proactive physical assistance, where a system continuously observes its environment and determines whether and when to intervene under standing responsibilities rather than waiting for an explicit task instruction. The proposed ProAgent separates a Timing Model from a Response Model, allowing observation and intervention decisions to continue during response generation or physical execution. The paper introduces an automated pipeline for constructing causal timing supervision and incorporates incremental caching, dynamic token dropping, and prefix-constrained decoding for efficient streaming inference. Evaluation covers public streaming-video benchmarks and adapted proactive-response protocols, a 60-case embodied evaluation, and controlled efficiency experiments.

Critical strengths
  1. Well-motivated and coherent system design. Separating intervention timing from response generation addresses the different computational requirements of continuous monitoring and substantive responses. The framework also treats remaining silent as an explicit decision and connects intervention decisions to available device capabilities and execution feedback. Together with causal timing supervision, these choices provide a reasonable implementation of proactive assistance rather than simply accelerating a conventional question-answering system.
  2. Extensive experiments with informative controls and efficiency analysis. The evaluation covers intervention timing, response quality, physical outcomes, and inference efficiency. The shared-Worker comparison helps isolate the benefit of timing-specific training, improving Explicit-condition F1 from 30.49 to 42.43. The cumulative comparisons also appropriately examine the three complementary efficiency mechanisms: together, they reduce median model inference latency from 346.84 ms to 147.05 ms while retaining 98.65% decision agreement with full recomputation. These measurements exclude image decoding and preprocessing, and the paper correctly distinguishes decision agreement from task accuracy.
Critical weaknesses
  1. The methodological novelty could be established more clearly. The separation of continuous perception, intervention decisions, and asynchronous responses has a close precedent in Dispider [1]. StreamMind [2] similarly uses event-gated invocation to connect continuous observation with selective response generation, while RoboOmni [3] combines contextual intent recognition, interaction, and physical execution without relying solely on explicit commands. These approaches are not identical to ProAgent, but they establish related architectural and task-level precedents. The paper should more explicitly distinguish its original methodological contributions from established design principles, particularly whether the main advance lies in role-conditioned timing supervision, the persistent execution runtime, or their integration.

  2. The overview figure is difficult to read at normal page size. The internal labels in Figure 1 are substantially smaller than the body text, making the timing-control, context-caching, and execution-feedback paths difficult to inspect. Since this figure communicates the central architecture, its readability affects understanding of the asynchronous design. Enlarging the labels, simplifying the layout, or separating the high-level overview from implementation details would improve the presentation. This is a localized and readily fixable issue.

Questions for authors

What is the most important methodological distinction between ProAgent and the related mechanisms in [1–3]? In particular, which capability is enabled by the proposed role-conditioned timing supervision and persistent execution loop beyond existing response-triggering and contextual-assistance frameworks? A concise component-level comparison would help clarify the contribution without requiring a broad new set of experiments.

References

[1] Rui Qian et al. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. CVPR, 2025. arXiv:2501.03218.

[2] Xin Ding et al. StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition. ICCV, 2025. arXiv:2503.06220.

[3] Siyin Wang et al. RoboOmni: Proactive Robot Manipulation in Omni-modal Context. ICLR, 2026. arXiv:2510.23763.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3Ceq

Thank you for the constructive suggestions on contribution clarity and presentation. We address the methodological distinction and Figure 1 below.

Response to W1 and Q1: Methodological contribution and component-level comparison

We thank the reviewer for suggesting a component-level comparison. The table below contrasts the published designs along the four components underlying our contributions.

ComponentDispiderStreamMindRoboOmniProAgent (ours)
Decision objectiveDecide when to respond using streaming video, questions, and interaction history.Trigger responses to query-relevant events.Infer implicit intent from multimodal context and generate dialogue or manipulation actions.Learn whether and when intervention is warranted under standing responsibilities, without naming the target event in advance in the Role setting.
Training supervisionStreaming QA supervision covering response and waiting decisions.Event-aligned dialogue supervision with silence/response labels.Contextual conversations paired with visual observations and manipulation trajectories.Construct causal decision sequences with role-conditioned intervention windows, confirmed non-intervention periods, and unknown-interval masking.
Streaming computationDisentangled perception, decision, and asynchronous response generation.State-space event representations and gated LLM invocation.Unified autoregressive generation of dialogue and action tokens.Combine historical feature/KV reuse, cross-frame visual-token pruning, and decision-prefix early termination to reduce repeated timing-inference costs.
Execution interfaceGenerate responses for streaming video interaction.Generate responses for streaming dialogue.Decode action tokens into robot control commands.Invoke registered physical capabilities through PCI; track ongoing operations and return execution feedback to subsequent decisions.

The resulting distinction is a learned, efficient intervention policy connected to capability-based physical execution. Role-conditioned supervision trains the system to identify when an emerging situation calls for assistance within its responsibilities, including when to remain silent. PCI and the persistent runtime translate a selected intervention into an available physical operation and incorporate its outcome into the continuing interaction.

The data pipeline supplies 224,268 causal sequences from 20,258 videos for this learning objective. With the same Worker, timing-specific training improves Role F1 from 28.72% to 33.45% and reduces unmatched responses from 0.62 to 0.18 per minute (Table 1). The three inference optimizations reduce median model inference latency from 346.84 ms to 147.05 ms, retaining 98.65% decision agreement (Table 4). The 60-case Reachy Mini evaluation further tests the connection between these intervention decisions and physical outcomes.

We will incorporate this comparison into the revision to distinguish our learning, data, inference, and execution contributions from the shared architectural foundations.

Response to W2: Readability of Figure 1

Thank you for pointing this out. We will simplify Figure 1 and enlarge its internal labels, with particular attention to the timing-control, context-caching, and execution-feedback paths. We will move secondary implementation details to the caption or appendix and check readability at the figure's final size in the paper.

Thank you for the detailed response. I appreciate the more explicit comparison across the four aspects of the proposed framework. The clarification of role-conditioned intervention is particularly helpful. Unlike query- or event-triggered approaches, ProAgent aims to learn when intervention is warranted under standing responsibilities, without requiring the target event to be specified in advance. The reported data scale and the matched-Worker results also provide useful evidence for the effectiveness of the proposed timing supervision.

However, I consider the novelty concern only partially addressed.

First, the component-level comparison is helpful for clarifying the intended differences, but it remains largely descriptive. I am still not fully convinced about the methodological advance of the proposed PCI and persistent execution runtime over existing agentic systems, particularly RoboOmni. More importantly, although the Role setting does not explicitly specify the target event at inference time, Appendix D.1 indicates that the original QA reference answers and response windows are retained for evaluation. Therefore, it would be helpful to clarify the extent to which this setting demonstrates open-ended proactive need discovery, rather than generalization from explicitly annotated QA events to role-conditioned prompts.

Second, the 60-case physical evaluation supports the feasibility of connecting intervention decisions to physical capabilities, but it does not independently establish the advantage of the proposed execution interface over existing agentic tool-use frameworks.

Regarding Figure 1, I appreciate the authors' willingness to improve its readability. However, I would prefer to see the actual revised figure in the updated manuscript rather than a promise to modify it in a future version. According to the official ICLR 2027 Author Guidelines, authors are allowed to revise their paper during the public discussion period, with a revised PDF subject to the specified page limit and PDF-diff comparison.

Could the authors therefore upload a revised PDF with Figure 1 simplified and its internal labels enlarged to ensure readability at the actual paper size? Please indicate the corresponding changes so that I can evaluate the revised presentation directly.

Overall, I appreciate the additional clarification and consider the response to have partially addressed my concerns regarding methodological positioning. However, I would like to see the actual PDF revision before considering the presentation concern resolved.

AuthorsOpenReview ↗

Response to Reviewer 3Ceq

Thank you for the thoughtful follow-up. We apologize for the delayed response: the notification email was filtered out of our inbox, and we only just became aware of your comment. We appreciate your recognition of the role-conditioned formulation and timing-supervision results, and would like to clarify the remaining points.

  1. What retaining the QA references means for the Role evaluation.

The key distinction is between specifying an event to the model and using an annotated event to evaluate its autonomous decisions. In the Role setting, the original question is removed. Reference answers and response windows are available exclusively to the evaluator; the model observes the full video stream under general responsibilities and must determine which developments warrant intervention.

For example, the cooking role in Appendix D.1 does not name the avocado or prescribe its preparation steps, and the driving role does not identify the white car or its expected behavior. The model must connect the observed situation to its responsibilities, determine whether assistance is warranted, and select an appropriate time to respond.

Retaining the same references makes this a controlled evaluation: videos, target outcomes, and scoring criteria remain fixed while event-specific instructions are removed. Thus, the references provide a common measurement basis; they do not supply the missing task specification to the model. The resulting evidence concerns identifying annotated intervention opportunities from standing responsibilities on held-out videos. Discovery of additional valid needs outside those annotations is a broader evaluation question.

Under this protocol, the matched-Worker comparison improves Role F1 from 28.72% to 33.45% and reduces unmatched responses from 0.62 to 0.18 per minute. These results show that the proposed supervision improves the relevance and timing of autonomous intervention decisions under role conditioning.

  1. How PCI and the persistent runtime contribute.

Our intended distinction is the connection between a learned intervention policy and a continuing physical interaction loop. PCI exposes registered device capabilities to the Response Model, while the runtime dispatches operations, maintains execution state, and returns feedback to subsequent decisions. This allows the timing policy to initiate assistance through available capabilities within an ongoing observation–decision–execution process.

The component comparison with RoboOmni therefore identifies a concrete implementation distinction: RoboOmni generates action tokens decoded into robot commands, whereas ProAgent invokes registered capabilities through a separate execution interface. The contribution of this interface is its role in integrating the learned timing policy with physical execution; the methodological evidence for improved intervention decisions comes from the supervision and matched comparisons.

The 60-case experiment tests whether those improvements remain useful when responses have consequences in live interactions. With the Worker and execution runtime held fixed, replacing the base Timing Model improves Outcome from 0.514 to 0.910, 0.426 to 0.893, and 0.673 to 0.849 across Guidance, Warning, and Interaction. This establishes a direct connection between timing-specific learning and improved physical outcomes under a common execution layer. Appendix A.1 separates these learning, inference, and execution contributions explicitly.

  1. Figure 1 and the revised PDF.

Thank you for your valuable feedback and suggestions. We have now updated Figure 1 to improve its readability by enlarging the internal labels and adjusting the layout and spacing, particularly around timing control, context caching, and execution feedback. The revised figure now uses the full text width while preserving the original architecture and its connections. We have uploaded the revised PDF to OpenReview for your review.

Thank you again for your careful assessment and constructive feedback. We hope these clarifications and the updated manuscript address your concerns, and we would greatly appreciate your consideration in your final assessment.

AuthorsOpenReview ↗

Response to Reviewer 3Ceq

Thank you again for your thoughtful feedback and careful consideration.

We sincerely apologize for not responding to your latest comments sooner. Unfortunately, due to a recent illness and an email filtering issue, we did not notice your follow-up in time.

We would like to let you know that we have now uploaded a revised PDF to OpenReview, incorporating improvements to Figure 1, particularly its label sizes and readability. We truly appreciate the time and effort you have devoted to reviewing our work. We hope that our clarifications and concrete revisions help address your concerns, and we would be sincerely grateful if you could take these efforts into consideration in your final assessment.

Thank you again for your understanding and constructive feedback.

Reviewer_HJSCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates Proactive Physical AI, where no task is given in advance and a system must decide from a continuous observation stream and a standing role whether, when, and how to intervene. The proposed ProAgent splits this into an asynchronous Timing Model (ProAgent-Time-4B, post-trained from Qwen3.5-4B to emit only <response>/<silence>) and a Response model (a Qwen3.8-27B Worker) that produces text for people or function calls for devices. Timing supervision comes from an automated pipeline that turns 20,258 annotated videos into 224,268 causal decision sequences. Validation covers streaming timing and helpfulness benchmarks (Omni-MMI, a ProactiveVideoQA subset under Explicit and Role conditions, OVO-Bench, StreamingBench), 60 real-robot cases on Reachy Mini, and an efficiency study.

Critical strengths
  • A clean matched comparison isolates the learned Timing Model, and the gain carries over to benchmarks kept separate from training. With the same Worker and input budget, replacing the base Qwen3.5-4B with ProAgent-Time-4B raises Explicit F1 from 30.49 to 42.43 and cuts unmatched responses from 0.70 to 0.04 per minute (Table 1); PAUC nearly doubles (0.3355 → 0.6131) while first-text latency stays flat (1.09 → 1.07 s; Table 2), so the gain comes from better decisions rather than speed. On benchmarks declared disjoint from training, CRR rises from 43.75 to 83.33 and PO from 11.60 to 57.20 (Table B.2). This matters because it shows that "when to speak" can be learned as its own skill.
  • A careful, practical setup for learning timing. Timing is kept separate from response generation, so observation continues during long replies. Labels are built causally (only past frames as input, unknown intervals masked rather than treated as silence), and the pipeline scales to 224k sequences without hand-written task instructions. Efficiency gains (median 346.84 → 147.05 ms) are reported with decision agreement (98.65%, Table 4). This setup is directly reusable.
Critical weaknesses
  • The real-robot evaluation is too lightly controlled and documented to support the "Physical" part of the claim. The title, the abstract ("Real-robot evaluations further demonstrate ProAgent's ability to identify emerging needs and translate them into appropriate physical actions"), and Sec. 4.3 rest on Table 3, but (i) it has no matched baseline: the Base Qwen3.5-4B + Worker pair that makes Tables 1–2 convincing is missing, so the gain cannot be credited to the Timing Model; (ii) the closed-source baselines carry 14.55–19.81 s of remote latency in Table 2, which alone could explain timing errors such as 12.56 s vs 1.28 s; and (iii) App. E gives only categories and metrics, not whether cases were run live or replayed, how baselines were connected to the robot's function calls, who judged event matches, how many trials were run, or which Worker ProAgent used. This matters because all the rigorous evidence is on recorded video with mostly verbal responses, while what separates ProPAI from existing proactive streaming assistants (LiveStar, Dispider) is mainly the physical executor. As it stands, the evidence supports proactive video assistance more than proactive physical AI, in line with the authors' own note that "a timely response in recorded video does not establish physical benefit". A documented protocol with a matched baseline and repeated trials would address this; otherwise, the physical claims should be narrowed.
Questions for authors
  1. [Clarification] Could you give the full protocol for the 60 Reachy Mini cases (the points listed in (iii) of the weakness above)?
  2. [Clarification] App. G.2 lists the benchmarks that the training data is disjoint from, but not ProactiveVideoQA. Are the 200 selected ProactiveVideoQA videos disjoint at the source-video level from the training sources (e.g., Ego-Exo4D, HoloAssist, COIN, YouCook2)? Are the evaluation role prompts drawn from the same "shared role catalog" as the 91,864 role-conditioned training sequences?
  3. [Suggestion] Reporting Base Qwen3.5-4B + the same Worker on the 60 embodied cases in Table 3 is the single experiment most likely to change my assessment.
2 comments
AuthorsOpenReview ↗

Response to Reviewer HJSC (Part 1/2): W1

Thank you for the careful assessment and for identifying the matched embodied comparison as a particularly useful addition. We have completed this comparison and clarify the physical execution protocol, latency measurements, and evaluation scope below.

Response to W1

1. Physical AI formulation and choice of Reachy Mini.

ProAgent addresses proactive task initiation and capability-grounded action planning: identifying emerging needs from continuous observations, determining appropriate interventions, and coordinating available physical capabilities. Once the Timing Model triggers intervention, the Response Model generates a response or structured function calls. The runtime dispatches these calls to device controllers and returns execution feedback to the continuing interaction loop (Sections 2 and 3.1).

We chose Reachy Mini as our initial platform because its SDK and control interfaces support evaluating this observation-decision-execution loop using existing low-level controllers. This allows us to focus on whether the system can recognize a need for assistance and initiate an appropriate intervention without an explicit task instruction. The capability-based interface separates intervention reasoning from device-specific control.

2. Live embodied evaluation and execution protocol.

The 60 cases were conducted live in physical environments, with 20 each for Guidance, Warning, and Interaction. The robot continuously received live observations and autonomously produced speech or device actions; participants continued acting in response to its outputs. Reference intervention times and expected responses were hidden, and experimenters did not manually trigger responses.

All systems had access to the same speech and robot-function interfaces. Evaluation followed the predefined event-completion rubric in Appendix E. For events requiring device actions, success depended on observed action completion and outcomes; generating a function call alone was insufficient.

Each case was repeated three times per model, yielding 180 trials per model. Five actors enacted the scenarios, while three separate evaluators, blinded to model identity, independently scored the trials on site. Majority agreement determined event-level judgments. We will expand Appendix E with these details; Q1 in Part 2 provides the participant and baseline configurations.

3. Matched comparison and latency considerations.

The closed-source measurements include network and service-side processing time. We averaged results over three repetitions per case and did not observe pronounced latency fluctuations. Averaging does not remove systematic serving delays, so these comparisons describe performance under the tested deployments. We use the matched comparison to assess timing-specific training.

Response latency and intervention-time error are different quantities. Table 2 measures the time from the start of the response-producing processing cycle to the first visible text. Time in Table 3 measures the absolute deviation between observed intervention initiation and the reference time. Models with similar computational cost can therefore have different Time scores because they intervene at different moments.

Base + Worker and ProAgent have nearly identical response latencies: 1.09 versus 1.07 s under Explicit and 1.16 versus 1.14 s under Role (Table 2). Meanwhile, timing-specific training improves Explicit F1 from 30.49% to 42.43% and reduces unmatched responses from 0.70 to 0.04 per minute (Table 1).

We extended this matched comparison to all 60 live cases, fixing the Qwen3.8-27B Worker, asynchronous runtime, and deployment configuration. Only the Timing Model checkpoint differs; the Worker receives no additional training. The trained model improves both intervention accuracy and task completion across all three categories. The full results appear under Q3 in Part 2. We will add them to Table 3 and clarify the metric definitions.

4. Scope of our claims.

The current embodied evaluation is an initial validation of a proactive decision and execution framework connecting continuous observation, autonomous intervention decisions, and actionable physical capabilities. It is not comprehensive evidence of general-purpose robotic competence. We will clarify this scope in the abstract and Sec. 4.3; broader cross-embodiment validation remains future work.

AuthorsOpenReview ↗

Response to Reviewer HJSC (Part 2/2): Q1–Q3

Response to Q1: Full protocol for the 60 Reachy Mini cases

The evaluation comprises 60 live cases, with 20 each for Guidance, Warning, and Interaction. Five actors (three computer science master's students and two PhD students) enacted predefined scenario scripts. The robot continuously observed unfolding events, autonomously generated speech or actions, and actors continued according to its actual responses. Participants did not supply reference intervention times or expected answers, and experimenters did not manually trigger responses.

All systems operated robots of the same model, although not necessarily the same physical unit, with the same available speech and robot-function interfaces. Standalone baselines directly generated responses or function calls. ProAgent used its Timing Model with a Qwen3.8-27B Worker, without additional Worker training. Each case was repeated three times per model, giving 180 trials per model.

Three additional computer science master's students, separate from the actors and blinded to model identity, independently scored the trials on site using the predefined rule-based rubric. Majority agreement determined event-level judgments. Time measures mean absolute intervention-time error; Outcome measures completion of required events with correct content and entities, assigning zero when prescribed ordering constraints are violated. We will include these details in Appendix E.

Response to Q2: ProactiveVideoQA separation and shared role catalog

We checked the 200 selected ProactiveVideoQA videos at the source-video level and confirmed that they do not overlap with the training videos. We will explicitly include ProactiveVideoQA in the data-separation statement in Appendix G.2.

Training and evaluation use the same role catalog. We first established a unified catalog and then matched appropriate roles to training and evaluation videos. The "shared role catalog" in Appendix D.1 refers to this catalog. Role prompts specify an identity and general responsibilities without identifying the particular intervention event or its timing. The evaluation therefore tests applying these responsibilities to held-out videos. We will clarify this relationship in Appendix D.1.

Response to Q3: Matched Base + Worker embodied comparison

Thank you for this concrete suggestion. We completed the matched comparison on all 60 live embodied cases, using Base Qwen3.5-4B with the same Qwen3.8-27B Worker, asynchronous runtime, and deployment configuration as ProAgent. Both systems followed the same enactment and scoring protocol, with the same actors and evaluators. Only the Timing Model checkpoint differed; the Worker received no additional training.

CategoryBase + Worker Time (s)ProAgent Time (s)Base + Worker OutcomeProAgent Outcome
Guidance6.321.280.5140.910
Warning4.941.260.4260.893
Interaction7.541.980.6730.849

Time is mean absolute intervention-time error (lower is better); Outcome is event-completion score (higher is better).

The trained Timing Model improves both intervention accuracy and task completion across all three categories. Outcome increases by 0.396, 0.467, and 0.176 for Guidance, Warning, and Interaction, respectively. These gains show that improved intervention decisions translate into more effective embodied assistance under a fixed response model and runtime.

During the trials, we observed that frequent, poorly timed triggers from the base model led to unnecessary or mistimed responses, limiting the Worker's effectiveness. Timing-specific training helped the same Worker intervene more appropriately. The lower Time values reflect closer alignment with the reference intervention times, rather than simply faster response generation. We will incorporate this comparison into Table 3 and document the matched protocol in Appendix E.

Reviewer_d8J7OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper formalizes Proactive Physical AI as a setting where systems continuously observe the environment, identify emerging needs, and decide when intervention/interaction is appropriate. It introduces ProAgent, which separates proactive timing decisions from response generation and physical execution. I'm happy to re-do my reviews post-rebuttal if my critique and questions can be answered. This paper has strong potential.

Critical strengths

The paper presents a useful end-to-end framework for proactive physical assistance, connecting continuous observation, intervention timing, response generation, and physical execution. The distinction between explicit objectives and standing roles, along with the asynchronous architecture, is particularly relevant for real-world proactive agents. The experimental results demonstrate meaningful improvements in intervention timing. Promising E2E work.

Critical weaknesses
  1. Intervention windows are derived from video annotations and teacher-generated labels, sometimes using full-video context. While inference is causal, it remains unclear whether the model learns to identify emerging intervention needs or primarily reproduces timing patterns encoded in its supervision. Stronger generalization experiments would help clarify this distinction.
  2. The 60-case robotic evaluation is promising but limited to controlled scenarios with predefined events. It is unclear how reliably the system handles ambiguous situations, unnecessary interventions, or cases where remaining silent is preferable. These are important considerations for autonomous physical assistance.
  3. The literature survey seems to be missing critical work in Visual Gyms as citations, and additionally Procedural/Proactive work. Non exhaustive examples of the latter: Kundu et. al. (Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance) and Zhu et.al. (ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response)
Questions for authors
  1. How does ProAgent perform on intervention events or standing-role responsibilities that are structurally different from those represented in the automatic training pipeline?
  2. What is the agreement between the automatically generated intervention windows and independent human judgments about when assistance is actually warranted?
  3. Can the authors further separate the contribution of the timing model, asynchronous execution architecture, and response model to end-to-end embodied performance?
2 comments
AuthorsOpenReview ↗

Response to Reviewer d8J7 (Part 1/2): W1/Q1 and Q2

Thank you for recognizing the potential of this framework and for the constructive suggestions. We address generalization, annotation validity, unnecessary interventions, component contributions, and related work below.

Response to W1 and Q1: Generalization beyond training supervision

Thank you for raising this concern. We agree that the key question is whether learned timing decisions remain useful across different intervention contexts. Full-video context is used to construct reference windows offline; the Timing Model receives only causally available observations during training and evaluation.

Our training corpus covers instructional procedures, collaborative manipulation and correction, everyday activities, exercise feedback, and anomaly monitoring (Appendix G.2). Our evaluation benchmarks already include tests in different domains, notably movie scenes in OVO-Bench CRR and card-game and sports-broadcast scenarios in StreamingBench PO. CRR requires waiting until sufficient clues emerge, while PO requires monitoring conditions such as changes in scores and game states. These evaluations test the transfer of learned intervention timing across visual domains and evidence requirements. The trained Timing Model improves CRR from 43.75% to 83.33% and PO from 11.60% to 57.20% over the source backbone (Table B.2).

The live robot evaluation provides complementary evidence beyond recorded-video assessment. All 60 scenarios were newly constructed for evaluation and were absent from the training data. The robot observes events unfolding in a real environment, produces speech or actions, and participants subsequently react to its outputs. Intervention decisions therefore operate within an evolving physical interaction.

To isolate the contribution of timing-specific training, we additionally evaluated Base Qwen3.5-4B with the same Qwen3.8-27B Worker and asynchronous runtime, without additional Worker training. Both systems followed the same live enactment and scoring protocol. The trained Timing Model improves Outcome from 0.514 to 0.910 for Guidance, 0.426 to 0.893 for Warning, and 0.673 to 0.849 for Interaction. These consistent improvements on unseen live scenarios support the transfer of learned timing decisions to effective physical assistance, extending the evidence beyond reproducing temporal labels in recorded-video evaluation.

For standing roles, evaluation uses the shared role catalog described in Appendix D.1 and tests applying those responsibilities to new situations. The evidence therefore concerns transfer across events and interaction contexts; it does not constitute a held-out-role evaluation. We will clarify these distinctions and the training–evaluation source relationships in the revision.

Response to Q2: Human validation of intervention annotations

Thank you for this suggestion. We conducted an independent human audit of 200 sampled cases, assessed by three computer science master’s students. Human–human agreement was 89.4%, and human–automatic annotation agreement was 85.2%. For timing comparisons, timestamps within a two-second tolerance were considered consistent. These results indicate substantial agreement between the automatic annotations and independent human judgments, while also reflecting variability in judging appropriate intervention timing. We will add the audit protocol and agreement statistics to Appendix G.2.

AuthorsOpenReview ↗

Response to Reviewer d8J7 (Part 2/2): W2, Q3 and W3

Response to W2: Cases where the system should remain silent

Thank you for highlighting the importance of appropriate restraint. We added 20 no-intervention cases, each repeated three times per model. For this evaluation, Outcome is 1 only if the system remains silent throughout the trial, producing neither speech nor robot actions, and 0 otherwise. We report the mean across cases and repetitions.

ModelOutcome ↑
GPT-6-Astra0.750
Gemini 30.700
Claude Fable 50.600
Qwen3.5VL 27B0.600
Base Qwen3.5-4B + Worker0.500
ProAgent0.900

Outcome here measures complete-trial silence; it differs from the required-event completion score used for intervention cases.

ProAgent achieves an Outcome of 0.900, compared with 0.500 for the base model with the same Worker, providing direct evidence that timing-specific training improves restraint when assistance is unwarranted. We will include these results in Appendix E. They complement the intervention cases by directly testing unnecessary responses; broader ambiguous situations remain a useful direction for evaluation.

Response to Q3: Contributions of the Timing Model, asynchronous runtime, and Response Model

Thank you for this suggestion. The three components serve complementary purposes: the Timing Model improves intervention timing, asynchronous execution reduces blocking between continuous monitoring and response generation, and the Response Model enables effective communication and physical execution. We distinguish their roles through the following comparisons.

First, the end-to-end embodied comparison in Table 3 evaluates the complete system against standalone models. Compared with Qwen3.5VL 27B, ProAgent reduces mean absolute intervention-time error from 11.21 to 1.28 s for Guidance, 2.35 to 1.26 s for Warning, and 9.03 to 1.98 s for Interaction. Outcome also improves across all three categories:

SystemGuidanceWarningInteraction
Standalone Qwen3.5VL 27B0.7480.7190.618
Base Qwen3.5-4B + shared Worker0.5140.4260.673
ProAgent0.9100.8930.849

Second, the added Base Qwen3.5-4B + shared Worker comparison isolates the contribution of timing-specific training within the deployed system. Both configurations use the same Qwen3.8-27B Worker, asynchronous runtime, and live evaluation protocol; only the Timing Model checkpoint differs. The Worker receives no additional training. The consistent Outcome improvements therefore demonstrate that better intervention decisions help the same response model deliver more effective assistance. During the trials, we observed that frequent, poorly timed triggers from the base model led to unnecessary or mistimed responses, limiting the Worker’s effectiveness.

The asynchronous architecture serves a complementary operational role: it decouples continuous monitoring from response generation, allowing observation and timing assessment to continue while the Worker is occupied (Section 3.1; Appendix D.2). It is held fixed in the matched comparison. Thus, the standalone comparison establishes the complete system’s practical benefit, while the matched comparison attributes the additional gains to timing-specific training; neither comparison independently quantifies the effect of asynchronous scheduling. We will make this contribution breakdown explicit in the revision.

Response to W3: Related work

Thank you for these helpful suggestions. We interpret ‘Visual Gyms’ as interactive environments for training and evaluating visual agents. We have expanded Appendix A to cover this direction through VisGym, alongside the recommended procedural/proactive works, Plan, Watch, Recover and ProAct (Zhu et al.).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.