- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies proactive physical assistance, where a system continuously observes its environment and determines whether and when to intervene under standing responsibilities rather than waiting for an explicit task instruction. The proposed ProAgent separates a Timing Model from a Response Model, allowing observation and intervention decisions to continue during response generation or physical execution. The paper introduces an automated pipeline for constructing causal timing supervision and incorporates incremental caching, dynamic token dropping, and prefix-constrained decoding for efficient streaming inference. Evaluation covers public streaming-video benchmarks and adapted proactive-response protocols, a 60-case embodied evaluation, and controlled efficiency experiments.
Critical strengths
- Well-motivated and coherent system design. Separating intervention timing from response generation addresses the different computational requirements of continuous monitoring and substantive responses. The framework also treats remaining silent as an explicit decision and connects intervention decisions to available device capabilities and execution feedback. Together with causal timing supervision, these choices provide a reasonable implementation of proactive assistance rather than simply accelerating a conventional question-answering system.
- Extensive experiments with informative controls and efficiency analysis. The evaluation covers intervention timing, response quality, physical outcomes, and inference efficiency. The shared-Worker comparison helps isolate the benefit of timing-specific training, improving Explicit-condition F1 from 30.49 to 42.43. The cumulative comparisons also appropriately examine the three complementary efficiency mechanisms: together, they reduce median model inference latency from 346.84 ms to 147.05 ms while retaining 98.65% decision agreement with full recomputation. These measurements exclude image decoding and preprocessing, and the paper correctly distinguishes decision agreement from task accuracy.
Critical weaknesses
-
The methodological novelty could be established more clearly. The separation of continuous perception, intervention decisions, and asynchronous responses has a close precedent in Dispider [1]. StreamMind [2] similarly uses event-gated invocation to connect continuous observation with selective response generation, while RoboOmni [3] combines contextual intent recognition, interaction, and physical execution without relying solely on explicit commands. These approaches are not identical to ProAgent, but they establish related architectural and task-level precedents. The paper should more explicitly distinguish its original methodological contributions from established design principles, particularly whether the main advance lies in role-conditioned timing supervision, the persistent execution runtime, or their integration.
-
The overview figure is difficult to read at normal page size. The internal labels in Figure 1 are substantially smaller than the body text, making the timing-control, context-caching, and execution-feedback paths difficult to inspect. Since this figure communicates the central architecture, its readability affects understanding of the asynchronous design. Enlarging the labels, simplifying the layout, or separating the high-level overview from implementation details would improve the presentation. This is a localized and readily fixable issue.
Questions for authors
What is the most important methodological distinction between ProAgent and the related mechanisms in [1–3]? In particular, which capability is enabled by the proposed role-conditioned timing supervision and persistent execution loop beyond existing response-triggering and contextual-assistance frameworks? A concise component-level comparison would help clarify the contribution without requiring a broad new set of experiments.
References
[1] Rui Qian et al. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. CVPR, 2025. arXiv:2501.03218.
[2] Xin Ding et al. StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition. ICCV, 2025. arXiv:2503.06220.
[3] Siyin Wang et al. RoboOmni: Proactive Robot Manipulation in Omni-modal Context. ICLR, 2026. arXiv:2510.23763.
4 comments
Thank you for the detailed response. I appreciate the more explicit comparison across the four aspects of the proposed framework. The clarification of role-conditioned intervention is particularly helpful. Unlike query- or event-triggered approaches, ProAgent aims to learn when intervention is warranted under standing responsibilities, without requiring the target event to be specified in advance. The reported data scale and the matched-Worker results also provide useful evidence for the effectiveness of the proposed timing supervision.
However, I consider the novelty concern only partially addressed.
First, the component-level comparison is helpful for clarifying the intended differences, but it remains largely descriptive. I am still not fully convinced about the methodological advance of the proposed PCI and persistent execution runtime over existing agentic systems, particularly RoboOmni. More importantly, although the Role setting does not explicitly specify the target event at inference time, Appendix D.1 indicates that the original QA reference answers and response windows are retained for evaluation. Therefore, it would be helpful to clarify the extent to which this setting demonstrates open-ended proactive need discovery, rather than generalization from explicitly annotated QA events to role-conditioned prompts.
Second, the 60-case physical evaluation supports the feasibility of connecting intervention decisions to physical capabilities, but it does not independently establish the advantage of the proposed execution interface over existing agentic tool-use frameworks.
Regarding Figure 1, I appreciate the authors' willingness to improve its readability. However, I would prefer to see the actual revised figure in the updated manuscript rather than a promise to modify it in a future version. According to the official ICLR 2027 Author Guidelines, authors are allowed to revise their paper during the public discussion period, with a revised PDF subject to the specified page limit and PDF-diff comparison.
Could the authors therefore upload a revised PDF with Figure 1 simplified and its internal labels enlarged to ensure readability at the actual paper size? Please indicate the corresponding changes so that I can evaluate the revised presentation directly.
Overall, I appreciate the additional clarification and consider the response to have partially addressed my concerns regarding methodological positioning. However, I would like to see the actual PDF revision before considering the presentation concern resolved.
Response to Reviewer 3Ceq
Thank you for the thoughtful follow-up. We apologize for the delayed response: the notification email was filtered out of our inbox, and we only just became aware of your comment. We appreciate your recognition of the role-conditioned formulation and timing-supervision results, and would like to clarify the remaining points.
- What retaining the QA references means for the Role evaluation.
The key distinction is between specifying an event to the model and using an annotated event to evaluate its autonomous decisions. In the Role setting, the original question is removed. Reference answers and response windows are available exclusively to the evaluator; the model observes the full video stream under general responsibilities and must determine which developments warrant intervention.
For example, the cooking role in Appendix D.1 does not name the avocado or prescribe its preparation steps, and the driving role does not identify the white car or its expected behavior. The model must connect the observed situation to its responsibilities, determine whether assistance is warranted, and select an appropriate time to respond.
Retaining the same references makes this a controlled evaluation: videos, target outcomes, and scoring criteria remain fixed while event-specific instructions are removed. Thus, the references provide a common measurement basis; they do not supply the missing task specification to the model. The resulting evidence concerns identifying annotated intervention opportunities from standing responsibilities on held-out videos. Discovery of additional valid needs outside those annotations is a broader evaluation question.
Under this protocol, the matched-Worker comparison improves Role F1 from 28.72% to 33.45% and reduces unmatched responses from 0.62 to 0.18 per minute. These results show that the proposed supervision improves the relevance and timing of autonomous intervention decisions under role conditioning.
- How PCI and the persistent runtime contribute.
Our intended distinction is the connection between a learned intervention policy and a continuing physical interaction loop. PCI exposes registered device capabilities to the Response Model, while the runtime dispatches operations, maintains execution state, and returns feedback to subsequent decisions. This allows the timing policy to initiate assistance through available capabilities within an ongoing observation–decision–execution process.
The component comparison with RoboOmni therefore identifies a concrete implementation distinction: RoboOmni generates action tokens decoded into robot commands, whereas ProAgent invokes registered capabilities through a separate execution interface. The contribution of this interface is its role in integrating the learned timing policy with physical execution; the methodological evidence for improved intervention decisions comes from the supervision and matched comparisons.
The 60-case experiment tests whether those improvements remain useful when responses have consequences in live interactions. With the Worker and execution runtime held fixed, replacing the base Timing Model improves Outcome from 0.514 to 0.910, 0.426 to 0.893, and 0.673 to 0.849 across Guidance, Warning, and Interaction. This establishes a direct connection between timing-specific learning and improved physical outcomes under a common execution layer. Appendix A.1 separates these learning, inference, and execution contributions explicitly.
- Figure 1 and the revised PDF.
Thank you for your valuable feedback and suggestions. We have now updated Figure 1 to improve its readability by enlarging the internal labels and adjusting the layout and spacing, particularly around timing control, context caching, and execution feedback. The revised figure now uses the full text width while preserving the original architecture and its connections. We have uploaded the revised PDF to OpenReview for your review.
Thank you again for your careful assessment and constructive feedback. We hope these clarifications and the updated manuscript address your concerns, and we would greatly appreciate your consideration in your final assessment.
Response to Reviewer 3Ceq
Thank you again for your thoughtful feedback and careful consideration.
We sincerely apologize for not responding to your latest comments sooner. Unfortunately, due to a recent illness and an email filtering issue, we did not notice your follow-up in time.
We would like to let you know that we have now uploaded a revised PDF to OpenReview, incorporating improvements to Figure 1, particularly its label sizes and readability. We truly appreciate the time and effort you have devoted to reviewing our work. We hope that our clarifications and concrete revisions help address your concerns, and we would be sincerely grateful if you could take these efforts into consideration in your final assessment.
Thank you again for your understanding and constructive feedback.
Response to Reviewer 3Ceq
Thank you for the constructive suggestions on contribution clarity and presentation. We address the methodological distinction and Figure 1 below.
Response to W1 and Q1: Methodological contribution and component-level comparison
We thank the reviewer for suggesting a component-level comparison. The table below contrasts the published designs along the four components underlying our contributions.
The resulting distinction is a learned, efficient intervention policy connected to capability-based physical execution. Role-conditioned supervision trains the system to identify when an emerging situation calls for assistance within its responsibilities, including when to remain silent. PCI and the persistent runtime translate a selected intervention into an available physical operation and incorporate its outcome into the continuing interaction.
The data pipeline supplies 224,268 causal sequences from 20,258 videos for this learning objective. With the same Worker, timing-specific training improves Role F1 from 28.72% to 33.45% and reduces unmatched responses from 0.62 to 0.18 per minute (Table 1). The three inference optimizations reduce median model inference latency from 346.84 ms to 147.05 ms, retaining 98.65% decision agreement (Table 4). The 60-case Reachy Mini evaluation further tests the connection between these intervention decisions and physical outcomes.
We will incorporate this comparison into the revision to distinguish our learning, data, inference, and execution contributions from the shared architectural foundations.
Response to W2: Readability of Figure 1
Thank you for pointing this out. We will simplify Figure 1 and enlarge its internal labels, with particular attention to the timing-control, context-caching, and execution-feedback paths. We will move secondary implementation details to the caption or appendix and check readability at the figure's final size in the paper.