acceptodds
Under review as a conference paper at ICLR 2027

ONSTEP: Learning Proactive and Interactive Assistance in Streaming Egocentric Video

Abstract

Everyday procedural tasks require assistance that is timely, context-aware, proactive, and interactive. Although recent multimodal large language models (MLLMs) and streaming video assistants have advanced video understanding and real-time interaction, they still fall short of providing unified procedural assistance that tracks task progress, proactively intervenes at the right moment, and responds to user queries as the task unfolds. We instantiate this setting in egocentric cooking, a procedurally rich and error-prone domain with large-scale, step-annotated videos capturing both task progress and execution errors. To support learning and evaluation in this setting, we introduce ProactiveCooking, a temporally grounded dataset that augments proactive cooking demonstrations with 6,526 human-verified question-answer interactions spanning six complementary interaction types. ProactiveCooking provides both training supervision and a streaming evaluation testbed, covering standard recipe execution (Main set) as well as more challenging non-monotonic execution (Advanced set) with reordered, repeated, and skipped steps. Building on the training data, we introduce ONSTEP (ONset-gated, STEP-anchored Proactive guidance), an end-to-end streaming framework that explicitly models when to begin speaking, how long an utterance should continue, and where the user is in the task procedure. ONSTEP provides step-by-step instructions, visually grounded feedback on completed actions and mistakes, and real-time answers to user questions within a unified streaming loop. Across both the Main and Advanced test sets, ONSTEP consistently improves speaking-time F1 over the strongest proactive baseline from 35.04 to 39.70 and from 32.79 to 37.50, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Guidance, Egocentric, Proactivity, Streaming video understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wWi9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ProactiveCooking and ONSTEP, a framework for proactive and interactive assistance over streaming egocentric video. The goal is ambitious and practically relevant: instead of merely answering questions about a prerecorded video, an assistant must continuously track a user performing a multi-step procedure, determine when intervention is warranted, provide instructions and feedback, remain silent when appropriate, and answer user questions as they arise.

ProactiveCooking augments existing egocentric cooking demonstrations with 6,526 human-verified, temporally grounded question-answer interactions across six interaction types. ONSTEP builds on a streaming Qwen3-VL backbone and separates three aspects of procedural assistance: an onset gate determines when an utterance should begin, generated boundary tokens determine whether speech should continue or terminate, and a step-anchored contrastive objective encourages the model to represent its current location in an ordered procedure.

Experiments show consistent improvements over existing proactive/streaming baselines on both the Main and more difficult Advanced splits. In particular, speaking-time F1 improves from 35.04 to 39.70 on Main and from 32.79 to 37.50 on Advanced relative to Proact-VL. Ablations suggest that explicit onset modeling provides most of this improvement, with additional gains from onset-rate regularization and step-aware supervision.

Critical strengths
  1. The paper presents a useful unified formulation of proactive and interactive streaming assistance. Separating utterance onset from continuation is simple but well motivated. The cumulative ablation shows that the onset formulation accounts for a substantial portion of the timing improvement, while the step objective provides a smaller additional improvement. The Advanced split, containing reordered, repeated, and skipped steps, is particularly valuable because it tests whether the assistant can track execution rather than just advance monotonically through a recipe.
  2. The paper tackles an important correspondence/state-tracking problem that connects naturally to earlier tracking research. Procedural assistance requires maintaining the identity and state of task-relevant objects across time despite substantial visual change, occlusion, manipulation, and viewpoint changes. This connects to prior work demonstrating that neural networks can struggle when correspondence must be maintained independently of stable appearance cues. Linsley et al. (2021) showed that tracking identical-looking objects from motion alone remains challenging for standard architectures, while Muzellec et al. (2025) showed that state-of-the-art DNNs struggle when tracked objects change both appearance and location. This prior work seems particularly relevant to ONSTEP's reported failures in verifying fine-grained actions over time.
Critical weaknesses
  1. The largest gains concern intervention timing, while important failures remain in actual procedural and visual state tracking. The paper's failure analysis is concerning for the intended assistant setting. ONSTEP incorrectly answers many "how many steps are left" questions and can produce positive feedback when the reference indicates that an action was performed incorrectly. These are qualitatively different from slightly mistimed speech. An assistant saying the wrong thing at exactly the correct moment can be worse than remaining silent, especially when the feedback concerns whether a procedural step was performed correctly. This also raises a broader question about whether the step-anchored objective actually learns reliable procedural state. Its contribution to the timing metrics is measurable, but the model's explicit answers about procedural position remain weak. I would therefore prefer the paper to distinguish more sharply between timing-aware assistance and state-aware assistance.
  2. The interactive dataset is carefully verified, but the interactions themselves are synthetic rather than naturally elicited during task performance. The QA pairs are generated automatically and subsequently verified by humans. This is a reasonable way to construct supervision at scale, and the human verification is a strength. However, verification establishes correctness and timing of generated questions, not necessarily that real users would ask the same questions at those moments. This distinction matters for a model positioned as an interactive assistant because genuine user queries may depend on confusion, uncertainty, perceptual difficulty, or previous assistant behavior in ways that synthetic questions conditioned on the annotated step may not capture.
  3. The dataset in the paper seems to be based on a single task of cooking which skews the distribution of videos/tasks that can be solved through this method.
Questions for authors
  1. Clarification that could materially affect my assessment: How should the step-anchored representation be interpreted given the model's failures on explicit procedural-position questions? Is the improvement from the step objective evidence that the model genuinely tracks the current step, or could it primarily provide an auxiliary signal for predicting when reference utterances tend to occur? An evaluation that decodes current-step identity directly from the streaming state on Main and Advanced would help distinguish these possibilities.
  2. Clarification that could materially affect my assessment: Can the authors provide a more direct quantitative evaluation of action-state verification, particularly mistake detection, separately from timing? The demonstrated case where ONSTEP speaks at the correct time but confirms an incorrectly executed action as successful suggests that the temporal and perceptual contributions need to be evaluated more independently.
  3. Suggestion for positioning/future work: I encourage the authors to connect procedural state tracking more explicitly to the object-tracking/correspondence literature. Prior work shows that maintaining correspondence becomes difficult for neural networks when appearance cues are absent (Linsley et al., 2021) or when object appearance changes substantially during tracking (Muzellec et al., 2025). Cooking provides precisely these types of long-horizon transformations through manipulation, occlusion, viewpoint changes, and changing object state. This distinction may help separate failures of memory/procedural position from failures of visual correspondence.
  4. Did the authors get approval from the IRB for the human annotations? I did not find any such disclosures in the text.

References

Linsley, D., Malik, G., Kim, J., Govindarajan, L. N., Mingolla, E., & Serre, T. (2021). Tracking without re-recognition in humans and machines. Advances in Neural Information Processing Systems, 34, 19473–19486.

Muzellec, S., Linsley, D., Ashok, A. K., Mingolla, E., Malik, G., VanRullen, R., & Serre, T. (2025). Tracking objects that change in appearance with phase synchrony. The Thirteenth International Conference on Learning Representations.

4 comments
AuthorsOpenReview ↗

Part 1 of 2

We sincerely value your time and feedback. Below, we have carefully addressed your specific concerns and questions individually and updated the main paper to reflect the changes.

W2. interaction in the ProactiveCooking dataset.

We thank the reviewer for this point and agree with the distinction. Human verification establishes that a question is correct, answerable and well timed. It does not establish that a real user would have asked it at that moment. Still, we believe the retained questions are a reasonable stand-in for questions a real user would ask, for three reasons:

They are written from the user's point of view. The generator may only produce questions that "a cook would genuinely ask while cooking to make progress on the current step". It may not ask about trivial details, or about objects already in the cook's hands (Appendix B).

Human review removed irrelevant questions, not only incorrect ones. Of 9,486 generated candidates, fewer than 70% were kept. A text-only reviewer rejected 768 questions that were redundant, irrelevant or unnecessary in context, such as asking which spoon to use while already holding it (Appendix A.5). Three video annotators per item then rejected a further 169 questions that "should never have been asked" and corrected mistimed ones. 70.7% of their decisions were unanimous.

The question types reflect common in-task needs: instruction details, object location, culinary knowledge, recipe progress and mid-step state (Section 3.1).

We agree that this does not replace observing real users, but this multi-stage filtering keeps only questions that are plausible and well-timed for the situation shown.

W3. The dataset in the paper seems to be based on a single task of cooking which skews the distribution of videos/tasks that can be solved through this method.

We thank the reviewer for raising this, and we agree that all our experiments are in cooking. Below we explain why we chose cooking and how our method can be applied beyond it.

1. Our setting needs videos with time-aligned assistant instructions, feedback on correct and mistaken execution, and user interactions. As our comparison of datasets shows (Table 12, shown below as well), most procedural datasets do not provide this. Assembly-101, HoloAssist, COIN and Ego-Exo4D lack time-aligned instruction-and-feedback transcripts. The QIC benchmark on CaptainCook4D is, to our knowledge, the large-scale source that does, and it covers cooking. Cooking is also a demanding test case: the user's hands are busy, steps take different amounts of time, and errors are common and observable.

DatasetDomainMulti-step Goal DrivenStep-by-Step InstructionsTimed FeedbackTimed InteractionLength (hrs)
Epic-KitchensCooking✗✗✗✗100
Ego-4DDaily-life✗✗✗✗3670
Ego-Exo4DDaily-life✗✗✗✗1422
Assembly-101Toy Assembly✓✗✗✗513
HoloAssistObj. manip.✓✗✗✗166
HowTo100MDiverse✓✓✗✗134k
COINDiverse✓✓✗✗512
YouCook2Cooking✓✓✗✗176
WTAGCooking✓✓✗✗10
QEVDFitness✗✓✓✗474
QICCooking✓✓✓✗94
OursCooking✓✓✓✓94

2. ProactiveCooking spans many recipes, participants and kitchens. Its Advanced set adds non-monotonic execution, with reordered, repeated and skipped steps, which differs substantially from following a recipe in order. ONSTEP's gains hold on both sets: F1 rises from 35.04 to 39.70 on Main and from 32.79 to 37.50 on Advanced.

3. ONSTEP only requires a video stream and an ordered task plan:

  • the onset head learns when to start speaking;
  • the <|cont|>/<|done|> tokens learn when to continue or stop;
  • the step-anchored objective aligns the stream with a text encoding of the plan's steps.

The components are domain-agnostic. The same holds for our data pipeline: it generates questions from a step plan and the frames of each step, then verifies them with humans. It could therefore be applied to other step-annotated procedural videos, such as assembly or repair, once instruction and feedback timing are available.

AuthorsOpenReview ↗

Part 2 of 2

W1Q1. interpretation of step-anchored representation and evaluation that decodes current-step identity directly from the streaming state

We thank the reviewer for this thoughtful question. We have added a discussion in SECTION G.2 of the appendix.

As suggested, we decoded the current step directly from the streaming state on both Main and Advanced. At every second, before the model generates any text, we compare its internal STEP feature with the recipe's step embeddings and take the closest step as its prediction. We also compare it against an elapsed-time prior: a baseline that predicts the step purely from how much time has passed, using typical step timing from the training videos. If the step representation only encoded "when utterances tend to occur", it should do little better than this prior.

It beats time-based guessing by a wide margin: about 3× the elapsed-time prior on Main and about 2× on Advanced. The lead holds against a stronger prior that is even told each test video's true duration (53.1% vs. 35.6% on Main). It also holds throughout the video, by quarter: 70.4 / 49.5 / 42.3 / 51.5% vs. 47.7 / 28.0 / 23.1 / 44.4% for the stronger time prior.

Accuracy on silent seconds is 53.1% on Main and 40.9% on Advanced, about the same as overall. So the representation is not tied to the moments when the model speaks. After the reference step changes, the decoded state switches to the new step in 73.3% of cases on Main, with a median delay of 1s. On Advanced it switches in 57.0% of cases, with a median delay of 2s. On Advanced, where users deviate from the recipe, it also recovers after 60.2% of skipped steps (median 2s) and 41.2% of backward jumps.

At the 131 "how many steps are left" questions, the step state identifies the current step exactly in 48.1% of cases and within one step in 77.9%. Yet the language model's spoken answers are wrong for 83 of 131 (36.6% accuracy). So the internal state often knows where the user is, but turning that into a correct verbal count is a separate capability, and it is weaker. This is especially true when steps are skipped or reordered.

SetSeconds EvaluatedONSTEP Step StateElapsed Time PriorOracle Time PriorChance
Main9311053.118.235.67.5
Advanced2752139.821.136.110.5

Q2. a direct quantitative evaluation of action-state verification.

We evaluated the content of each model's feedback independently of timing, scoring every response against the human success/mistake labels. An LLM judge read all the response sentences from ONSTEP and Proact-VL, with the method names hidden. For each sentence, it identified the annotated action it refers to and checked the model's claim against the human label. There is no timing requirement.

SetMethodAcc. of Verdict ↑Mistake-Claim Precision ↑False Alarm on Successes ↓False Praise of Mistakes ↓Mistakes Caught ↑
MainONSTEP(ours)59.033.616.359.124.4
MainProact-VL56.732.620.166.222.6
AdvancedONSTEP(ours)37.351.950.727.136.1
AdvancedProact-VL32.550.358.829.235.2

Q3. Connection with Linsley et al., 2021 and Muzellec et al., 2025

Thank you for proposing this suggestion and pointing us to these works. Procedural state tracking needs both: knowing which step is under way and recognising an object as it is chopped, melted, browned or hidden from view. Our analysis (Appendix I.1) shows both kinds of error. Step-count mistakes reflect procedural memory, and missed execution errors reflect visual verification. We have added a discussion section that connects procedural state tracking to the object-tracking literature (Linsley et al., 2021; Muzellec et al., 2025). It proposes evaluations that separate memory failures from correspondence failures, and discusses combining step-anchored state with explicit object-level tracking.

Q4. IRB review

The study received approval from the relevant institutional ethics review committee prior to data collection. Further institutional details are withheld to preserve anonymity during the review process.

Thank you for the clarifications. No additional information needed at my end.

We appreciate your positive assessment of our work and thank you for taking the time to review our rebuttal and consider our clarifications. We are glad that we were able to address your concerns and sincerely appreciate your constructive feedback.

Reviewer_7prCOpenReview ↗
Rating
4(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ONSTEP, a framework for proactive streaming egocentric video assistance that combines a human-verified interaction dataset, onset prediction, cross-chunk response control, and step-aware contrastive learning to improve response timing and interaction performance.

Critical strengths

The work is comprehensive, covering dataset construction, model training, and evaluation. The dataset incorporates diverse question types and interaction patterns, including both instant and proactive responses. The proposed method combines onset prediction, cross-chunk response control, and step-aware contrastive learning to address key challenges in proactive streaming interaction. Furthermore, the evaluation employs multiple metrics to assess different aspects of model performance, including response accuracy and timing.

Critical weaknesses
  1. ProactiveCooking primarily focuses on procedural guidance. Although it incorporates both instant responses and proactive responses, which are representative of proactive streaming interaction, its coverage of other interaction scenarios remains limited.
  2. Proactive streaming interaction is intended for real-time applications, where inference latency is critical. However, the paper does not provide an analysis of actual inference latency, making it difficult to assess the practical applicability of the proposed framework.
Questions for authors
  1. Given that ProactiveCooking primarily focuses on procedural guidance, do the authors expect ONSTEP to generalize to other proactive streaming interaction scenarios, such as real-time monitoring or event alerts? How generalizable is the proposed framework beyond guidance-oriented tasks?
  2. Could the authors provide an analysis of the model's inference latency, particularly under continuous streaming inputs, to demonstrate its feasibility for real-time applications?
  3. Since the timing of proactive responses is inherently subjective, multiple nearby response timings may be equally reasonable, while the current speaking-time F1 requires strict temporal overlap with the reference annotations. Could the authors additionally report tolerance-based metrics, such as F1@3s and F1@5s, to provide a more flexible evaluation of response timing?
5 comments
AuthorsOpenReview ↗

Part 1 of 2

We greatly value your time and feedback. Below, we have carefully addressed your specific concerns and questions individually. We have integrated the changes into the main paper.

Q1. generalization beyond guidance-oriented tasks?

We thank the reviewer for this question. Yes, we expect ONSTEP to generalize beyond guidance-oriented tasks.

ONSTEP already handles real-time event queries. About a third of Main set's test questions (413/1142) ask about the scene as it unfolds. They must be answered from the live stream alone: where an object is right now ("Where's the olive oil bottle right now?"), what is happening now ("Is the pan covered right now?"), and the state of an ongoing process ("Is the oil hot enough yet?"). These are the most demanding questions in the benchmark, because answers must localize objects in the live scene, which our annotation analysis (Appendix A.4) identifies as the main source of difficulty. We rated ONSTEP's answers with the paper's AnswerQ rubric (1–5), using an independent LLM judge:

MethodReal-time questionsCorrectMean AnswerQ
Proact-VL41357.9%2.90
ONSTEP(ours)41359.8%2.93

ONSTEP performs slightly better than Proact-VL.

Event alerts. In principle, ONSTEP can support event alerts. Its proactive feedback is already issued without a user prompt: the model detects an event in the stream and decides when to speak. An alert follows a similar pattern but ProactiveCooking does not contain dedicated alert events, such as safety violations or abnormal states, so we have not yet validated this setting. We will extend ONSTEP to event-alert scenarios in future work. We have added this in the discussion section of the main paper.

Q2. analysis of the model's inference latency, under continuous streaming inputs

We have added a detailed analysis in Section 5.4, and discuss the details below.

MetricValue
Full chunk latency, median/p9595.2/348.2 ms
Silent Chunks: mean/p95120.7/112.2 ms
Speaking Chunks: mean/median/p95399.8/339.0/439.3 ms
Visual Prefill + speak/stay-silent decision, median/p9564.3/81.9 ms
Generation throughput32.14 tokens/s
Real Time factor per stream(processing time ÷ video time)0.151, 6.62x faster than playback

In table above, the model receives one second of video at a time and must finish processing it before the next second arrives. In practice, half of all chunks are processed in under 0.1 s (median 95.2 ms). 95% are processed within 0.35 s (p95 348.2 ms). Most of the time the model decides to stay silent, and those chunks are the fastest. When the model does speak, generating the response raises the chunk time to about 0.34 s (median), which is still well within the one-second budget. Generation runs at about 32 tokens per second, so a short response is produced in a fraction of a second. Overall, ONSTEP processes video 6.6× faster than playback (real-time factor 0.151): one second of video takes about 0.15 s of compute.

Below, we split each chunk's time, taking in the new frames and updating the cache ("Cache"), then producing the output ("Forward"). Across all chunks, an average chunk takes 0.15 s. Chunks where the model speaks take 0.40 s on average, and most of that is spent generating the response. Excluding the reader's 60-second refills and the first few chunks of each video, the average chunk takes 0.12 s and no chunk exceeds one second. This confirms that the slow chunks come from file reading, not from the model.

ChunksPeak MemoryCache update(s)Forward(s)Full chunk(s)Per token(s)
All20.25 GB0.09360.05750.15110.0311
Speaking20.25 GB0.12230.27740.39980.0311

Further, speed does not degrade as a stream gets longer. Throughput and latency after the first five minutes are essentially the same as in the first five minutes (p95 345.8 ms vs. 364.9 ms). Because the KV cache has a fixed budget, memory stays bounded however long the video runs. A 35.4-minute video was processed continuously in 5.3 minutes.

Part of streamGeneration Throughputp95 chunk latencyReal-time factor
First 5 minutes31.64 tok/sec364.9 ms0.154
After 5 minutes32.46 tok/sec345.8 ms0.150
AuthorsOpenReview ↗

Part 2 of 2

Q3. report tolerance-based metrics, such as F1@3s and F1@5s

Thank you for the constructive suggestion. We used F1 to follow the practice of previous work in streaming-video settings [1,2]. But, we agree that F1 requires strict temporal overlap. Below, we have reported the two tolerant metrics, F1@3 and F1@5. Two points stand out. First, all models score higher as the tolerance widens. Second, ONSTEP ranks first at both tolerances on both test sets, so its lead over every baseline holds even when nearby timings are accepted. We discuss these results in Appendix E.3.

ModelMain(orig)Main(F1@3)Main(F1@5)Adv(orig)Adv(F1@3)Adv(F1@5)
Qwen319.0042.2351.2822.5045.7354.48
LiveStar8.4020.9526.7110.1725.3231.00
MMDuet217.7035.3943.4020.7438.5946.26
VideoLLM-Online33.6652.2760.1929.0050.8059.96
LiveCC-Base12.2321.0923.4117.1527.0929.99
LiveCC-Instruct18.7240.1849.2522.5945.1254.10
MageVL11.3939.0649.4012.8744.1753.82
JoyAI-VL21.6455.7664.3919.1350.4059.20
Proact-VL35.0454.5162.9132.7953.8062.42
ONSTEP(ours)39.7062.1170.1637.5059.6567.99

[1] Proact-VL: A Proactive Video LLM for Real-Time AI Companions: Weicai Yan and Yuhong Dai and Qi Ran and Haodong Li and Wang Lin and Tao Jin and Xing Xie and Hao Liao and Jianxun Lian, ICML 2026

[2] Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction: Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, Jiaqi Wang, CVPR 2025

Dear Reviewer,

As the discussion period approaches its end, we would like to extend our sincere gratitude for your time and thoughtful feedback. In our previous response, we carefully considered your comments and offered detailed clarifications on the following points:

  • Generalization beyond guidance-oriented tasks
  • Analysis of inference latency
  • Tolerance-based F1 metric

We hope these explanations have addressed your concerns. Should you have any remaining questions or require further details, we would be glad to provide additional clarification.

Thank you once again for your valuable review of our work.

Reviewer_qMSZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ProactiveCooking, which extends existing cooking demonstrations with 6,526 human-verified, temporally grounded question–answer interactions. It also proposes ONSTEP, a streaming assistant that combines an onset-prediction head, continuation and termination tokens, and a step-anchored contrastive objective. The reported improvements primarily concern speaking-time alignment, with relatively small changes in answer correctness.

Critical strengths
  1. The dataset addresses a useful setting: combining proactive guidance with user-initiated questions during procedural activities. Human verification of both answer content and question timing, together with the reported correction and agreement statistics, adds value beyond automatic question generation.

  2. Separating utterance onset from continuation is well motivated. The comparison between isolated-step and causal-prefix representations in Appendix G is also informative and supports the importance of distinguishing repeated instructions at different procedural positions.

Critical weaknesses
  1. The baseline training conditions are insufficiently specified.

    Section 5.1 describes fine-tuning ONSTEP on the proposed data, but does not clearly establish whether the competing models receive comparable task-specific training, supervision, and optimization budgets. This is particularly important because several baselines perform near the floor on interactive QA.

    Without this information, the results cannot cleanly distinguish architectural improvements from the benefits of adaptation to the new dataset. A same-backbone streaming baseline trained on the same data with a conventional speaking/silence objective is necessary to establish the contribution of the proposed formulation.

  2. The methodological contribution is incremental, and the central design is not sufficiently isolated.

    ONSTEP builds on an existing chunk-wise streaming interface and combines a linear onset classifier, weighted binary cross-entropy, generated boundary tokens, and an InfoNCE objective. These components form a reasonable system, but the paper does not yet demonstrate a substantial methodological advance beyond their task-specific combination.

    In Table 3, the onset head and boundary tokens are introduced together, producing the largest improvement. This does not isolate the benefit of onset-only supervision from that of explicit continuation control or other changes associated with streaming adaptation. The step objective adds 1.51 F1 points, but repeated-run variability is not reported. More controlled comparisons are needed to establish which design choices are necessary and how robust their contributions are.

  3. The timing supervision relies on synthetic assumptions that require stronger validation.

    Appendix D.3 constructs speaking intervals using a fixed rate of five words per second and a six-second duration cap. It also places feedback backward from the annotated completion timestamp. Consequently, the evaluation measures alignment with partly constructed speaking schedules, rather than directly observed conversational timing.

    Backward placement raises an additional causal-grounding question: feedback about a completed action may be scheduled to begin before the action is complete. The paper should establish that the feedback is supported by the video prefix available at its assigned onset. This matters because timing is both the primary methodological target and the main source of reported improvement.

Questions for authors
  1. Which baselines were fine-tuned on ProactiveCooking, and were their training data, supervision, backbone sizes, and optimization budgets comparable to ONSTEP?

  2. Can the authors report actual inference latency and throughput, and provide evidence from real user interactions or a closed-loop evaluation?

5 comments
AuthorsOpenReview ↗

Part 1 of 2

We greatly appreciate your time and feedback. Below, we have carefully addressed your specific concerns and questions individually and integrated changes into the main paper.

W1Q1. baseline training conditions.

Thank you for the suggestion. Results from Tables 1 and 2 already cover the models trained from the same data, below we summarize the comparisons.

MethodBackboneSizeTraining dataGPU-hoursTest setSetting
VideoLLM-OnlineLlama-38BProactiveCooking70Main & AdvancedStreaming
Proact-VLQwen3-VL8BProactiveCooking70Main & AdvancedStreaming
ONSTEP(ours)Qwen3-VL8BProactiveCooking70Main & AdvancedStreaming

We can see that, Proact-VL the best performing baseline, shares the same backbone and was trained on same data with same computational budget with a conventional speaking/silence objective. Below, we can see that ONSTEP performs better than Proact-VL across various metrics, confirming that the gains come from the ONSTEP formulation rather than from adapting to the dataset.

MethodTest SetF1 ↑TimeDiff ↓pAUC ↑
Proact-VLMain35.044.0437.33
VideoLLM-OnlineMain33.666.2516.73
ONSTEP(ours)Main39.703.4338.03
Proact-VLAdvanced32.794.4033.60
VideoLLM-OnlineAdvanced29.008.2111.90
ONSTEP(ours)Advanced37.503.7434.51

We have also added the details in Appendix Section H.

W2. incremental methodological contribution

ONSTEP introduces several components that set it apart from previous work:

1. Technically, the onset head, the boundary tokens, and the step embeddings with their supervision are all new. (i) Conventional models label every second of an utterance as "speak". Most of these labels (58.7%) fall on seconds where the assistant is already talking. ONSTEP supervises only onsets. Termination becomes a generated token (<|cont|> / <|done|>), which adds no loss term or parameters. (ii) Our causal-prefix step embeddings tell the model explicitly where it is in the recipe, which improves its awareness of the current stage.

To our knowledge, prior proactive methods have explored neither design, especially in goal-oriented settings. So, ONSTEP introduces an end-to-end proactive streaming assistant that dynamically decides when to speak or remain silent, while keeping track of the user's progress in goal-oriented tasks.

W2. Isolating the contribution of the onset head

We thank the reviewer for this suggestion. We did not isolate the two, as they go hand in hand, the onset head tells when to start and boundary tokens control the stopping. To isolate the effect of Onset head, we trained our model with the onset head alone, without the <|cont|> / <|done|> boundary tokens, keeping all other settings identical. Without boundary tokens, the onset head is queried every second and the model speaks when it fires.

ConfigurationF1 ↑TimeDiff ↓pAUC ↑
Qwen3-VL19.008.1022.71
+ onset head only26.774.0032.55
+ onset head + boundary tokens35.334.2137.10

The two components play complementary roles:

  • The onset head decides when to start speaking. On its own, it halves TimeDiff (8.10 → 4.00 s) and improves F1 by 7.77 and pAUC by 9.84 points over the backbone.
  • The boundary tokens decide how long to continue. Without them, the model starts in as many reference utterances (coverage 48.7% vs. 47.5%) but stops early: it speaks 22% fewer seconds, and recall of reference speaking time falls from 36.5% to 24.5%. Adding the tokens raises F1 by a further 8.56 points and pAUC by 4.55 points.

TimeDiff changes little with the boundary tokens because it measures when each utterance starts, not whether it is completed. The slightly lower value without them reflects fewer seconds of speech outside the reference windows when the model speaks less, not better timing. We have added this ablation to Table 3 and its discussion in the main paper.

AuthorsOpenReview ↗

Part 2 of 2

Q2. inference latency and throughput

We provide a detailed analysis in Section 5.4, and discuss the details below.

MetricValue
Full chunk latency, median/p9595.2/348.2 ms
Silent Chunks: mean/p95120.7/112.2 ms
Speaking Chunks: mean/median/p95399.8/339.0/439.3 ms
Visual Prefill + speak/stay-silent decision, median/p9564.3/81.9 ms
Generation throughput32.14 tokens/s
Real Time factor per stream(processing time ÷ video time)0.151, 6.62x faster than playback

In table above, the model receives one second of video at a time and must finish processing it before the next second arrives. In practice, half of all chunks are processed in under 0.1 s (median 95.2 ms). 95% are processed within 0.35 s (p95 348.2 ms). Most of the time the model decides to stay silent, and those chunks are the fastest. When the model does speak, generating the response raises the chunk time to about 0.34 s (median), which is still well within the one-second budget. Generation runs at about 32 tokens per second, so a short response is produced in a fraction of a second. Overall, ONSTEP processes video 6.6× faster than playback (real-time factor 0.151): one second of video takes about 0.15 s of compute.

Below, we split each chunk's time, taking in the new frames and updating the cache ("Cache"), then producing the output ("Forward"). Across all chunks, an average chunk takes 0.15 s. Chunks where the model speaks take 0.40 s on average, and most of that is spent generating the response. Excluding the reader's 60-second refills and the first few chunks of each video, the average chunk takes 0.12 s and no chunk exceeds one second. This confirms that the slow chunks come from file reading, not from the model.

ChunksPeak MemoryCache update(s)Forward(s)Full chunk(s)Per token(s)
All20.25 GB0.09360.05750.15110.0311
Speaking20.25 GB0.12230.27740.39980.0311

Further, speed does not degrade as a stream gets longer. Throughput and latency after the first five minutes are essentially the same as in the first five minutes (p95 345.8 ms vs. 364.9 ms). Because the KV cache has a fixed budget, memory stays bounded however long the video runs. A 35.4-minute video was processed continuously in 5.3 minutes.

Part of streamGeneration Throughputp95 chunk latencyReal-time factor
First 5 minutes31.64 tok/sec364.9 ms0.154
After 5 minutes32.46 tok/sec345.8 ms0.150

Q2. Real user interactions / closed-loop evaluation

Each test video was recorded in a real-world setting, and its instructions and feedback are the guidance actually given to the person during the task. Humans annotated or verified every QA pair, both proactive and passive, and rewrote any pair that was not realistic. The test set spans diverse tasks and environments and three interaction types: instructions, feedback and answers. A closed-loop evaluation would need new live participants for every model and ablation, and its results could not be compared fairly, because each system would see a different interaction. Recorded interactions keep the input identical across methods as done in previous work [1]. We will also release video demos of ONSTEP.

[1] Proact-VL: A Proactive VideoLLM for Real-Time AI Companions: Weicai Yan and Yuhong Dai and Qi Ran and Haodong Li and Wang Lin and Tao Jin and Xing Xie and Hao Liao and Jianxun Lian, ICML 2026

Dear Reviewer,

As the discussion phase nears its conclusion, we would like to express our sincere appreciation for your time and thoughtful comments. In our previous response, we carefully reviewed your feedback and provided detailed clarifications regarding:

  • Clarifying the baseline training conditions
  • Clarification about the technical contribution
  • Isolating the ONSET head's contribution
  • Analysis of inference latency
  • User interaction evaluation

We hope our explanations have adequately addressed your concerns. If there are any remaining questions or points that would benefit from further clarification, we would be happy to elaborate.

Thank you once again for your valuable assessment of our work.

Comments on the paper

1 comment

Dear Reviewers and Area Chairs,

We sincerely thank all reviewers for their insightful and constructive feedback. Below, we summarize the current status of the discussion and the changes made to the paper.

Score Summary: Our initial ratings were [2, 3, 3], and they currently stand at [2, 4, 3].

Reviewer Engagement:

  • Reviewer 7prC confirmed that our additional experiments and clarifications resolved all of their concerns and raised their score from 3 to 4 (clear acceptance).
  • Reviewer wWi9 acknowledged our clarifications, said no further information was needed, and kept their recommendation for acceptance.
  • Reviewer qMSZ has not yet responded to our rebuttal. We believe our responses address each of their concerns: baseline training conditions, the onset-head ablation, the technical contribution, inference latency, and the evaluation protocol.
  1. We clarify the baseline training conditions. Proact-VL uses the same Qwen3-VL-8B backbone, training data, optimization settings, and streaming interface as ONSTEP, and differs only in its training objective. This is in Appendix H, in blue (reviewer qMSZ).
  2. We isolate the contribution of the onset head from the boundary tokens. We added an "onset head only" ablation and its analysis in Table 3 and Sec. 5.3, in blue (reviewer qMSZ).
  3. We add an inference latency and throughput analysis: per-chunk latency, real-time factor of 0.151 (6.62× faster than playback), latency breakdown, and long-stream efficiency. This is in Sec. 5.4 and Tables 4–6, in blue (reviewer qMSZ, 7prC).
  4. We add tolerance-based timing metrics (F1@3s and F1@5s) for all models on the Main and Advanced sets. This is in Appendix E.3 and Table 15, in blue (reviewer 7prC).
  5. We add a direct evaluation of whether the step-anchored representation tracks the current step. We decode the step from the streaming state and compare it against elapsed-time and oracle time priors, recovery after step changes, and step-count questions. This is in Appendix G.2 and Table 18, in blue (reviewer wWi9).
  6. We discuss how procedural state tracking connects to the object-tracking and correspondence literature in Sec. 6, in blue (reviewer wWi9).
  7. We discuss how ONSTEP could generalize to event-alert scenarios beyond guidance-oriented tasks in Sec. 6, in blue (reviewer 7prC).

Summary of Contributions:

  • A temporally anchored, grounded interactive cooking dataset. ProactiveCooking is a benchmark of six types of question-answer turns, built with an agentic generation pipeline followed by two stages of human verification.
  • A unified proactive and interactive guidance system. We introduce ONSTEP, which combines onset prediction with a step-anchored loss to provide real-time guidance.
  • Consistent gains in timing and assistance quality. Our evaluation shows that ONSTEP outperforms baselines on both guidance timing and response quality.

We thank the reviewers and Area Chairs again for their time and valuable feedback.

The Authors of Submission #22380

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.