ONSTEP: Learning Proactive and Interactive Assistance in Streaming Egocentric Video

Everyday procedural tasks require assistance that is timely, context-aware, proactive, and interactive. Although recent multimodal large language models (MLLMs) and streaming video assistants have advanced video understanding and real-time interaction, they still fall short of providing unified procedural assistance that tracks task progress, proactively intervenes at the right moment, and responds to user queries as the task unfolds. We instantiate this setting in egocentric cooking, a procedurally rich and error-prone domain with large-scale, step-annotated videos capturing both task progress and execution errors. To support learning and evaluation in this setting, we introduce ProactiveCooking, a temporally grounded dataset that augments proactive cooking demonstrations with 6,526 human-verified question-answer interactions spanning six complementary interaction types. ProactiveCooking provides both training supervision and a streaming evaluation testbed, covering standard recipe execution (Main set) as well as more challenging non-monotonic execution (Advanced set) with reordered, repeated, and skipped steps. Building on the training data, we introduce ONSTEP (ONset-gated, STEP-anchored Proactive guidance), an end-to-end streaming framework that explicitly models when to begin speaking, how long an utterance should continue, and where the user is in the task procedure. ONSTEP provides step-by-step instructions, visually grounded feedback on completed actions and mistakes, and real-time answers to user questions within a unified streaming loop. Across both the Main and Advanced test sets, ONSTEP consistently improves speaking-time F1 over the strongest proactive baseline from 35.04 to 39.70 and from 32.79 to 37.50, respectively.