- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ProactiveCooking and ONSTEP, a framework for proactive and interactive assistance over streaming egocentric video. The goal is ambitious and practically relevant: instead of merely answering questions about a prerecorded video, an assistant must continuously track a user performing a multi-step procedure, determine when intervention is warranted, provide instructions and feedback, remain silent when appropriate, and answer user questions as they arise.
ProactiveCooking augments existing egocentric cooking demonstrations with 6,526 human-verified, temporally grounded question-answer interactions across six interaction types. ONSTEP builds on a streaming Qwen3-VL backbone and separates three aspects of procedural assistance: an onset gate determines when an utterance should begin, generated boundary tokens determine whether speech should continue or terminate, and a step-anchored contrastive objective encourages the model to represent its current location in an ordered procedure.
Experiments show consistent improvements over existing proactive/streaming baselines on both the Main and more difficult Advanced splits. In particular, speaking-time F1 improves from 35.04 to 39.70 on Main and from 32.79 to 37.50 on Advanced relative to Proact-VL. Ablations suggest that explicit onset modeling provides most of this improvement, with additional gains from onset-rate regularization and step-aware supervision.
Critical strengths
- The paper presents a useful unified formulation of proactive and interactive streaming assistance. Separating utterance onset from continuation is simple but well motivated. The cumulative ablation shows that the onset formulation accounts for a substantial portion of the timing improvement, while the step objective provides a smaller additional improvement. The Advanced split, containing reordered, repeated, and skipped steps, is particularly valuable because it tests whether the assistant can track execution rather than just advance monotonically through a recipe.
- The paper tackles an important correspondence/state-tracking problem that connects naturally to earlier tracking research. Procedural assistance requires maintaining the identity and state of task-relevant objects across time despite substantial visual change, occlusion, manipulation, and viewpoint changes. This connects to prior work demonstrating that neural networks can struggle when correspondence must be maintained independently of stable appearance cues. Linsley et al. (2021) showed that tracking identical-looking objects from motion alone remains challenging for standard architectures, while Muzellec et al. (2025) showed that state-of-the-art DNNs struggle when tracked objects change both appearance and location. This prior work seems particularly relevant to ONSTEP's reported failures in verifying fine-grained actions over time.
Critical weaknesses
- The largest gains concern intervention timing, while important failures remain in actual procedural and visual state tracking. The paper's failure analysis is concerning for the intended assistant setting. ONSTEP incorrectly answers many "how many steps are left" questions and can produce positive feedback when the reference indicates that an action was performed incorrectly. These are qualitatively different from slightly mistimed speech. An assistant saying the wrong thing at exactly the correct moment can be worse than remaining silent, especially when the feedback concerns whether a procedural step was performed correctly. This also raises a broader question about whether the step-anchored objective actually learns reliable procedural state. Its contribution to the timing metrics is measurable, but the model's explicit answers about procedural position remain weak. I would therefore prefer the paper to distinguish more sharply between timing-aware assistance and state-aware assistance.
- The interactive dataset is carefully verified, but the interactions themselves are synthetic rather than naturally elicited during task performance. The QA pairs are generated automatically and subsequently verified by humans. This is a reasonable way to construct supervision at scale, and the human verification is a strength. However, verification establishes correctness and timing of generated questions, not necessarily that real users would ask the same questions at those moments. This distinction matters for a model positioned as an interactive assistant because genuine user queries may depend on confusion, uncertainty, perceptual difficulty, or previous assistant behavior in ways that synthetic questions conditioned on the annotated step may not capture.
- The dataset in the paper seems to be based on a single task of cooking which skews the distribution of videos/tasks that can be solved through this method.
Questions for authors
- Clarification that could materially affect my assessment: How should the step-anchored representation be interpreted given the model's failures on explicit procedural-position questions? Is the improvement from the step objective evidence that the model genuinely tracks the current step, or could it primarily provide an auxiliary signal for predicting when reference utterances tend to occur? An evaluation that decodes current-step identity directly from the streaming state on Main and Advanced would help distinguish these possibilities.
- Clarification that could materially affect my assessment: Can the authors provide a more direct quantitative evaluation of action-state verification, particularly mistake detection, separately from timing? The demonstrated case where ONSTEP speaks at the correct time but confirms an incorrectly executed action as successful suggests that the temporal and perceptual contributions need to be evaluated more independently.
- Suggestion for positioning/future work: I encourage the authors to connect procedural state tracking more explicitly to the object-tracking/correspondence literature. Prior work shows that maintaining correspondence becomes difficult for neural networks when appearance cues are absent (Linsley et al., 2021) or when object appearance changes substantially during tracking (Muzellec et al., 2025). Cooking provides precisely these types of long-horizon transformations through manipulation, occlusion, viewpoint changes, and changing object state. This distinction may help separate failures of memory/procedural position from failures of visual correspondence.
- Did the authors get approval from the IRB for the human annotations? I did not find any such disclosures in the text.
References
Linsley, D., Malik, G., Kim, J., Govindarajan, L. N., Mingolla, E., & Serre, T. (2021). Tracking without re-recognition in humans and machines. Advances in Neural Information Processing Systems, 34, 19473–19486.
Muzellec, S., Linsley, D., Ashok, A. K., Mingolla, E., Malik, G., VanRullen, R., & Serre, T. (2025). Tracking objects that change in appearance with phase synchrony. The Thirteenth International Conference on Learning Representations.
4 comments
Part 2 of 2
W1Q1. interpretation of step-anchored representation and evaluation that decodes current-step identity directly from the streaming state
We thank the reviewer for this thoughtful question. We have added a discussion in SECTION G.2 of the appendix.
As suggested, we decoded the current step directly from the streaming state on both Main and Advanced. At every second, before the model generates any text, we compare its internal STEP feature with the recipe's step embeddings and take the closest step as its prediction. We also compare it against an elapsed-time prior: a baseline that predicts the step purely from how much time has passed, using typical step timing from the training videos. If the step representation only encoded "when utterances tend to occur", it should do little better than this prior.
It beats time-based guessing by a wide margin: about 3× the elapsed-time prior on Main and about 2× on Advanced. The lead holds against a stronger prior that is even told each test video's true duration (53.1% vs. 35.6% on Main). It also holds throughout the video, by quarter: 70.4 / 49.5 / 42.3 / 51.5% vs. 47.7 / 28.0 / 23.1 / 44.4% for the stronger time prior.
Accuracy on silent seconds is 53.1% on Main and 40.9% on Advanced, about the same as overall. So the representation is not tied to the moments when the model speaks. After the reference step changes, the decoded state switches to the new step in 73.3% of cases on Main, with a median delay of 1s. On Advanced it switches in 57.0% of cases, with a median delay of 2s. On Advanced, where users deviate from the recipe, it also recovers after 60.2% of skipped steps (median 2s) and 41.2% of backward jumps.
At the 131 "how many steps are left" questions, the step state identifies the current step exactly in 48.1% of cases and within one step in 77.9%. Yet the language model's spoken answers are wrong for 83 of 131 (36.6% accuracy). So the internal state often knows where the user is, but turning that into a correct verbal count is a separate capability, and it is weaker. This is especially true when steps are skipped or reordered.
| Set | Seconds Evaluated | ONSTEP Step State | Elapsed Time Prior | Oracle Time Prior | Chance |
|---|---|---|---|---|---|
| Main | 93110 | 53.1 | 18.2 | 35.6 | 7.5 |
| Advanced | 27521 | 39.8 | 21.1 | 36.1 | 10.5 |
Q2. a direct quantitative evaluation of action-state verification.
We evaluated the content of each model's feedback independently of timing, scoring every response against the human success/mistake labels. An LLM judge read all the response sentences from ONSTEP and Proact-VL, with the method names hidden. For each sentence, it identified the annotated action it refers to and checked the model's claim against the human label. There is no timing requirement.
| Set | Method | Acc. of Verdict ↑ | Mistake-Claim Precision ↑ | False Alarm on Successes ↓ | False Praise of Mistakes ↓ | Mistakes Caught ↑ |
|---|---|---|---|---|---|---|
| Main | ONSTEP(ours) | 59.0 | 33.6 | 16.3 | 59.1 | 24.4 |
| Main | Proact-VL | 56.7 | 32.6 | 20.1 | 66.2 | 22.6 |
| Advanced | ONSTEP(ours) | 37.3 | 51.9 | 50.7 | 27.1 | 36.1 |
| Advanced | Proact-VL | 32.5 | 50.3 | 58.8 | 29.2 | 35.2 |
Q3. Connection with Linsley et al., 2021 and Muzellec et al., 2025
Thank you for proposing this suggestion and pointing us to these works. Procedural state tracking needs both: knowing which step is under way and recognising an object as it is chopped, melted, browned or hidden from view. Our analysis (Appendix I.1) shows both kinds of error. Step-count mistakes reflect procedural memory, and missed execution errors reflect visual verification. We have added a discussion section that connects procedural state tracking to the object-tracking literature (Linsley et al., 2021; Muzellec et al., 2025). It proposes evaluations that separate memory failures from correspondence failures, and discusses combining step-anchored state with explicit object-level tracking.
Q4. IRB review
The study received approval from the relevant institutional ethics review committee prior to data collection. Further institutional details are withheld to preserve anonymity during the review process.
Thank you for the clarifications. No additional information needed at my end.
We appreciate your positive assessment of our work and thank you for taking the time to review our rebuttal and consider our clarifications. We are glad that we were able to address your concerns and sincerely appreciate your constructive feedback.
Part 1 of 2
We sincerely value your time and feedback. Below, we have carefully addressed your specific concerns and questions individually and updated the main paper to reflect the changes.
We thank the reviewer for this point and agree with the distinction. Human verification establishes that a question is correct, answerable and well timed. It does not establish that a real user would have asked it at that moment. Still, we believe the retained questions are a reasonable stand-in for questions a real user would ask, for three reasons:
They are written from the user's point of view. The generator may only produce questions that "a cook would genuinely ask while cooking to make progress on the current step". It may not ask about trivial details, or about objects already in the cook's hands (Appendix B).
Human review removed irrelevant questions, not only incorrect ones. Of 9,486 generated candidates, fewer than 70% were kept. A text-only reviewer rejected 768 questions that were redundant, irrelevant or unnecessary in context, such as asking which spoon to use while already holding it (Appendix A.5). Three video annotators per item then rejected a further 169 questions that "should never have been asked" and corrected mistimed ones. 70.7% of their decisions were unanimous.
The question types reflect common in-task needs: instruction details, object location, culinary knowledge, recipe progress and mid-step state (Section 3.1).
We agree that this does not replace observing real users, but this multi-stage filtering keeps only questions that are plausible and well-timed for the situation shown.
We thank the reviewer for raising this, and we agree that all our experiments are in cooking. Below we explain why we chose cooking and how our method can be applied beyond it.
1. Our setting needs videos with time-aligned assistant instructions, feedback on correct and mistaken execution, and user interactions. As our comparison of datasets shows (Table 12, shown below as well), most procedural datasets do not provide this. Assembly-101, HoloAssist, COIN and Ego-Exo4D lack time-aligned instruction-and-feedback transcripts. The QIC benchmark on CaptainCook4D is, to our knowledge, the large-scale source that does, and it covers cooking. Cooking is also a demanding test case: the user's hands are busy, steps take different amounts of time, and errors are common and observable.
2. ProactiveCooking spans many recipes, participants and kitchens. Its Advanced set adds non-monotonic execution, with reordered, repeated and skipped steps, which differs substantially from following a recipe in order. ONSTEP's gains hold on both sets: F1 rises from 35.04 to 39.70 on Main and from 32.79 to 37.50 on Advanced.
3. ONSTEP only requires a video stream and an ordered task plan:
<|cont|>/<|done|>tokens learn when to continue or stop;The components are domain-agnostic. The same holds for our data pipeline: it generates questions from a step plan and the frames of each step, then verifies them with humans. It could therefore be applied to other step-annotated procedural videos, such as assembly or repair, once instruction and feedback timing are available.