acceptodds
Withdrawn from ICLR 2027

Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Proactive multimodal assistants must decide guidance is needed and to say, while tracking progress through extended procedures and helping users recover when they depart from the expected plan. Progress is limited by benchmarks that score isolated clips, cover narrow domains, or omit realistic deviations. We introduce , a smart-glasses dataset of procedures across four domains in which users commit step-level deviations and recover from them, paired with reference guidance. We combine it with five re-annotated egocentric and instructional datasets to form , a cross-domain benchmark that scores intervention timing, procedure completion, and response quality. Zero-shot, no frontier or open-weight model completes more than 26% of a procedure's interventions before its first error. We propose (Plan, Watch, Recover), which keeps an explicit plan state in the model's context and revises it after every intervention, including recovery steps after a deviation. Across three open-weight backbones, training on without a plan barely changes performance, while adding the dynamic plan raises average procedure completion from 20% to 27% and improves turn-level decisions and response quality for every backbone. The gains hold on held-out datasets and on deviations never seen in training.

Primary area:
datasets and benchmarks
Keywords:
Proactive AI, Procedural AI, Wearable AI, Egocentric AI, Guided Assistance, Multimodal Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dQ6wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EgoProactive, a dataset with scripted task deviations and recovery guidance, and Pro²Bench, which combines it with re-annotated public datasets. It also proposes PWR, where an interaction model decides whether to stay silent or give guidance. After an intervention, a separate planner updates the completed, current, and remaining steps. This lets the plan change when execution differs from the original instructions.

Critical strengths
  1. Recording mistakes together with recovery and intervention guidance addresses an important problem. Combining multiple sources gives the benchmark broader coverage. I see the data as the main contribution.

  2. The method improves across several backbones. The same-data, no-plan comparisons help show the benefit of keeping an explicit task plan.

Critical weaknesses
  1. Recovery is tested only on the authors' scripted dataset. Testing on other benchmarks like Ego-MC-Bench would help show whether the gains extend to independently collected mistakes and corrective guidance. It would also be useful to discuss which real mistakes the scripts do not cover.

  2. The method is fairly incremental. Updating a plan as execution progresses is a natural extension of using a fixed plan. This concern does not outweigh the value of the data for me.

  3. The evaluation uses recorded actions and reference dialogue, so it does not directly measure successful task completion under the model's own guidance.

Questions for authors
  1. Can you test the trained system on Ego-MC-Bench?

  2. How were the scripted mistakes selected, and how well do they cover mistakes that users make naturally?

  3. With the trained interaction model fixed, how does performance change when the planner is used without fine-tuning? This would help clarify what planner training adds.

  4. Since no OOP sessions are used in training, how does the planner learn to generate recovery steps? Does this mainly rely on the pretrained model, or do the training annotations contain corrective guidance even without OOP labels?

  5. Should a task completion rate or success rate be reported?

Reviewer_qczbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies proactive assistance for multistep tasks, including when to intervene and how to guide users after procedural deviations. It introduces EgoProactive, a dataset pairing deviations with recovery guidance, and Pro2Bench, a unified evaluation benchmark.

Critical strengths
  1. The paper addresses a meaningful problem that goes beyond video understanding to timely assistance and recovery guidance.
  2. EgoProactive is a useful contribution, capturing deviations and subsequent recovery within continuous videos across multiple domains.
Critical weaknesses
  1. The FT, no plan setting is not clear. Section 4 describes a model trained without plan inputs, whereas Appendices B.1 and C.2 describe using the same PWR interaction checkpoint with the plan removed at evaluation. These are different experiments. If the latter is used, the limited gains could reflect a train–test mismatch rather than show that training without plans is ineffective. The claim that gains come from planning rather than training data requires a separately trained, matched no-plan baseline.

  2. The ablations do not isolate the benefit of dynamic planning. The plan includes current-step information and visual completion cues, which directly support intervention decisions. It remains unclear whether the gains require revising a full plan or could be obtained through simpler progress tracking or history summaries.

  3. The formatting differs from the official ICLR template. The authors should verify compliance with the ICLR 2027 formatting requirements.

Questions for authors

see weakness above

Reviewer_nAGyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoProactive is the part I most want to see published. A step-level deviation and its recovery in one continuous smart-glasses take, across four domains, is something QICD and Ego-MC-Bench don't offer (cooking only, errors inside one step). The 247 OOP sessions will be useful whatever happens to the method.

Critical strengths

EgoProactive is the part I most want to see published. A step-level deviation and its recovery in one continuous smart-glasses take, across four domains, is something QICD and Ego-MC-Bench don't offer (cooking only, errors inside one step). The 247 OOP sessions will be useful whatever happens to the method.

Critical weaknesses
  1. The main claim, that the gains come from the plan rather than the training, isn't backed by the paper's own control. Sec. 4 says FT-no-plan was trained without a plan, but App. B.1 says it is the PWR checkpoint with the plan removed. The numbers disagree too (Qwen .298 in B.2 vs .248 in Table 2), and that cell is what Sec. 4.1's "all nine comparisons" depends on. The recovery claim has no table at all. The static-vs-dynamic comparison is cited as "Table ??".

  2. The metrics go easy on the two failures that matter most for a proactive assistant. Working back from Tables 12-16, PWR barely changes false interrupts (PWR-Gemma goes from about 4,615 to 4,565 of 9,950 silent points), even though the deployed model decides every 500 ms. Wrong advice can also score as correct. In Fig. 9 the golden tells the user to make the scripted mistake, and PWR repeats it and gets a check mark.

Questions for authors

Was FT-no-plan trained without plans, or is it the PWR checkpoint with the plan removed? If the latter, Table 2 needs a retrained control. On the OOP sessions, how often does each system interrupt at the deviation onset with a correct fix? A false-interruptions-per-minute number on full sessions would also help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.