acceptodds
Under review as a conference paper at ICLR 2027

Improving Proactive AI Assistance with Hierarchical Procedural Understanding

Abstract

Proactive procedural assistance aims to continuously monitor user progress on procedural tasks like cooking and provide guidance on the next step when the current step is completed. Existing approaches mainly construct step-level guidance datasets. However, step-level supervision alone provides limited information for tracking task progress and completion in a streaming setting. A step consists of multiple lower-level actions and also belongs to a higher-level phase. In understanding the progress of the current step, lower-level actions provide evidence of what has been completed and what remains, while higher-level phase context can clarify the purpose and expected outcome of the current step. Also, modeling of the phase–step–action hierarchy allows the generation of guidance at different granularities according to the user's needs. In this paper, we introduce *ProactiveCoach-Instruct*, a training dataset with verbal guidance hierarchically organized by phases, steps, and actions, and *ProactiveCoachBench*, a benchmark for evaluating multi-level proactive guidance and guidance-level adaptation. Using ProactiveCoach-Instruct, we propose *ProactiveCoach*, a method that jointly learns proactive guidance across all three levels in a single vision-language model. Our method consistently improves proactive assistance performance compared with step-only training. We further build an adaptive guidance system with a lightweight router, allowing the guidance level to change according to the user's request. An anonymous dataset preview is available at [https://anonymous.4open.science/r/dataset-review-7c3e/](https://anonymous.4open.science/r/dataset-review-7c3e/).

Primary area:
datasets and benchmarks
Keywords:
Proactive AI Assistance, Proactive Procedural Assistance, Hierarchical Procedural Understanding, Hierarchically Structured Guidance, Guidance-Level Adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HM7MOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that proactive procedural assistants, which watch a streaming egocentric video and decide when to give the next instruction, are limited by step-only supervision. It introduces a dataset with guidance organised hierarchically into phases, steps and actions: ProactiveCoach-Instruct (10K samples) and ProactiveCoachBench (1.1K samples). The dataset is built from eight egocentric sources by an LLM pipeline (GPT-4.1-mini and Gemini 2.5 Flash). The method, ProactiveCoach, fine-tunes a VLM to emit guidance for all three levels in one output sequence, with hierarchical IDs. A lightweight LLM router picks which level to show the user. Across four Qwen backbones, multi-level training improves step-level results over step-only training by about 7–9 Avg points, and the fine-tuned models beat zero-shot baselines on ProactiveCoachBench and improve on the EgoProactive validation set.

Critical strengths
  1. A useful problem framing and a substantial resource. Modelling proactive guidance as a phase–step–action hierarchy, with guidance timing at each level and user-adjustable granularity, is a natural and practically motivated extension of step-only benchmarks such as ProAssist, Pro2 Bench and GuideMe. The dataset is larger than prior proactive datasets: 7.2K videos, 1,452 hours, eight sources, split so that no video appears in both training and test.
  2. Controlled comparison against step-only supervision, repeated across backbones. Tables 3–4 compare multi-level and step-only training on the same data and backbones, and the gain repeats on all four models. Adding either phase or action supervision helps, and using all three levels helps most.
Critical weaknesses
  1. The main finding, that hierarchical supervision improves step-level guidance, is confounded by richer ground-truth history at test time.

    • Every model is evaluated at fixed anchors with the complete ground-truth dialogue history (§5.1). The multi-level model is evaluated with its own prompt (Fig. 24) and a "three-level GT history" (Table 5). That history includes ground-truth action-level guidance for all actions completed so far. The step-only model and all baselines see only step-level history.
    • A step ends exactly when its last child action ends. So at step-level anchors, the multi-level model is effectively told by the annotations which sub-actions are finished. This is precisely the "evidence of what has been completed and what remains" that the paper credits to learned hierarchical understanding.
    • The step-level gains in Tables 2–4 may therefore come mainly from privileged test-time information, not from what training taught the model.
    • The key controls are missing:
      • evaluate the multi-level model with step-only history;
      • give the step-only model and the zero-shot baselines the same three-level history;

    Without these, the paper's central claim is not supported.

  2. The benchmark, metrics and adaptation experiment are weak evidence of real proactive ability, and gains outside the paper's own benchmark are modest.

    • LLM-built data, thinly checked. The hierarchy, guidance text and timing are all produced by LLMs. Human validation covers only 50 test samples, with no reported number of reviewers or inter-rater agreement. Action-level timing alignment is the weakest criterion (3.80/5), yet it is the level the method relies on.
    • Easy anchors. Response anchors sit exactly at ground-truth boundaries, while silent anchors are drawn from the middle 30–70% of segments. The task thus reduces to telling a boundary from a mid-segment moment at a given timestamp, with no timing tolerance and no near-boundary negatives. PQS gives an always-silent model 50 by construction.
    • Router comparison is not informative. The guidance-level adaptation result (Table 6, e.g. 85.54 vs 9.46) compares a fine-tuned model plus router, which simply looks up a level the model already generated, against a non-fine-tuned model prompted in context. It shows nothing about adaptation ability beyond the fine-tuning itself.
    • External benchmark. On EgoProactive, the only benchmark not built by the paper's own pipeline, the fine-tuned models (47.1–49.0 Avg) remain below zero-shot Gemini 3 Flash (51.35). sF1 also drops relative to the base model (e.g. 36.48 vs 39.93 for Qwen3-VL-4B). Most of the reported gains are therefore in-domain on a benchmark produced by the same pipeline as the training data.

Please report: a larger human-validated subset with agreement statistics, timing-tolerant streaming evaluation with hard negatives near boundaries, and a fine-tuned non-router baseline for adaptation.

Questions for authors

See Weaknesses

6 comments
AuthorsOpenReview ↗

Clarifications on Hierarchical Supervision, Annotation Quality and Evaluation Analysis

We sincerely thank the reviewer for carefully reviewing our paper and providing constructive and valuable feedback. We are pleased that your feedback will help us further improve the quality and completeness of our paper. We hope that our responses below address your concerns. If any concerns remain unresolved, we would greatly appreciate it if you could let us know what is still insufficient so that we can address it further.

W1. Effectiveness of Hierarchical Supervision. To further demonstrate the effectiveness of hierarchical supervision, we conducted an additional ablation study with six experimental configurations using Qwen3.5-4B. Specifically, we considered three training strategies: (1) no fine-tuning, (2) step-only SFT, and (3) multi-level SFT. We also considered two types of in-context dialogue history: (1) step-only GT history and (2) multi-level GT history.

The table below reveals an interesting finding. Step-only SFT performs better with step-only GT history than with multi-level GT history, whereas multi-level SFT achieves substantially better performance with multi-level GT history than with step-only GT history. These results suggest that richer dialogue history alone is insufficient to improve performance, highlighting the importance of learning to effectively utilize hierarchical information during training.

To avoid any potential confusion, we would like to clarify that both models in Table 3 were evaluated with the same multi-level GT history to ensure identical inference-time context and isolate the effect of the training strategy. We will make this evaluation setting clearer and incorporate the additional analysis into the revised manuscript.

TrainingStep-only GT historyMulti-level GT history
sF1 ↑gF1 ↑PQS ↑Avg. ↑sF1 ↑gF1 ↑PQS ↑Avg. ↑
Zero-shot22.3450.9938.3537.2320.8649.1039.9036.62
Step-only SFT45.1366.7047.3453.0641.0164.5344.6250.06
Multi-level SFT (Ours)44.6662.1441.0749.2949.3572.2756.2759.30

W2-1. Human Validation and Annotation Quality. We are currently conducting a larger-scale human quality review to further validate our dataset. We will update the results once the evaluation is complete.

W2-2. Timing-Tolerant Streaming Evaluation and Hard Negatives. We sampled silent anchors from the middle 30–70% of procedural segments to avoid temporal ambiguity near guidance boundaries. To further evaluate our model under more challenging conditions, we conducted an additional experiment by sampling silent anchors from different positions within procedural segments.

As silent anchors move toward the end of a step, the model increasingly struggles to remain silent. This may be because the model tends to remain silent shortly after providing guidance, but is more likely to provide unnecessary guidance toward the end of a step. These results demonstrate that near-boundary silent anchors present a more challenging evaluation setting.

Regarding PQS, an always-silent model receives a score of 50 under our balanced evaluation setting. However, this does not indicate effective proactive assistance, as the model fails to provide guidance when needed. Thus, PQS should be interpreted alongside sF1 and gF1 to provide a more comprehensive evaluation of proactive guidance performance.

silent anchor position within procedural stepsF1gF1PQSAvg.
0–10%52.9578.5462.9164.80
10–30%52.1076.7260.9863.27
30–70%49.3572.2756.2759.30
70–90%47.3966.7050.3854.82
90–100%46.8563.8547.4252.71
AuthorsOpenReview ↗

Clarifications on Guidance-Level Adaptation and Generalization

W2-3. Guidance-Level Adaptation with a Fine-Tuned Non-Router Baseline. To further demonstrate the effectiveness of our guidance-level adaptation system, we conducted an additional comparison using the same fine-tuned model with and without the router. Without the router, the model directly receives the user's level-change request through in-context prompting. In contrast, our router selects the requested guidance level from the model's multi-level outputs.

As shown in the table below, our router-based system consistently outperforms the fine-tuned non-router baseline across all four guidance-level transitions. These results demonstrate that separating guidance generation from level selection improves adaptation performance beyond fine-tuning alone. We will include this additional comparison and clarify the contribution of the router in the revised manuscript.

TransitionFT without routerFT with router
sF1 ↑gF1 ↑PQS ↑Avg. ↑sF1 ↑gF1 ↑PQS ↑Avg. ↑
Step → Phase6.8812.7555.5125.0567.6693.7895.1785.54
Step → Action32.8058.4946.9946.0954.1467.9551.6957.93
Phase → Step12.5525.7351.3129.8647.8972.9378.8666.56
Action → Step28.6151.3654.8344.9359.9882.6878.4573.70

W2-4. Generalization to EgoProactive. For EgoProactive, our method outperforms several evaluated proprietary models, demonstrating its competitiveness beyond our own benchmark. Although some models exhibit a decrease in sF1 after fine-tuning, proactive assistance requires not only generating appropriate guidance but also determining when to respond and when to remain silent. The substantial improvements in gF1 and PQS indicate better response–silence decisions and overall guidance performance, leading to higher overall scores. We will clarify these findings in the revised manuscript.

AuthorsOpenReview ↗

Clarification on Dataset Quality

W2-1. Human Validation and Annotation Quality. Although our dataset construction pipeline is automated, it was carefully developed through iterative quality assessment. At each stage, we manually inspected the generated annotations and repeatedly refined the prompts to improve their quality and consistency. Furthermore, before generating proactive guidance annotations, our hierarchy checking and refinement process iteratively revises intermediate results to ensure a consistent and reliable procedural structure.

To provide a larger human-validated subset and agreement statistics, we retained the original 50 quality review videos and selected 50 additional videos. Three annotators rated all 100 ProactiveCoachBench test videos on four criteria at each guidance level using a 0–5 scale.

The first table shows the mean and variance of the ratings for each criterion at each guidance level. Although video alignment received the lowest ratings among the four criteria, the mean scores remained close to 4.0. These scores indicate that annotators generally considered the guidance to be well aligned with the observed activities and appropriately timed, as illustrated in Figures 14–19 and our anonymous dataset preview.

The second table shows the consistency of ratings among annotators. Across all evaluation criteria and guidance levels, 88.1% of paired ratings differed by no more than one point, indicating that annotators generally provided consistent assessments. Together, these results suggest that our iterative quality assessment process produced high-quality annotations, whose quality and consistency were further supported by human evaluation. We will include these additional validation results in the revised manuscript.

CriterionPhase meanPhase var.Step meanStep var.Action meanAction var.
Video alignment4.170.984.071.164.091.23
Hierarchical coverage4.450.564.500.544.470.73
Goal relevance4.630.494.480.644.370.73
Guidance quality4.520.444.370.504.470.57
CriterionPhaseStepAction
Video alignment86.7%80.3%81.3%
Hierarchical coverage88.0%86.0%83.3%
Goal relevance95.0%90.7%92.0%
Guidance quality92.3%92.0%90.0%
Reviewer_v62TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ProactiveCoach-Instruct and ProactiveCoachBench, a training dataset and benchmark for hierarchical proactive procedural assistance at the phase, step, and action levels. It further proposes ProactiveCoach, which jointly learns multi-level guidance in a single VLM, and an adaptive guidance system that adjusts guidance granularity based on user requests. Experiments across multiple backbone models and benchmarks demonstrate consistent improvements over step-only supervision and existing proactive assistance methods.

Critical strengths

A. The proposed hierarchical guidance framework is an interesting approach that leverages complementary information across phase, step, and action levels to improve procedural assistance. The Guidance-Level Adaptation mechanism further enables flexible guidance at different granularities according to user needs, making the framework more practical for real-world applications.

B. The proposed ProactiveCoach-Instruct and ProactiveCoachBench address the lack of hierarchical guidance data for proactive procedural assistance. They provide valuable resources for training and evaluating future models, supporting further research in this direction.

Critical weaknesses

A. Limited Methodological Novelty. Although the proposed SFT approach demonstrates effectiveness, it primarily relies on conventional supervised fine-tuning with hierarchically structured guidance data, without introducing substantial methodological innovation. This limits the technical contribution of the proposed learning framework.

B. Lack of Inference Efficiency Analysis. Since proactive assistance operates on streaming video and requires timely responses, inference speed and latency are critical for practical deployment. However, the paper lacks an analysis of computational overhead and response latency, making it difficult to assess the feasibility of the proposed framework in real-time applications.

Questions for authors

A. How are behaviors categorized into actions, steps, and phases? Could the authors provide clearer definitions and criteria for distinguishing these three levels?

B. Line 265: "HUPPA" appears to be a typo. Please correct it.

C. Is guidance generated at all three levels for every video frame, or only at specific time intervals?

D. Table 5 appears to be redundant, as the zero-shot results are already reported in Table 2.

E. Could the authors provide experimental results and discussion on inference speed and response latency? Given the streaming nature of proactive assistance, these factors are important for evaluating the practicality of the proposed framework.

5 comments
AuthorsOpenReview ↗

Clarifications on Main Contributions, Inference Efficiency and Procedural Definitions.

We sincerely thank the reviewer for carefully reviewing our paper and providing constructive and valuable feedback. We are pleased that your feedback will help us further improve the quality and completeness of our paper. We hope that our responses below address your concerns. If any concerns remain unresolved, we would greatly appreciate it if you could let us know what is still insufficient so that we can address it further.

W-A. Main Contributions. Our work aims to advance proactive AI assistance by introducing a unified framework for learning and evaluating hierarchical procedural guidance. Our work makes three main contributions. First, we introduce ProactiveCoach-Instruct and ProactiveCoachBench, providing training and evaluation data with hierarchically structured guidance and guidance timing across different procedural levels. Second, we propose a hierarchical guidance learning framework that enables VLMs to jointly learn fine-grained task progress and broader procedural context, improving their ability to determine when and what guidance to provide. Third, we develop an adaptive guidance system that adjusts guidance granularity according to user requests without additional fine-tuning of the guide model.

These contributions provide a foundation for studying and improving proactive AI assistance through hierarchical procedural understanding, rather than focusing solely on developing a new optimization algorithm. We will further clarify these contributions in the revised manuscript.

W-B, Q-E. Inference Efficiency and Practicality. To evaluate the practical feasibility of our approach, we conducted an additional inference latency analysis using 100 randomly selected test samples under a practical streaming setting.

As shown in the table below, both silence decisions and guidance responses take at most approximately 1 second. Furthermore, the lightweight 1B router, which does not process video inputs, introduces only about 0.02 seconds of additional latency. These results demonstrate the practical feasibility of our approach for streaming proactive assistance, enabling our system to adapt guidance levels while maintaining timely responses. We will include this analysis in the revised manuscript.

Model | LatencySilent AnchorGuidance AnchorRouter
Qwen3-VL-4B0.433 s0.654 s0.020 s
Qwen3-VL-8B0.448 s0.730 s0.020 s
Qwen3.5-4B0.534 s1.044 s0.020 s
Qwen3.5-9B0.596 s1.046 s0.020 s

Q-A. Definitions of Action, Step, and Phase. As described in Section 3.1, we distinguish three procedural levels based on their procedural scope. A phase represents an intermediate goal within the overall task, consisting of multiple steps that collectively achieve this goal. A step represents a specific procedural task within a phase, consisting of multiple actions required to complete it. An action is the finest-level unit, representing an observable event that can be guided independently.

During dataset construction, we provided level-specific definitions and representative examples to guide the generation of hierarchical annotations. We further employed an iterative hierarchy checking and refinement process to verify the relationships between procedural levels and eliminate redundant structures, ensuring a consistent and coherent hierarchy. We will further clarify these definitions and provide more explicit examples in the revised manuscript.

Q-B. Typographical Error. Thank you for pointing out this typo. "HUPPA" was an earlier name of our method, and we will correct it to "ProactiveCoach" in the revised manuscript.

AuthorsOpenReview ↗

Clarification on Evaluation Settings

Q-C. Guidance Generation Frequency. Our model does not generate guidance at all three levels for every video frame. As described in Section 5.1, videos are sampled at 1 FPS and processed in 2-second chunks, with the model deciding whether to provide guidance at each chunk.

At each chunk, our model determines whether new guidance is needed at the phase, step, or action level, and can generate guidance at multiple levels simultaneously. For example, if the model determines that both step-level and action-level guidance are needed, it generates guidance at these two levels without generating phase-level guidance. If no new guidance is needed at any level, it outputs <silent>. We will clarify this process in the revised manuscript.

Q-D. Differences Between Tables 2 and 5. Although both tables provide GT guidance history across all three levels, they differ in their evaluation settings. Table 2 evaluates models at individual guidance levels, whereas Table 5 evaluates their ability to jointly generate multi-level guidance.

The results in Table 5 show that zero-shot prompting alone is insufficient to achieve reliable multi-level guidance, highlighting the effectiveness of hierarchical guidance learning. We will clarify the differences between Tables 2 and 5 in the revised manuscript.

Additional Analysis on Hierarchical Supervision. We would also like to share an additional finding from our rebuttal experiments. To further investigate the effectiveness of hierarchical supervision, we conducted an ablation study with six configurations using Qwen3.5-4B. We considered three training strategies (no fine-tuning, step-only SFT, and multi-level SFT) and two types of in-context dialogue history (step-only and multi-level GT history).

Interestingly, step-only SFT performs better with step-only GT history, whereas multi-level SFT achieves substantially better performance with multi-level GT history. These results suggest that richer dialogue history alone is insufficient to improve performance, highlighting the importance of learning to effectively utilize hierarchical information during training. We will incorporate this additional analysis into the revised manuscript.

TrainingStep-only GT historyMulti-level GT history
sF1 ↑gF1 ↑PQS ↑Avg. ↑sF1 ↑gF1 ↑PQS ↑Avg. ↑
Zero-shot22.3450.9938.3537.2320.8649.1039.9036.62
Step-only SFT45.1366.7047.3453.0641.0164.5344.6250.06
Multi-level SFT (Ours)44.6662.1441.0749.2949.3572.2756.2759.30
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer v62T, Thank you for your time. We have completed additional experiments on inference efficiency and hierarchical supervision, along with clarifications on our main contributions and evaluation settings. Please feel free to review them in the shared comments. We are also more than happy to discuss any remaining concerns or questions you might have.

Reviewer_4FAnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors introduce ProactiveCoach-Instruct and ProactiveCoachBench, designed to train & evaluate proactive procedural assistance across three hierarchical levels: phase, step, and action. The dataset comprises 11K+ samples from egocentric videos across eight source datasets. The authors propose ProactiveCoach, jointly learning multi-level guidance in a single VLM, and a lightweight router for adapting guidance granularity to user requests.

Critical strengths
  • Hierarchical guidance design. Existing proactive procedural assistance primarily focuses on step-level guidance, whereas the proposed work explicitly models phase–step–action relationships -- which is a useful extension for tracking task progress & providing different levels of detail.

  • Broad dataset coverage. The proposed benchmark covers multiple procedural domains sourced from diverse datasets, including DIY, cooking, household activities etc. with guidance content & timing annotations at all three levels.

  • Consistent and model-agnoisic gains over step-only supervision across four backbones, validating the usefulness of the additional supervision proposed. The phase–step and step–action empirical ablations further support the claim.

Critical weaknesses
  • While the main novelty of the claim is on the model architecture that adds hierarchy, the evaluation design does not fully isolate its component. Multi-level training adds more supervision targets, while multi-level history can provide additional progress cues, which can all lead to improved input context. A more careful evaluation design / ablation studies would help distinguish hierarchical understanding from simply having richer supervision signals.

  • Limited end-to-end streaming evaluation, in contrast to the nature of the proposed scenario / tasks. Models are evaluated at predefined anchors using the latest 10 seconds of video & ground-truth dialog history, rather than their own generated history at inference time. Studying the accumulated errors and missed guidance would be critical - thus recommending a more careful evaluation design.

  • Some missing literature that could be good to add to comparisons, especially on procedural assistance:

[1] Kundu et al., "Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance", 2026.

Questions for authors

Please see above

4 comments
AuthorsOpenReview ↗

Clarifications on Hierarchical Supervision and Evaluation Protocol

We sincerely thank the reviewer for carefully reviewing our paper and providing constructive and valuable feedback. We are pleased that your feedback will help us further improve the quality and completeness of our paper. We hope that our responses below address your concerns. If any concerns remain unresolved, we would greatly appreciate it if you could let us know what is still insufficient so that we can address it further.

W1. Effectiveness of Hierarchical Supervision. To further demonstrate the effectiveness of hierarchical supervision, we conducted an additional ablation study with six experimental configurations using Qwen3.5-4B. Specifically, we considered three training strategies: (1) no fine-tuning, (2) step-only SFT, and (3) multi-level SFT. We also considered two types of in-context dialogue history: (1) step-only GT history and (2) multi-level GT history. The table below reveals an interesting finding. Step-only SFT performs better with step-only GT history than with multi-level GT history, whereas multi-level SFT achieves substantially better performance with multi-level GT history than with step-only GT history. These results suggest that richer dialogue history alone is insufficient to improve performance, highlighting the importance of learning to effectively utilize hierarchical information during training.

To avoid any potential confusion, we would like to clarify that both models in Table 3 were evaluated with the same multi-level GT history to ensure identical inference-time context and isolate the effect of the training strategy. We will make this evaluation setting clearer and incorporate the additional analysis into the revised manuscript.

TrainingStep-only GT historyMulti-level GT history
sF1 ↑gF1 ↑PQS ↑Avg. ↑sF1 ↑gF1 ↑PQS ↑Avg. ↑
Zero-shot22.3450.9938.3537.2320.8649.1039.9036.62
Step-only SFT45.1366.7047.3453.0641.0164.5344.6250.06
Multi-level SFT (Ours)44.6662.1441.0749.2949.3572.2756.2759.30

W2. Evaluation Protocol and Error Accumulation. Our evaluation primarily focuses on the model's ability to determine when to provide guidance and what guidance to provide at the current moment. Since incorrect or missed guidance in previous predictions can affect subsequent decisions, we use ground-truth (GT) dialogue history. This protocol allows us to evaluate the model's guidance capability at the current moment without the influence of accumulated errors. It is also adopted in previous studies, such as GuideMe [1] and PWR [2].

Nevertheless, we also conducted an additional experiment comparing GT history and model-generated history to investigate the effect of error accumulation. In the generated-history setting, previous model predictions were accumulated and used as the dialogue history for subsequent decisions. We randomly sampled 100 video sequences for this evaluation, as generating and accumulating predictions across preceding timesteps requires substantially greater computational cost than evaluating individual anchor points with GT history.

As shown in the table below, replacing GT history with generated history consistently degrades performance. In particular, the substantial decreases in sRecall and Response F1 indicate that accumulated errors lead the model to remain silent more frequently, causing it to miss necessary guidance. This indicates that early prediction errors compound over time, leading to increasingly unreliable dialogue history and affecting subsequent guidance decisions. These results support our choice of GT history for the primary evaluation, as it allows us to assess the model's guidance capability at the current moment without the influence of accumulated errors. We will include this additional analysis in our revised manuscript.

History sourceMain metricsSub Metrics
sF1 ↑gF1 ↑PQS ↑Avg. ↑sPrecisionsRecallResponse F1Silence F1
GT history49.9172.9355.9159.5860.6144.3070.2275.75
Generated history18.0338.9848.6535.2245.8612.5323.0066.06
AuthorsOpenReview ↗

Clarification on Related Work

W3. Comparison with Plan, Watch, Recover (PWR). We agree that PWR [2] would be a valuable addition to the experimental comparisons for proactive procedural assistance. As mentioned in lines 326–327, we selected guidance assistant methods with publicly available model weights and code to ensure reproducible comparisons. Although we attempted to reproduce PWR using its publicly available resources, we could not obtain a sufficiently reliable implementation for a fair comparison. Therefore, we were unable to include PWR as a direct baseline in our experiments.

Furthermore, PWR primarily focuses on tracking procedural states and recovering from out-of-plan deviations through explicit plan updates. In contrast, our work investigates how hierarchical procedural understanding improves proactive assistance by enabling better task progress tracking, more appropriate guidance timing and content, and flexible guidance granularity according to user needs. We will further clarify these differences and expand our discussion of PWR in the revised manuscript.

[1] “GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video”, ECCV 2026

[2] “Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance”, Preprint 2026

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 4FAn, Thank you for your time. We have completed additional experiments and clarifications on hierarchical supervision and the evaluation protocol, along with further discussion of relevant prior work. Please feel free to review them in the shared comments. We are also more than happy to discuss any remaining concerns or questions you might have.

We would like to kindly remind you that less than 24 hours remain in the Author–Reviewer Discussion period. We sincerely appreciate your valuable feedback, which has been helpful for improving our paper.

Comments on the paper

1 comment

Dear Reviewers,

Authors posted their responses to your queries. Please read these messages and post your replies so that we get a chance to clarify points which are possibly still unclear and may require further discussion.

Thank you for supporting the ICLR revision process and contributing to discussion.

Best, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.