- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that proactive procedural assistants, which watch a streaming egocentric video and decide when to give the next instruction, are limited by step-only supervision. It introduces a dataset with guidance organised hierarchically into phases, steps and actions: ProactiveCoach-Instruct (10K samples) and ProactiveCoachBench (1.1K samples). The dataset is built from eight egocentric sources by an LLM pipeline (GPT-4.1-mini and Gemini 2.5 Flash). The method, ProactiveCoach, fine-tunes a VLM to emit guidance for all three levels in one output sequence, with hierarchical IDs. A lightweight LLM router picks which level to show the user. Across four Qwen backbones, multi-level training improves step-level results over step-only training by about 7–9 Avg points, and the fine-tuned models beat zero-shot baselines on ProactiveCoachBench and improve on the EgoProactive validation set.
Critical strengths
- A useful problem framing and a substantial resource. Modelling proactive guidance as a phase–step–action hierarchy, with guidance timing at each level and user-adjustable granularity, is a natural and practically motivated extension of step-only benchmarks such as ProAssist, Pro2 Bench and GuideMe. The dataset is larger than prior proactive datasets: 7.2K videos, 1,452 hours, eight sources, split so that no video appears in both training and test.
- Controlled comparison against step-only supervision, repeated across backbones. Tables 3–4 compare multi-level and step-only training on the same data and backbones, and the gain repeats on all four models. Adding either phase or action supervision helps, and using all three levels helps most.
Critical weaknesses
-
The main finding, that hierarchical supervision improves step-level guidance, is confounded by richer ground-truth history at test time.
- Every model is evaluated at fixed anchors with the complete ground-truth dialogue history (§5.1). The multi-level model is evaluated with its own prompt (Fig. 24) and a "three-level GT history" (Table 5). That history includes ground-truth action-level guidance for all actions completed so far. The step-only model and all baselines see only step-level history.
- A step ends exactly when its last child action ends. So at step-level anchors, the multi-level model is effectively told by the annotations which sub-actions are finished. This is precisely the "evidence of what has been completed and what remains" that the paper credits to learned hierarchical understanding.
- The step-level gains in Tables 2–4 may therefore come mainly from privileged test-time information, not from what training taught the model.
- The key controls are missing:
- evaluate the multi-level model with step-only history;
- give the step-only model and the zero-shot baselines the same three-level history;
Without these, the paper's central claim is not supported.
-
The benchmark, metrics and adaptation experiment are weak evidence of real proactive ability, and gains outside the paper's own benchmark are modest.
- LLM-built data, thinly checked. The hierarchy, guidance text and timing are all produced by LLMs. Human validation covers only 50 test samples, with no reported number of reviewers or inter-rater agreement. Action-level timing alignment is the weakest criterion (3.80/5), yet it is the level the method relies on.
- Easy anchors. Response anchors sit exactly at ground-truth boundaries, while silent anchors are drawn from the middle 30–70% of segments. The task thus reduces to telling a boundary from a mid-segment moment at a given timestamp, with no timing tolerance and no near-boundary negatives. PQS gives an always-silent model 50 by construction.
- Router comparison is not informative. The guidance-level adaptation result (Table 6, e.g. 85.54 vs 9.46) compares a fine-tuned model plus router, which simply looks up a level the model already generated, against a non-fine-tuned model prompted in context. It shows nothing about adaptation ability beyond the fine-tuning itself.
- External benchmark. On EgoProactive, the only benchmark not built by the paper's own pipeline, the fine-tuned models (47.1–49.0 Avg) remain below zero-shot Gemini 3 Flash (51.35). sF1 also drops relative to the base model (e.g. 36.48 vs 39.93 for Qwen3-VL-4B). Most of the reported gains are therefore in-domain on a benchmark produced by the same pipeline as the training data.
Please report: a larger human-validated subset with agreement statistics, timing-tolerant streaming evaluation with hard negatives near boundaries, and a fine-tuned non-router baseline for adaptation.
Questions for authors
See Weaknesses
6 comments
Clarifications on Guidance-Level Adaptation and Generalization
W2-3. Guidance-Level Adaptation with a Fine-Tuned Non-Router Baseline. To further demonstrate the effectiveness of our guidance-level adaptation system, we conducted an additional comparison using the same fine-tuned model with and without the router. Without the router, the model directly receives the user's level-change request through in-context prompting. In contrast, our router selects the requested guidance level from the model's multi-level outputs.
As shown in the table below, our router-based system consistently outperforms the fine-tuned non-router baseline across all four guidance-level transitions. These results demonstrate that separating guidance generation from level selection improves adaptation performance beyond fine-tuning alone. We will include this additional comparison and clarify the contribution of the router in the revised manuscript.
| Transition | FT without router | FT with router | ||||||
|---|---|---|---|---|---|---|---|---|
| sF1 ↑ | gF1 ↑ | PQS ↑ | Avg. ↑ | sF1 ↑ | gF1 ↑ | PQS ↑ | Avg. ↑ | |
| Step → Phase | 6.88 | 12.75 | 55.51 | 25.05 | 67.66 | 93.78 | 95.17 | 85.54 |
| Step → Action | 32.80 | 58.49 | 46.99 | 46.09 | 54.14 | 67.95 | 51.69 | 57.93 |
| Phase → Step | 12.55 | 25.73 | 51.31 | 29.86 | 47.89 | 72.93 | 78.86 | 66.56 |
| Action → Step | 28.61 | 51.36 | 54.83 | 44.93 | 59.98 | 82.68 | 78.45 | 73.70 |
W2-4. Generalization to EgoProactive. For EgoProactive, our method outperforms several evaluated proprietary models, demonstrating its competitiveness beyond our own benchmark. Although some models exhibit a decrease in sF1 after fine-tuning, proactive assistance requires not only generating appropriate guidance but also determining when to respond and when to remain silent. The substantial improvements in gF1 and PQS indicate better response–silence decisions and overall guidance performance, leading to higher overall scores. We will clarify these findings in the revised manuscript.
Clarification on Dataset Quality
W2-1. Human Validation and Annotation Quality. Although our dataset construction pipeline is automated, it was carefully developed through iterative quality assessment. At each stage, we manually inspected the generated annotations and repeatedly refined the prompts to improve their quality and consistency. Furthermore, before generating proactive guidance annotations, our hierarchy checking and refinement process iteratively revises intermediate results to ensure a consistent and reliable procedural structure.
To provide a larger human-validated subset and agreement statistics, we retained the original 50 quality review videos and selected 50 additional videos. Three annotators rated all 100 ProactiveCoachBench test videos on four criteria at each guidance level using a 0–5 scale.
The first table shows the mean and variance of the ratings for each criterion at each guidance level. Although video alignment received the lowest ratings among the four criteria, the mean scores remained close to 4.0. These scores indicate that annotators generally considered the guidance to be well aligned with the observed activities and appropriately timed, as illustrated in Figures 14–19 and our anonymous dataset preview.
The second table shows the consistency of ratings among annotators. Across all evaluation criteria and guidance levels, 88.1% of paired ratings differed by no more than one point, indicating that annotators generally provided consistent assessments. Together, these results suggest that our iterative quality assessment process produced high-quality annotations, whose quality and consistency were further supported by human evaluation. We will include these additional validation results in the revised manuscript.
| Criterion | Phase mean | Phase var. | Step mean | Step var. | Action mean | Action var. |
|---|---|---|---|---|---|---|
| Video alignment | 4.17 | 0.98 | 4.07 | 1.16 | 4.09 | 1.23 |
| Hierarchical coverage | 4.45 | 0.56 | 4.50 | 0.54 | 4.47 | 0.73 |
| Goal relevance | 4.63 | 0.49 | 4.48 | 0.64 | 4.37 | 0.73 |
| Guidance quality | 4.52 | 0.44 | 4.37 | 0.50 | 4.47 | 0.57 |
| Criterion | Phase | Step | Action |
|---|---|---|---|
| Video alignment | 86.7% | 80.3% | 81.3% |
| Hierarchical coverage | 88.0% | 86.0% | 83.3% |
| Goal relevance | 95.0% | 90.7% | 92.0% |
| Guidance quality | 92.3% | 92.0% | 90.0% |
Clarifications on Hierarchical Supervision, Annotation Quality and Evaluation Analysis
We sincerely thank the reviewer for carefully reviewing our paper and providing constructive and valuable feedback. We are pleased that your feedback will help us further improve the quality and completeness of our paper. We hope that our responses below address your concerns. If any concerns remain unresolved, we would greatly appreciate it if you could let us know what is still insufficient so that we can address it further.
W1. Effectiveness of Hierarchical Supervision. To further demonstrate the effectiveness of hierarchical supervision, we conducted an additional ablation study with six experimental configurations using Qwen3.5-4B. Specifically, we considered three training strategies: (1) no fine-tuning, (2) step-only SFT, and (3) multi-level SFT. We also considered two types of in-context dialogue history: (1) step-only GT history and (2) multi-level GT history.
The table below reveals an interesting finding. Step-only SFT performs better with step-only GT history than with multi-level GT history, whereas multi-level SFT achieves substantially better performance with multi-level GT history than with step-only GT history. These results suggest that richer dialogue history alone is insufficient to improve performance, highlighting the importance of learning to effectively utilize hierarchical information during training.
To avoid any potential confusion, we would like to clarify that both models in Table 3 were evaluated with the same multi-level GT history to ensure identical inference-time context and isolate the effect of the training strategy. We will make this evaluation setting clearer and incorporate the additional analysis into the revised manuscript.
W2-1. Human Validation and Annotation Quality. We are currently conducting a larger-scale human quality review to further validate our dataset. We will update the results once the evaluation is complete.
W2-2. Timing-Tolerant Streaming Evaluation and Hard Negatives. We sampled silent anchors from the middle 30–70% of procedural segments to avoid temporal ambiguity near guidance boundaries. To further evaluate our model under more challenging conditions, we conducted an additional experiment by sampling silent anchors from different positions within procedural segments.
As silent anchors move toward the end of a step, the model increasingly struggles to remain silent. This may be because the model tends to remain silent shortly after providing guidance, but is more likely to provide unnecessary guidance toward the end of a step. These results demonstrate that near-boundary silent anchors present a more challenging evaluation setting.
Regarding PQS, an always-silent model receives a score of 50 under our balanced evaluation setting. However, this does not indicate effective proactive assistance, as the model fails to provide guidance when needed. Thus, PQS should be interpreted alongside sF1 and gF1 to provide a more comprehensive evaluation of proactive guidance performance.