Improving Proactive AI Assistance with Hierarchical Procedural Understanding

Proactive procedural assistance aims to continuously monitor user progress on procedural tasks like cooking and provide guidance on the next step when the current step is completed. Existing approaches mainly construct step-level guidance datasets. However, step-level supervision alone provides limited information for tracking task progress and completion in a streaming setting. A step consists of multiple lower-level actions and also belongs to a higher-level phase. In understanding the progress of the current step, lower-level actions provide evidence of what has been completed and what remains, while higher-level phase context can clarify the purpose and expected outcome of the current step. Also, modeling of the phase–step–action hierarchy allows the generation of guidance at different granularities according to the user's needs. In this paper, we introduce *ProactiveCoach-Instruct*, a training dataset with verbal guidance hierarchically organized by phases, steps, and actions, and *ProactiveCoachBench*, a benchmark for evaluating multi-level proactive guidance and guidance-level adaptation. Using ProactiveCoach-Instruct, we propose *ProactiveCoach*, a method that jointly learns proactive guidance across all three levels in a single vision-language model. Our method consistently improves proactive assistance performance compared with step-only training. We further build an adaptive guidance system with a lightweight router, allowing the guidance level to change according to the user's request. An anonymous dataset preview is available at [https://anonymous.4open.science/r/dataset-review-7c3e/](https://anonymous.4open.science/r/dataset-review-7c3e/).