LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos

Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a VLM to translate visual observations into text embeddings, predicts actions, and extracts corresponding embeddings. Finally, a diffusion model conditioned on the text embeddings of the start and goal actions and augmented text embeddings of intermediated actions generates the plan. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margins, demonstrating the significant advantage of language-aware planning.