acceptodds
Withdrawn from ICLR 2027

LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a VLM to translate visual observations into text embeddings, predicts actions, and extracts corresponding embeddings. Finally, a diffusion model conditioned on the text embeddings of the start and goal actions and augmented text embeddings of intermediated actions generates the plan. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margins, demonstrating the significant advantage of language-aware planning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
procedure planning, VLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zHvEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LAP converts endpoint video observations into action descriptions using a fine-tuned multimodal model, matches descriptions to action labels, and embeds canonical text descriptions. A small language model proposes intermediate descriptions; their embeddings select a training-derived cluster that conditions a diffusion planner. Predictions are combined with a base diffusion planner. Experiments on CrossTask, COIN, and NIV report large procedure-success improvements.

Critical strengths
  1. The semantic endpoint bottleneck is a useful way to connect video understanding with procedure planning. Evaluating three datasets and longer action sequences provides meaningful scope, with substantial reported gains, such as CrossTask length-three success of 54.3% versus 33.9% for the strongest listed comparator.

  2. The paper reruns baselines on the curated CrossTask split and reports component comparisons, including visual-only and conditioned/base planning variants. The clustering description explicitly restricts fitting to training data, which is appropriate.

Critical weaknesses
  1. Two central procedural ambiguities prevent verification of soundness. Equations 2 and 3 minimize the true-mode classification BCE through the VLM parameters, which encourages teacher-forcing/free-running distinguishability; the motivation instead suggests aligning the regimes. No adversarial sign, gradient reversal, or alternating generator update is specified. Separately, Section 3.3 compares generated descriptions with “ground truth descriptions of actions.” It is unclear whether these are a fixed dictionary of all action classes or the annotation of the evaluated clip. The former is valid; the latter would introduce unavailable endpoint information. I am not asserting that the implementation uses an oracle, but the manuscript must define an annotation-free inference procedure.

  2. The large visual-only gap does not isolate language semantics from changes in endpoint recognition, model capacity, and supervision. LAP uses a large fine-tuned VLM and a discrete action-label bottleneck, whereas LAP-vo uses a different visual extractor. A stronger control should retain the same endpoint predictor and training data, comparing visual features, class-label embeddings, and canonical text embeddings. Otherwise, the main gain could reflect improved endpoint classification rather than the proposed language-aware planning mechanism.

Questions for authors
  1. What exact gradients update the VLM and regularizer? Please give the signed objectives, detach operations, and update schedule, explaining how they align rather than separate the two decoding regimes.

  2. Please give complete test-time pseudocode: which descriptions are available for matching, how candidates are selected, and whether every planner input endpoint action is predicted without access to clip-specific labels.

  3. Can you repeat the representation comparison with the same VLM endpoint predictions and supervision, using visual features, class-label embeddings, and text embeddings? How much of the reported gain remains?

Reviewer_EX6HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LAP, a language-aware framework for procedure planning in instructional videos. LAP first fine-tunes a VLM to translate start and goal visual observations into language descriptions and corresponding action representations. It then employs diffusion-based planning conditioned on the predicted start/goal action embeddings, together with an additional language-conditioned branch based on LLM-generated intermediate-action descriptions. The two planning branches are ensembled at inference. Experiments on CrossTask, COIN, and NIV report substantial improvements over existing procedure-planning methods across multiple metrics and planning horizons.

Critical strengths
  1. The paper targets an important limitation of visual-only procedure planning. The observation that visually similar states may correspond to different actions is well motivated, and introducing language as an intermediate semantic representation is a reasonable direction.

  2. The empirical improvements are substantial and consistent. LAP reports large gains over prior methods on CrossTask, COIN, and NIV and across different planning horizons, suggesting that the proposed pipeline contains useful information for the task.

Critical weaknesses
  1. The central claim that language representations are superior to visual representations is not convincingly isolated experimentally. In the main language-versus-vision ablation, LAP uses language representations derived through a large fine-tuned VLM, whereas LAP-vo uses visual features extracted by the older Miech et al. visual encoder. Therefore, the very large improvement may reflect differences in pretrained backbone quality, model capacity, fine-tuning, or supervision rather than the modality change itself. A fairer comparison should use the same modern VLM/vision backbone and change only whether the planner receives its visual or textual representation.

  2. The inference protocol for converting generated descriptions into action labels is unclear and potentially problematic. Section 3.3 states that generated descriptions are compared with “ground truth descriptions of actions” using ROUGE-1 to determine the predicted action. It is unclear exactly what ground-truth information is available during test-time inference. If sample-specific ground-truth descriptions are used, this would constitute information leakage. If the method instead matches against a fixed vocabulary of class descriptions known from training, this should be stated explicitly and the same semantic information should ideally be made available to competing methods or an appropriate baseline.

Questions for authors
  1. During inference, exactly what texts are used to compute the ROUGE-1 score in the Predictor module? Are generated descriptions compared against a fixed set of class-level action descriptions constructed from the training set, or against the ground-truth description associated with the current test sample? Please clarify the complete inference procedure and confirm that no test-time annotation is used.

  2. Can the authors provide a controlled comparison between visual and language representations using the same pretrained backbone? For example, using InternVL3.5 visual embeddings directly as the conditioning representation and comparing them with text embeddings generated by the same model would help determine whether the improvement truly comes from language rather than from using a stronger pretrained representation.

Reviewer_8nkWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

(This review would like be up-front that this is the 3d time reviewing this manuscript. So, some of the previous review will be pasted here. Will highlight revised components if any).

This paper proposes LAP, a language-aware framework for procedure planning in instructional videos. The core idea is to replace visually ambiguous start/goal observations with more discriminative language-grounded representations by first using an LLM to elaborate action descriptions, then fine-tuning a VLM to map visual clips into that textual space, and finally using a diffusion model to generate intermediate action sequences conditioned on the predicted start/goal text embeddings. Across CrossTask, Coin, and NIV, the paper reports strong gains over prior planning baselines, supporting the claim that language can provide a cleaner and more separable latent space than raw visual features for long-horizon procedural reasoning.

Critical strengths
  • The paper is built on a strong and practically important insight: many failures in procedure planning come from weak or ambiguous state representations, not only from poor sequence modeling. The motivating example in Figure 1 makes this intuitive: visually similar frames can correspond to different actions, whereas text descriptions are often more separable

  • Compared with the earlier concerns, the current version does a better job isolating why the method works. In particular, the new ablations on language enhancement and text-vs-visual planning are useful: LAP improves start/goal action prediction over the non-enhanced version, and LAP clearly outperforms the visual-only planning variant, especially on Coin and NIV. That materially strengthens the main claim.

  • The experimental gains are large enough to be hard to dismiss as marginal engineering. On CrossTask, Coin, and NIV, LAP improves over strong baselines by substantial margins on SR and mAcc, and the gap to LAP-vo supports the claim that language is doing real work rather than serving as cosmetic side information.

  • The method is also well aligned with the actual structure of instructional video data. These datasets already come with action vocabularies and task descriptions, so leveraging language to reduce semantic ambiguity is not artificial—it is a natural use of available structure.

Critical weaknesses
Concern Raised in Previous ReviewCurrent SubmissionUpdated Assessment
Early collapse of uncertainty into a single action labelSection 3.3 still uses ROUGE-1 matching to assign generated descriptions to discrete action labels, followed by extraction of the corresponding canonical text embeddings.Partially addressed. The unknown-label mechanism handles unmatched descriptions, but the method still collapses uncertainty into a single action label.
Dependence on external foundation models and curated language resourcesThe framework continues to rely on pretrained VLMs, LLMs, and HowTo100M text representations.Insufficiently investigated. Such dependencies are reasonable, but their individual contributions to the reported gains are not fully isolated.
Lack of systematic evaluation of language-description qualityAppendix B.1 provides ROUGE-1 score distributions, but does not directly evaluate semantic correctness, discriminability, or downstream utility of different descriptions.Partially addressed. Lexical similarity is analyzed.
Unclear contribution of language representations versus planning architectureTable 4 provides language-versus-visual comparisons, and Figure 4(d) evaluates the planning ensemble.Partially addressed. These experiments demonstrate the effectiveness of the proposed combination.
Computational complexity and efficiencyAppendix A provides implementation and training details, but limited end-to-end runtime and computational-cost comparisons.Insufficiently evaluated. Reproducibility is supported by implementation details, while the accuracy–efficiency trade-off remains unclear.
Limited failure-case analysis and error propagationComponent-level experiments are provided, but systematic evaluation of how incorrect start/goal predictions propagate into planning errors remains limited.Largely unresolved. A targeted error-propagation analysis would strengthen the paper.
Questions for authors
  • NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.