- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LAP converts endpoint video observations into action descriptions using a fine-tuned multimodal model, matches descriptions to action labels, and embeds canonical text descriptions. A small language model proposes intermediate descriptions; their embeddings select a training-derived cluster that conditions a diffusion planner. Predictions are combined with a base diffusion planner. Experiments on CrossTask, COIN, and NIV report large procedure-success improvements.
Critical strengths
-
The semantic endpoint bottleneck is a useful way to connect video understanding with procedure planning. Evaluating three datasets and longer action sequences provides meaningful scope, with substantial reported gains, such as CrossTask length-three success of 54.3% versus 33.9% for the strongest listed comparator.
-
The paper reruns baselines on the curated CrossTask split and reports component comparisons, including visual-only and conditioned/base planning variants. The clustering description explicitly restricts fitting to training data, which is appropriate.
Critical weaknesses
-
Two central procedural ambiguities prevent verification of soundness. Equations 2 and 3 minimize the true-mode classification BCE through the VLM parameters, which encourages teacher-forcing/free-running distinguishability; the motivation instead suggests aligning the regimes. No adversarial sign, gradient reversal, or alternating generator update is specified. Separately, Section 3.3 compares generated descriptions with “ground truth descriptions of actions.” It is unclear whether these are a fixed dictionary of all action classes or the annotation of the evaluated clip. The former is valid; the latter would introduce unavailable endpoint information. I am not asserting that the implementation uses an oracle, but the manuscript must define an annotation-free inference procedure.
-
The large visual-only gap does not isolate language semantics from changes in endpoint recognition, model capacity, and supervision. LAP uses a large fine-tuned VLM and a discrete action-label bottleneck, whereas LAP-vo uses a different visual extractor. A stronger control should retain the same endpoint predictor and training data, comparing visual features, class-label embeddings, and canonical text embeddings. Otherwise, the main gain could reflect improved endpoint classification rather than the proposed language-aware planning mechanism.
Questions for authors
-
What exact gradients update the VLM and regularizer? Please give the signed objectives, detach operations, and update schedule, explaining how they align rather than separate the two decoding regimes.
-
Please give complete test-time pseudocode: which descriptions are available for matching, how candidates are selected, and whether every planner input endpoint action is predicted without access to clip-specific labels.
-
Can you repeat the representation comparison with the same VLM endpoint predictions and supervision, using visual features, class-label embeddings, and text embeddings? How much of the reported gain remains?