- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PAW, a framework that induces a natural-language abstract world model from egocentric human videos and then learns a VLM-based planner to use this model. The induced model contains high-level actions with preconditions, effects, information-gathering actions that resolve unknown predicates, and bimanual coordination constraints. The planner is fine-tuned to interleave action selection with successor-belief prediction, using the abstract world model both to reconstruct state-action supervision from videos and to condition planning at training and test time. The training objective is maximum likelihood over reconstructed plans: Validation includes three kitchen-style task domains, offline plan-quality comparison judged by an LLM, ablations, controlled experiments with incomplete world models and reduced fine-tuning data, online human-assistance trials, and a real-robot demonstration. The reported results show PAW outperforming symbolic planning over generated domains and VLM planners trained on the same videos, with improved online intervention rates and graceful degradation under incomplete models.
Critical strengths
The core idea is consequential: the paper does not treat an induced abstract world model as an exact symbolic specification, but instead learns a planner that tolerates incomplete preconditions/effects while still using explicit action semantics. This is a promising middle ground between brittle symbolic planning and end-to-end VLM planners, and the ablations (e.g., removing the action-model input, belief update, state-action prediction) support that the individual components matter.
The validation is broader than many planning papers: offline plan comparison, online human assistance, controlled robustness tests, and a real-robot demonstration together provide meaningful evidence that video-induced abstract models can support closed-loop planning beyond demonstrated procedures.
Critical weaknesses
The main empirical evidence relies heavily on an LLM-as-judge offline metric and relatively small online trials. The judge model is also part of the same proprietary model family used in the pipeline, and the paper does not report human agreement with the judge or statistical uncertainty in a way that fully rules out evaluator bias. This limits how strongly the offline ranking can be interpreted.
Reproducibility and clarity around the induction pipeline are not fully satisfactory. The method is called PAW in the main text but DPK in figures/appendices, and the pipeline depends on several proprietary LLM/VLM components plus optional manual contact annotations. The paper would be stronger with clearer release details, exact model/version availability, prompt sensitivity analysis, and a more transparent accounting of annotation and model-induction costs.
Questions for authors
Please clarify the naming inconsistency between PAW and DPK. Is this purely a rebranding/typo, or are they intended to denote different components?
Can you report human-LLM judge agreement on a subset of offline plans, and provide confidence intervals or paired significance tests for the main comparisons and online intervention rates?
How sensitive are the induced abstract world models and final planning results to the choice/version of the proprietary LLM/VLM components, prompt variants, and the optional contact annotations?
4 comments
Clarification on confidence intervals
Thank you for the suggestion. Tables 1 and 3 report means and standard deviations; we now add two-sided 95% confidence intervals, shown as mean [95% CI].
Offline (Table 1). Student-t 95% intervals, with per method, computed from the reported means and standard deviations. PAW has the highest mean on every criterion in both settings.
New Action Sequence:
| Method | Model | Presence | No Extra Steps | Ordering |
|---|---|---|---|---|
| VLM-as-Planner | GPT-5.6 Sol | 4.22 [4.13, 4.31] | 4.37 [4.29, 4.45] | 4.41 [4.33, 4.49] |
| VLM-as-Planner | Qwen3-VL | 3.48 [3.36, 3.60] | 4.20 [4.11, 4.29] | 4.16 [4.06, 4.26] |
| Finetuned VLM-as-Planner | Qwen3-VL | 4.39 [4.30, 4.48] | 4.48 [4.40, 4.56] | 4.43 [4.35, 4.51] |
| LLM-as-Domain-Generator | GPT-5.6 Sol | 2.55 [2.38, 2.72] | 4.64 [4.57, 4.71] | 2.69 [2.50, 2.88] |
| PAW | Qwen3-VL | 4.63 [4.57, 4.69] | 4.79 [4.74, 4.84] | 4.88 [4.84, 4.92] |
New Strategy:
| Method | Model | Presence | No Extra Steps | Ordering |
|---|---|---|---|---|
| VLM-as-Planner | GPT-5.6 Sol | 4.25 [4.16, 4.34] | 4.39 [4.31, 4.47] | 4.36 [4.27, 4.45] |
| VLM-as-Planner | Qwen3-VL | 3.46 [3.34, 3.58] | 4.21 [4.11, 4.31] | 4.27 [4.17, 4.37] |
| Finetuned VLM-as-Planner | Qwen3-VL | 4.32 [4.23, 4.41] | 4.45 [4.37, 4.53] | 4.49 [4.41, 4.57] |
| LLM-as-Domain-Generator | GPT-5.6 Sol | 2.63 [2.45, 2.81] | 4.60 [4.52, 4.68] | 2.74 [2.55, 2.93] |
| PAW | Qwen3-VL | 4.61 [4.55, 4.67] | 4.76 [4.71, 4.81] | 4.80 [4.75, 4.85] |
Online (Table 3). Bias-corrected and accelerated (BCa) bootstrap 95% intervals from 100,000 resamples of matched scenarios within each domain and setting, using the same scenarios for both methods. Baseline = Finetuned VLM-as-Planner; n is per method. PAW has higher SPL and a lower intervention rate than the baseline in all six domain–setting combinations.
| Setting / Domain | n | Baseline SPL | PAW SPL | Baseline intervention rate | PAW intervention rate |
|---|---|---|---|---|---|
| NAS / Boil Water | 6 | 0.861 [0.789, 0.958] | 0.976 [0.905, 1.000] | 0.178 [0.056, 0.272] | 0.028 [0.000, 0.111] |
| NAS / Make Instant Noodles | 8 | 0.873 [0.808, 0.922] | 0.948 [0.862, 1.000] | 0.156 [0.093, 0.260] | 0.065 [0.000, 0.178] |
| NAS / Clean Up Kitchen | 4 | 0.886 [0.840, 0.967] | 0.967 [0.900, 1.000] | 0.135 [0.038, 0.192] | 0.038 [0.000, 0.115] |
| NS / Boil Water | 6 | 0.848 [0.778, 0.926] | 0.948 [0.911, 0.983] | 0.194 [0.101, 0.318] | 0.058 [0.019, 0.100] |
| NS / Make Instant Noodles | 8 | 0.834 [0.803, 0.865] | 0.942 [0.874, 0.988] | 0.203 [0.161, 0.250] | 0.069 [0.014, 0.153] |
| NS / Clean Up Kitchen | 6 | 0.899 [0.873, 0.960] | 0.958 [0.897, 1.000] | 0.116 [0.049, 0.146] | 0.049 [0.000, 0.118] |
Result: sensitivity to models, prompts and annotations
Thank you for raising this point. Regarding the contact annotations, Table 2 in the original paper shows that removing contact annotations lowers PAW's mean from 4.75 to 4.57, still above the strongest baseline (4.43). We have now added new experiments to test sensitivity to models and prompts. We replaced the segmentation VLM, the model induction LLM, and the planner prompt (with a simpler one). All variants stay within 0.09 of the standard setting.
| Method | Planner | NAS Presence | NAS No Extra | NAS Ordering | NS Presence | NS No Extra | NS Ordering | Mean |
|---|---|---|---|---|---|---|---|---|
| PAW (standard) | Qwen3-VL-8B | 4.63 ± 0.56 | 4.79 ± 0.47 | 4.88 ± 0.37 | 4.61 ± 0.59 | 4.76 ± 0.50 | 4.80 ± 0.44 | 4.75 |
| Segmentation: Gemini 3 Flash | Qwen3-VL-8B | 4.59 ± 0.62 | 4.81 ± 0.49 | 4.84 ± 0.43 | 4.59 ± 0.63 | 4.74 ± 0.54 | 4.75 ± 0.52 | 4.72 |
| Segmentation: Gemini 2.5 Flash | Qwen3-VL-8B | 4.50 ± 0.71 | 4.73 ± 0.55 | 4.79 ± 0.50 | 4.62 ± 0.63 | 4.64 ± 0.63 | 4.73 ± 0.53 | 4.67 |
| Model induction: GPT-5.6 Terra | Qwen3-VL-8B | 4.55 ± 0.65 | 4.75 ± 0.52 | 4.89 ± 0.39 | 4.57 ± 0.65 | 4.68 ± 0.60 | 4.76 ± 0.51 | 4.70 |
| Model induction: Claude Sonnet 5 | Qwen3-VL-8B | 4.52 ± 0.69 | 4.74 ± 0.54 | 4.84 ± 0.43 | 4.55 ± 0.64 | 4.78 ± 0.49 | 4.71 ± 0.58 | 4.69 |
| Model induction: GPT-5.4 | Qwen3-VL-8B | 4.65 ± 0.62 | 4.70 ± 0.59 | 4.73 ± 0.60 | 4.52 ± 0.71 | 4.65 ± 0.64 | 4.68 ± 0.62 | 4.66 |
| Simpler planner prompt | Qwen3-VL-8B | 4.58 ± 0.62 | 4.80 ± 0.46 | 4.82 ± 0.45 | 4.52 ± 0.68 | 4.77 ± 0.49 | 4.77 ± 0.49 | 4.71 |
The original planner prompt is in Appendix A.4; we will add the simpler prompt to the appendix.
Clarification on model naming
DPK is an earlier name for PAW, not a separate component. We apologize for the leftover labels in the figures and appendix and will correct them.
Result: validation of the offline LLM judge
Thank you for the suggestion. To test whether the judge can be trusted, we ran a human study of agreement between human and LLM judgments, in both ratings and method rankings. We randomly sampled 150 plans from the New Action Sequence setting, 30 per method. Three annotators each scored all 150 plans from 1 to 5 on the judge's criteria (Presence, No Extra Steps, Ordering), giving 1,350 human ratings. Method identities were hidden, and each annotator saw the plans one at a time in a separately randomized order. All human–LLM comparisons use the original LLM scores for these same 150 plans.
Rating agreement. For each criterion, we computed quadratic-weighted Cohen's κ between each annotator and the LLM judge and averaged over annotators. As an inter-annotator reference, we averaged κ over the three human–human pairs. We also report exact human–LLM score agreement.
Method ranking. Human and LLM judges ranked the five methods identically, from lowest to highest mean score:
LLM-as-Domain-Generator (GPT-5.6 Sol) < VLM-as-Planner (Qwen3-VL) < VLM-as-Planner (GPT-5.6 Sol) < Finetuned VLM-as-Planner (Qwen3-VL) < PAW (Qwen3-VL).
In this order, the human mean scores were 3.33, 3.87, 4.16, 4.31 and 4.79, and the LLM mean scores were 3.37, 3.81, 4.19, 4.31 and 4.82. The two judges are therefore aligned, and both rank PAW highest. Because offline scores are relative to the demonstrated sequence, we complement them with the online evaluation, which has no reference plan and accepts any feasible plan that reaches the goal. We will include the annotation interface in the appendix.