acceptodds
Under review as a conference paper at ICLR 2027

Learning to Plan with Natural-Language World Models from Human Egocentric Videos

Abstract

Human demonstrations provide examples of how tasks are performed, but embodied agents need reusable general knowledge of when actions apply, how they change the world, and how they can be recomposed into new plans. We present Planning in Abstract Worlds (PAW), a framework that learns natural-language abstract world models from human egocentric videos and fine-tunes a vision-language model (VLM) to plan with them in a closed loop. PAW represents tasks with natural-language predicates and high-level actions with explicit preconditions, effects, and bimanual coordination types. Instead of treating the induced abstract world model as an exact transition system, it uses the model to reconstruct state-action supervision from unlabeled video and as structured context for a learned planner. The planner maintains a belief state, supports information gathering under partial observability, and replans from new visual observations. Across offline planning and online human assistance experiments in three everyday manipulation domains, PAW outperforms VLM planner baselines and symbolic planning with induced models, generalizes to unseen action compositions, and remains effective with incomplete models. Robot demonstrations further illustrate transfer from human videos to physical execution. Together, these results support learning reusable abstract world models from human videos and learning to plan with them.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Abstract World Models, Task Planning, VLMs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6HG7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PAW, a framework that induces a natural-language abstract world model from egocentric human videos and then learns a VLM-based planner to use this model. The induced model contains high-level actions with preconditions, effects, information-gathering actions that resolve unknown predicates, and bimanual coordination constraints. The planner is fine-tuned to interleave action selection with successor-belief prediction, using the abstract world model both to reconstruct state-action supervision from videos and to condition planning at training and test time. The training objective is maximum likelihood over reconstructed plans: θ∗=arg⁡max⁡θ∑ℓ,bx,σ∈D~log⁡pθ(σ∣bx,ℓ,P,A,τ).\theta^*=\arg\max_\theta \sum_{\ell,b_x,\sigma\in\tilde D}\log p_\theta(\sigma\mid b_x,\ell,\mathcal P,\mathcal A,\tau). Validation includes three kitchen-style task domains, offline plan-quality comparison judged by an LLM, ablations, controlled experiments with incomplete world models and reduced fine-tuning data, online human-assistance trials, and a real-robot demonstration. The reported results show PAW outperforming symbolic planning over generated domains and VLM planners trained on the same videos, with improved online intervention rates and graceful degradation under incomplete models.

Critical strengths

The core idea is consequential: the paper does not treat an induced abstract world model as an exact symbolic specification, but instead learns a planner that tolerates incomplete preconditions/effects while still using explicit action semantics. This is a promising middle ground between brittle symbolic planning and end-to-end VLM planners, and the ablations (e.g., removing the action-model input, belief update, state-action prediction) support that the individual components matter.

The validation is broader than many planning papers: offline plan comparison, online human assistance, controlled robustness tests, and a real-robot demonstration together provide meaningful evidence that video-induced abstract models can support closed-loop planning beyond demonstrated procedures.

Critical weaknesses

The main empirical evidence relies heavily on an LLM-as-judge offline metric and relatively small online trials. The judge model is also part of the same proprietary model family used in the pipeline, and the paper does not report human agreement with the judge or statistical uncertainty in a way that fully rules out evaluator bias. This limits how strongly the offline ranking can be interpreted.

Reproducibility and clarity around the induction pipeline are not fully satisfactory. The method is called PAW in the main text but DPK in figures/appendices, and the pipeline depends on several proprietary LLM/VLM components plus optional manual contact annotations. The paper would be stronger with clearer release details, exact model/version availability, prompt sensitivity analysis, and a more transparent accounting of annotation and model-induction costs.

Questions for authors

Please clarify the naming inconsistency between PAW and DPK. Is this purely a rebranding/typo, or are they intended to denote different components?

Can you report human-LLM judge agreement on a subset of offline plans, and provide confidence intervals or paired significance tests for the main comparisons and online intervention rates?

How sensitive are the induced abstract world models and final planning results to the choice/version of the proprietary LLM/VLM components, prompt variants, and the optional contact annotations?

4 comments
AuthorsOpenReview ↗

Result: validation of the offline LLM judge

Thank you for the suggestion. To test whether the judge can be trusted, we ran a human study of agreement between human and LLM judgments, in both ratings and method rankings. We randomly sampled 150 plans from the New Action Sequence setting, 30 per method. Three annotators each scored all 150 plans from 1 to 5 on the judge's criteria (Presence, No Extra Steps, Ordering), giving 1,350 human ratings. Method identities were hidden, and each annotator saw the plans one at a time in a separately randomized order. All human–LLM comparisons use the original LLM scores for these same 150 plans.

Rating agreement. For each criterion, we computed quadratic-weighted Cohen's κ between each annotator and the LLM judge and averaged over annotators. As an inter-annotator reference, we averaged κ over the three human–human pairs. We also report exact human–LLM score agreement.

CriterionHuman–LLM κHuman–Human κHuman–LLM exact agreement
Presence0.9100.84378.0%
No Extra Steps0.8760.77286.7%
Ordering0.8070.69366.2%
Macro-average0.8640.76977.0%

Method ranking. Human and LLM judges ranked the five methods identically, from lowest to highest mean score:

LLM-as-Domain-Generator (GPT-5.6 Sol) < VLM-as-Planner (Qwen3-VL) < VLM-as-Planner (GPT-5.6 Sol) < Finetuned VLM-as-Planner (Qwen3-VL) < PAW (Qwen3-VL).

In this order, the human mean scores were 3.33, 3.87, 4.16, 4.31 and 4.79, and the LLM mean scores were 3.37, 3.81, 4.19, 4.31 and 4.82. The two judges are therefore aligned, and both rank PAW highest. Because offline scores are relative to the demonstrated sequence, we complement them with the online evaluation, which has no reference plan and accepts any feasible plan that reaches the goal. We will include the annotation interface in the appendix.

AuthorsOpenReview ↗

Clarification on confidence intervals

Thank you for the suggestion. Tables 1 and 3 report means and standard deviations; we now add two-sided 95% confidence intervals, shown as mean [95% CI].

Offline (Table 1). Student-t 95% intervals, xˉ±t0.975, n−1 s/n\bar{x} \pm t_{0.975,\,n-1}\, s/\sqrt{n} with n=360n = 360 per method, computed from the reported means and standard deviations. PAW has the highest mean on every criterion in both settings.

New Action Sequence:

MethodModelPresenceNo Extra StepsOrdering
VLM-as-PlannerGPT-5.6 Sol4.22 [4.13, 4.31]4.37 [4.29, 4.45]4.41 [4.33, 4.49]
VLM-as-PlannerQwen3-VL3.48 [3.36, 3.60]4.20 [4.11, 4.29]4.16 [4.06, 4.26]
Finetuned VLM-as-PlannerQwen3-VL4.39 [4.30, 4.48]4.48 [4.40, 4.56]4.43 [4.35, 4.51]
LLM-as-Domain-GeneratorGPT-5.6 Sol2.55 [2.38, 2.72]4.64 [4.57, 4.71]2.69 [2.50, 2.88]
PAWQwen3-VL4.63 [4.57, 4.69]4.79 [4.74, 4.84]4.88 [4.84, 4.92]

New Strategy:

MethodModelPresenceNo Extra StepsOrdering
VLM-as-PlannerGPT-5.6 Sol4.25 [4.16, 4.34]4.39 [4.31, 4.47]4.36 [4.27, 4.45]
VLM-as-PlannerQwen3-VL3.46 [3.34, 3.58]4.21 [4.11, 4.31]4.27 [4.17, 4.37]
Finetuned VLM-as-PlannerQwen3-VL4.32 [4.23, 4.41]4.45 [4.37, 4.53]4.49 [4.41, 4.57]
LLM-as-Domain-GeneratorGPT-5.6 Sol2.63 [2.45, 2.81]4.60 [4.52, 4.68]2.74 [2.55, 2.93]
PAWQwen3-VL4.61 [4.55, 4.67]4.76 [4.71, 4.81]4.80 [4.75, 4.85]

Online (Table 3). Bias-corrected and accelerated (BCa) bootstrap 95% intervals from 100,000 resamples of matched scenarios within each domain and setting, using the same scenarios for both methods. Baseline = Finetuned VLM-as-Planner; n is per method. PAW has higher SPL and a lower intervention rate than the baseline in all six domain–setting combinations.

Setting / DomainnBaseline SPLPAW SPLBaseline intervention ratePAW intervention rate
NAS / Boil Water60.861 [0.789, 0.958]0.976 [0.905, 1.000]0.178 [0.056, 0.272]0.028 [0.000, 0.111]
NAS / Make Instant Noodles80.873 [0.808, 0.922]0.948 [0.862, 1.000]0.156 [0.093, 0.260]0.065 [0.000, 0.178]
NAS / Clean Up Kitchen40.886 [0.840, 0.967]0.967 [0.900, 1.000]0.135 [0.038, 0.192]0.038 [0.000, 0.115]
NS / Boil Water60.848 [0.778, 0.926]0.948 [0.911, 0.983]0.194 [0.101, 0.318]0.058 [0.019, 0.100]
NS / Make Instant Noodles80.834 [0.803, 0.865]0.942 [0.874, 0.988]0.203 [0.161, 0.250]0.069 [0.014, 0.153]
NS / Clean Up Kitchen60.899 [0.873, 0.960]0.958 [0.897, 1.000]0.116 [0.049, 0.146]0.049 [0.000, 0.118]
AuthorsOpenReview ↗

Result: sensitivity to models, prompts and annotations

Thank you for raising this point. Regarding the contact annotations, Table 2 in the original paper shows that removing contact annotations lowers PAW's mean from 4.75 to 4.57, still above the strongest baseline (4.43). We have now added new experiments to test sensitivity to models and prompts. We replaced the segmentation VLM, the model induction LLM, and the planner prompt (with a simpler one). All variants stay within 0.09 of the standard setting.

MethodPlannerNAS PresenceNAS No ExtraNAS OrderingNS PresenceNS No ExtraNS OrderingMean
PAW (standard)Qwen3-VL-8B4.63 ± 0.564.79 ± 0.474.88 ± 0.374.61 ± 0.594.76 ± 0.504.80 ± 0.444.75
Segmentation: Gemini 3 FlashQwen3-VL-8B4.59 ± 0.624.81 ± 0.494.84 ± 0.434.59 ± 0.634.74 ± 0.544.75 ± 0.524.72
Segmentation: Gemini 2.5 FlashQwen3-VL-8B4.50 ± 0.714.73 ± 0.554.79 ± 0.504.62 ± 0.634.64 ± 0.634.73 ± 0.534.67
Model induction: GPT-5.6 TerraQwen3-VL-8B4.55 ± 0.654.75 ± 0.524.89 ± 0.394.57 ± 0.654.68 ± 0.604.76 ± 0.514.70
Model induction: Claude Sonnet 5Qwen3-VL-8B4.52 ± 0.694.74 ± 0.544.84 ± 0.434.55 ± 0.644.78 ± 0.494.71 ± 0.584.69
Model induction: GPT-5.4Qwen3-VL-8B4.65 ± 0.624.70 ± 0.594.73 ± 0.604.52 ± 0.714.65 ± 0.644.68 ± 0.624.66
Simpler planner promptQwen3-VL-8B4.58 ± 0.624.80 ± 0.464.82 ± 0.454.52 ± 0.684.77 ± 0.494.77 ± 0.494.71

The original planner prompt is in Appendix A.4; we will add the simpler prompt to the appendix.

AuthorsOpenReview ↗

Clarification on model naming

DPK is an earlier name for PAW, not a separate component. We apologize for the leftover labels in the figures and appendix and will correct them.

Reviewer_M2DCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Planning in Abstract Worlds (PAW). Abstract world model is induced from human ego video. Compared to strict PDDL, this natural language-based representation allows planning under imperfect domain. The induced abstract world model is used to construct training supervision for a VLM planner, which is trained to predict actions and the resulting beliefs. Experiments show superior performance against VLM-as-planner or domain generator. However, the method is only evaluated on three self-designed domains, leaving the generalizability insufficiently established.

Critical strengths

• The proposed method shows strong performance on three offline tasks and oneline human assistance, compared to VLM as planner and domain generator baselines.

• Ablations show that explicit preconditions/effects, belief tracking etc contribute to planning success.

Critical weaknesses

• The evaluation is restricted to three self-designed tasks. The "new action sequence" and "new strategy" tasks are also from the same domains. Results on more established public benchmarks will better establish the difficulty of the claimed generalization.

• Abstract world model induction relies on pretrained language and video models. Sensitivity of the method to the choice of induction models remains unclear.

• The ablations do not isolate the advantage of representing predicate precondition and effects in natural language rather than PDDL. A comparison using natural language versus PDDL with the same learned planner would clarify it.

Questions for authors

• How sensitive is the quality of the abstract world model to the induction models used?

9 comments
AuthorsOpenReview ↗

Clarification on evaluation – (1) why we built our own benchmark

Thank you for raising this point. We address it in four parts: (1) why we built our own benchmark, (2) validation of the offline LLM judge, (3) the online human-assistance protocol, and (4) the open release.

(1) Why we built our own benchmark. Our evaluation targets planning: given the actions observed in human demonstrations, can a method compose them into new action sequences that achieve a high-level goal from the current observation? Planning from real observations raises six requirements, and no public benchmark we know of evaluates them jointly:

  1. High-level goal. The instruction states the desired outcome; the method must infer which actions to take and in what order.
  2. Novel compositions. Test tasks require action sequences not seen in training, built from trained actions.
  3. Executable dependencies. Plans are executed, so an action whose preconditions were not established by earlier actions fails.
  4. Long horizon. Test plans reach 10 or more high-level actions.
  5. Observational uncertainty. The correct next action depends on a fact the agent must first observe.
  6. Bimanual coordination. Plans must respect how the two hands' actions can be combined.

The table below compares widely used robotics and video benchmarks against these requirements (Y = required by the benchmark's evaluation).

BenchmarkHigh-level goalNovel compositionsExecutable dependenciesLong horizonObservational uncertaintyBimanual
BEHAVIOR-1K 2026 ChallengeYNYYNY
CALVINNYYNNN
ALFREDNNYNNN
LIBERONNYNNN
RoboTwin 2.0NNYNNY
RoboCasa365NYYYNN
RoboVQAYNYYNN
EgoPlan-BenchYNNNNN
CrossTaskYNNNNN
COINYNNNNN
NIVYNNNNN
PAW benchmark (ours)YYYYYY

We use four examples to illustrate the gap. RoboCasa365 evaluates new compositions of trained skills on 16 unseen composite tasks, but most of its instructions spell out the procedure; ArrangeTea, for example, ends with "Then close the cabinet doors," a step no goal of arranging tea would imply. The BEHAVIOR Challenge evaluates held-out instances of the same 100 task definitions covered by its 20,000 demonstrations, so no new action sequence is required. ALFRED pairs each goal with step-by-step instructions and draws all tasks from seven task types (Shridhar et al., 2020), which FILM (Min et al., 2022) solves with one subgoal template per type. Procedure planning on CrossTask and COIN predicts 3–4 steps from a small, task-specific set and scores success by exact match against a single annotated sequence; because instructional videos often leave steps implicit, these sequences are not guaranteed to be executable.

Like prior work that evaluates new planning methods on purpose-built domains (BLADE, Liu et al., 2024; pix2pred, Athalye et al., 2025; ExoPredicator, Liang et al., 2026; UniDomain, Ye et al., 2025), we designed our three domains around these requirements. Each goal can be achieved by several strategies, such as heating water with a kettle, a pot or a microwave, giving 22, 16 and 8 distinct action sequences; the New Strategy setting tests recombining strategies seen separately in training. Goals state outcomes rather than steps: "Fill the mug with water" can be achieved from the faucet or from a water bottle, and the planner must decide which. Information gathering determines the procedure, for example checking whether the kettle already holds water, or whether the water is hot enough to soak noodles. Clean Up Kitchen requires coordinated and uncoordinated bimanual actions, and its plans reach 15 actions.

AuthorsOpenReview ↗

Clarification on evaluation – (2) validation of the offline LLM judge

(2) Validation of the offline LLM judge. Our offline evaluation scores 360 test plans per method with an LLM judge. To test whether the judge can be trusted, we ran a human study of agreement between human and LLM judgments, in both ratings and method rankings. We randomly sampled 150 plans from the New Action Sequence setting, 30 per method. Three annotators each scored all 150 plans from 1 to 5 on the judge's criteria (Presence, No Extra Steps, Ordering), giving 1,350 human ratings. Method identities were hidden, and each annotator saw the plans one at a time in a separately randomized order. All human–LLM comparisons use the original LLM scores for these same 150 plans.

Rating agreement. For each criterion, we computed quadratic-weighted Cohen's κ between each annotator and the LLM judge and averaged over annotators. As an inter-annotator reference, we averaged κ over the three human–human pairs. We also report exact human–LLM score agreement.

CriterionHuman–LLM κHuman–Human κHuman–LLM exact agreement
Presence0.9100.84378.0%
No Extra Steps0.8760.77286.7%
Ordering0.8070.69366.2%
Macro-average0.8640.76977.0%

Method ranking. Human and LLM judges ranked the five methods identically, from lowest to highest mean score:

LLM-as-Domain-Generator (GPT-5.6 Sol) < VLM-as-Planner (Qwen3-VL) < VLM-as-Planner (GPT-5.6 Sol) < Finetuned VLM-as-Planner (Qwen3-VL) < PAW (Qwen3-VL).

In this order, the human mean scores were 3.33, 3.87, 4.16, 4.31 and 4.79, and the LLM mean scores were 3.37, 3.81, 4.19, 4.31 and 4.82. The two judges are therefore aligned, and both rank PAW highest. Because offline scores are relative to the demonstrated sequence, we complement them with the online evaluation, which has no reference plan and accepts any feasible plan that reaches the goal. We will include the annotation interface in the appendix.

AuthorsOpenReview ↗

Clarification on evaluation – (3) online human-assistance protocol

(3) Online human-assistance protocol. The original manuscript did not describe the trial allocation and intervention protocol in enough detail; we clarify both here and will add them to the appendix.

Trial allocation. We ran 40 trials per method for PAW and Finetuned VLM-as-Planner: 20 under New Action Sequence and 20 under New Strategy, each split into 6 Boil Water, 8 Make Instant Noodles and 6 Clean Up Kitchen trials. Figures A1–A3 show the exact action sequences tested. The 12 paired episodes in Appendix B are selected diagnostic examples, not additional trials.

Intervention protocol. As in RoboVQA (Sermanet et al., 2024), a human performs the physical actions so that planning is evaluated separately from low-level control. For both methods, the same executor carried out the proposed actions and judged interventions using the same criteria, without being told which planner was running. Environments were initialized from the held-out test scenarios; paired runs shared the language goal and initial task conditions, although object layouts could differ. At each step, the planner observed the scene and proposed the next action. The executor intervened when the proposal:

  • had unmet preconditions or required information not yet established (e.g., pouring from a capped bottle, or using a closed microwave without checking that it is empty);
  • required an unavailable object, tool or ingredient;
  • repeated a completed action (e.g., soaping an already-soaped plate);
  • violated a task or hand-use requirement, including a one-hand restriction or a required two-hand composition;
  • declared completion while a task requirement remained unsatisfied.

The executor rejected such an action before executing it, gave textual feedback naming the reason, and let the planner replan; the executor never performed a corrective step on the planner's behalf. Each rejected proposal counted as one intervention, however many reasons it had. Accepted actions were executed, followed by a new observation and replanning. A feasible alternative procedure that satisfied the goal and constraints was never rejected for differing from the reference plan.

Reviewer_yDLmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAW induces natural-language action preconditions and effects from egocentric videos, uses them to reconstruct belief-action supervision, and fine-tunes a planner conditioned on this model. Visual belief updates enable closed-loop replanning. Evaluation covers offline plan scoring, human execution in three kitchen domains, and qualitative robot demonstrations.

Critical strengths
  1. Combining explicit action semantics with learned planning is a useful approach to handling imperfect rules and partial observability. Ablations support the roles of action semantics, belief tracking, and state-action supervision.

  2. The human-execution gains are substantial: intervention-free completion rises from 24% to 78% for new action sequences and from 11% to 60% for new strategies. These results provide evidence beyond LLM-judge scores.

Critical weaknesses
  1. The contribution of video induction and specialized planner training is not fully isolated. Many illustrated rules resemble LM commonsense; SCHEMA (Niu et al., ICLR 2024) already uses LLM-generated state-change descriptions. A text-only rule-induction control is missing. Likewise, although the same-backbone ablation supports SFT, the stronger GPT baseline lacks PAW’s belief and preconditions/effects (Appendix A.5). A matched-input comparison is needed to distinguish gains from structured knowledge versus planner training.

  2. Evaluation remains narrow and difficult to compare independently: offline experiments cover only Boil Water, closed-loop tests use human execution in three author-designed kitchen domains, and robot results are qualitative. The gains are encouraging, but a standardized executable evaluation—such as a suitable adaptation of ALFRED or BEHAVIOR-1K—or a comparably reproducible protocol would strengthen evidence beyond these environments and human-intervention procedures.

Questions for authors
  1. What evidence isolates the value of video-derived rules over LM-generated rules, and of SFT over a strong prompted LLM given the same belief and rules?

  2. Can you clarify how the 40 trials per method are distributed across domains/settings and how human interventions are standardized for reproducibility?

  3. Are failed or redundant demonstration steps filtered? Since SFT imitates reconstructed trajectories, how are unsuccessful actions distinguished from nominal effects? Incomplete-rule ablations do not test this form of demonstration noise.

9 comments
AuthorsOpenReview ↗

Result: video-based model induction vs text-based model induction

Thank you for the suggestion. Following SCHEMA, we generated the abstract world model with an LLM from action names alone, without videos, and fine-tuned the same planner on it. This planner scores below the one trained with our video-induced model (mean 4.47 vs. 4.75).

MethodPlannerNAS PresenceNAS No ExtraNAS OrderingNS PresenceNS No ExtraNS OrderingMean
PAW (standard)Qwen3-VL-8B4.63 ± 0.564.79 ± 0.474.88 ± 0.374.61 ± 0.594.76 ± 0.504.80 ± 0.444.75
SCHEMA-style (text only)Qwen3-VL-8B4.36 ± 0.854.54 ± 0.704.58 ± 0.774.31 ± 0.904.52 ± 0.744.53 ± 0.824.47

Qualitatively, the text-only model cannot represent observational uncertainty or bimanual coordination, and it is wrong when action names are ambiguous.

For an information-gathering action that checks whether a closed kettle holds water, our model uses a three-valued predicate that changes from unknown to known, and records that the lid is opened; SCHEMA's model captures neither.

SCHEMA's model:

{
  "action_name": "Ensure the kettle contains water",
  "precondition": [],
  "effect": ["The kettle contains water."],
  "manuality": "unimanual"
}

Our model:

{
  "action_name": "inspect the closed kettle for water and find it contains water",
  "precondition": [
    "It is not that the kettle lid is open.",
    "It is not that it is known whether the kettle contains water."
  ],
  "effect": [
    "It is known whether the kettle contains water.",
    "The kettle contains water.",
    "The kettle lid is open."
  ],
  "manuality": "unimanual"
}

For an uncoordinated bimanual action (one hand opens a food container while the other pours food into it), SCHEMA's preconditions and effects are correct, but it labels the action unimanual and does not decompose it into per-hand actions. Our model does both.

SCHEMA's model:

{
  "action_name": "open food container then pour food from plate into it",
  "precondition": [
    "It is not that the food container is open.",
    "The food is on the paper plate."
  ],
  "effect": [
    "The food container is open.",
    "The food is in the food container.",
    "It is not that the food is on the paper plate."
  ],
  "manuality": "unimanual"
}

Our model:

{
  "bimanual_action_definition": {
    "action_name": "open food container and pour food from paper plate into food container",
    "precondition": [
      "It is not that the food container is open.",
      "The food is on the paper plate."
    ],
    "effect": [
      "The food container is open.",
      "The food is in the food container.",
      "It is not that the food is on the paper plate."
    ],
    "manuality": "bimanual",
    "bimanual_type": "uncoordinated"
  },
  "unimanual_action_definitions": [
    {
      "action_name": "open food container",
      "precondition": ["It is not that the food container is open."],
      "effect": ["The food container is open."],
      "manuality": "unimanual"
    },
    {
      "action_name": "pour food from paper plate into food container",
      "precondition": [
        "The food container is open.",
        "The food is on the paper plate."
      ],
      "effect": [
        "The food is in the food container.",
        "It is not that the food is on the paper plate."
      ],
      "manuality": "unimanual"
    }
  ]
}
AuthorsOpenReview ↗

Result: strong pre-trained model as the planner vs trained planner

To separate the world model from planner training, we gave GPT-5.6 Sol our abstract world model, belief state and planning prompt, without fine-tuning. It uses the structure well, improving over VLM-as-Planner with the same model (mean 4.52 vs. 4.33), but stays below our fine-tuned planner (4.75), which isolates the gain from training.

MethodPlannerNAS PresenceNAS No ExtraNAS OrderingNS PresenceNS No ExtraNS OrderingMean
PAW (standard)Qwen3-VL-8B4.63 ± 0.564.79 ± 0.474.88 ± 0.374.61 ± 0.594.76 ± 0.504.80 ± 0.444.75
PAW world model, no fine-tuningGPT-5.6 Sol4.42 ± 0.774.56 ± 0.664.63 ± 0.634.39 ± 0.804.55 ± 0.684.57 ± 0.694.52
AuthorsOpenReview ↗

Clarification on evaluation – (1) why we built our own benchmark

Thank you for raising this point. We address it in four parts: (1) why we built our own benchmark, (2) validation of the offline LLM judge, (3) the online human-assistance protocol, and (4) the open release.

(1) Why we built our own benchmark. Our evaluation targets planning: given the actions observed in human demonstrations, can a method compose them into new action sequences that achieve a high-level goal from the current observation? Planning from real observations raises six requirements, and no public benchmark we know of evaluates them jointly:

  1. High-level goal. The instruction states the desired outcome; the method must infer which actions to take and in what order.
  2. Novel compositions. Test tasks require action sequences not seen in training, built from trained actions.
  3. Executable dependencies. Plans are executed, so an action whose preconditions were not established by earlier actions fails.
  4. Long horizon. Test plans reach 10 or more high-level actions.
  5. Observational uncertainty. The correct next action depends on a fact the agent must first observe.
  6. Bimanual coordination. Plans must respect how the two hands' actions can be combined.

The table below compares widely used robotics and video benchmarks against these requirements (Y = required by the benchmark's evaluation).

BenchmarkHigh-level goalNovel compositionsExecutable dependenciesLong horizonObservational uncertaintyBimanual
BEHAVIOR-1K 2026 ChallengeYNYYNY
CALVINNYYNNN
ALFREDNNYNNN
LIBERONNYNNN
RoboTwin 2.0NNYNNY
RoboCasa365NYYYNN
RoboVQAYNYYNN
EgoPlan-BenchYNNNNN
CrossTaskYNNNNN
COINYNNNNN
NIVYNNNNN
PAW benchmark (ours)YYYYYY

We analyze four examples to illustrate the gap. RoboCasa365 evaluates new compositions of trained skills on 16 unseen composite tasks, but most of its instructions spell out the procedure; ArrangeTea, for example, ends with "Then close the cabinet doors," a step no goal of arranging tea would imply. The BEHAVIOR Challenge evaluates held-out instances of the same 100 task definitions covered by its 20,000 demonstrations, so no new action sequence is required. ALFRED pairs each goal with step-by-step instructions and draws all tasks from seven task types (Shridhar et al., 2020), which FILM (Min et al., 2022) solves with one subgoal template per type. Procedure planning on CrossTask and COIN predicts 3–4 steps from a small, task-specific set and scores success by exact match against a single annotated sequence; because instructional videos often leave steps implicit, these sequences are not guaranteed to be executable.

Like prior work that evaluates new planning methods on purpose-built domains (BLADE, Liu et al., 2024; pix2pred, Athalye et al., 2025; ExoPredicator, Liang et al., 2026; UniDomain, Ye et al., 2025), we designed our three domains around these requirements. Each goal can be achieved by several strategies, such as heating water with a kettle, a pot or a microwave, giving 22, 16 and 8 distinct action sequences; the New Strategy setting tests recombining strategies seen separately in training. Goals state outcomes rather than steps: "Fill the mug with water" can be achieved from the faucet or from a water bottle, and the planner must decide which. Information gathering determines the procedure, for example checking whether the kettle already holds water, or whether the water is hot enough to soak noodles. Clean Up Kitchen requires coordinated and uncoordinated bimanual actions, and its plans reach 15 actions.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback, which has helped us improve the paper. We evaluated PAW on CALVIN, an established public benchmark, where it outperforms all baselines. We conducted a user study showing that human and LLM-judge ratings agree on our offline experiments. We also compared PAW's world model with alternative representations, including text-only LLM-induced action schemas and PDDL, and validated the method's robustness to different models and prompts. Together, these experiments provide further empirical evidence for our claims.

During the rebuttal we added the following results (details in each response):

  1. Public benchmark validation on CALVIN. On 60 held-out goal-only tasks needing 5–6-step plans, PAW (Qwen3-VL) succeeds on 95.0%, vs. 70.0% for the fine-tuned VLM-as-Planner and 46.7% for VLM-as-Planner (GPT-5.6 Sol).
  2. Evaluation. A comparison of our benchmark with 11 robotics and video benchmarks, the full online human-assistance protocol, and a commitment to open-source the benchmark.
  3. LLM-judge validation. 1,350 blinded human ratings of Boil Water offline plans: human–LLM κ = 0.864 vs. human–human κ = 0.769, and identical method rankings.
  4. Confidence intervals for all offline and online results.
  5. Video-induced vs. text-only world models (SCHEMA-style): mean 4.75 vs. 4.47.
  6. Fine-tuned vs. prompted planner with the same world model (GPT-5.6 Sol): 4.75 vs. 4.52.
  7. Natural-language vs. PDDL world model: 4.75 vs. 4.38.
  8. Sensitivity to the segmentation VLM, the definition LLM and the planner prompt: 4.66–4.72 vs. 4.75.
  9. Failed demonstration steps: a preliminary case in which a failed action is induced as information gathering rather than as a false effect.

Unless stated otherwise, scores are mean LLM-judge scores (1–5) on the Boil Water offline experiment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.