acceptodds
Under review as a conference paper at ICLR 2027

WorldGuide: Goal-Directed Video World Model for Procedural Task Execution

Abstract

Video generators and video-based world models can synthesize plausible visual trajectories, but long-horizon procedural tasks require generation to adapt to what has actually been produced. A model must determine the next action from its generated state, execute that action, and recognize when the task is complete. Open-loop generation cannot adapt to execution outcomes, while existing closed-loop systems often rely on pretrained executors or indirect verification. This leaves a gap between deciding an action and successfully realizing it. We formulate procedural video generation as *closed-loop task execution in visual world space* and introduce **WorldGuide**. Given only an initial image and a task goal, WorldGuide predicts an atomic action, generates its corresponding video clip, and uses the generated result to select the next action or terminate. The Planner and Executor are trained on the same step-level procedural demonstrations: the Planner learns to predict the next atomic action or task completion from visual progress, while the Executor is directly trained to realize the predicted actions. Hierarchical visual memory maintains state across long-horizon execution with bounded history token cost. Due to the lack of step-level action-video supervision for joint planner-executor training, we introduce **WorldGuide Bench**: approximately 59K step-annotated videos across 245 tasks and 27 procedural categories. WorldGuide achieves a **33.33%** Task Success on WorldGuide Bench, compared with **29.90%** for the strong recent video model MiniMax-H3, even though MiniMax-H3 receives reference action plans, and achieves **47.69%** on Video-CraftBench compared with **32.73%** for MiniMax-H3 under goal-only conditioning. These results demonstrate the importance of coupling planning with learned execution for goal-directed procedural video generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video world models, goal-directed video generation, vision-language models, procedural tasks, action-conditioned generation, long-horizon reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BDB2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats procedural video generation (cooking, origami, assembly) as closed-loop task execution. Given only an initial image and a task goal, WorldGuide loops as follows:

  • A ContextPlanner (fully fine-tuned Qwen2.5-VL-7B) predicts the next atomic action, or a DONE token. It sees the goal and the last K=3 generated clips and actions.
  • An Executor (fine-tuned HunyuanVideo-1.5, conditioned on the frozen planner's action embeddings) renders that action as a short clip.
  • The new clip updates the planner's context and the Executor's hierarchical compressed memory. That memory schedule is adopted from YUME/FramePack.

The two modules are trained one after the other, with teacher forcing, on WorldGuide Bench. This is a new set of about 59K YouTube instructional videos (245 tasks, 27 categories), filtered and annotated by Gemini 2.5 Flash into 1–5 s atomic action clips with an explicit completion step. A human audit rates about 83% of annotations correct. The test split holds out 4 videos per task, so it covers unseen videos of seen tasks.

Evaluation is mostly by a Gemini 3.5 Flash judge (Task Success, Plan Accuracy, Order, Repeat/Skip) plus VBench metrics:

  • WorldGuide Bench (980 videos): 33.33% Task Success. MiniMax-H3 gets 29.90%, but receives the ground-truth action plan.
  • Video-CraftBench (294 samples, goal-only for all models): 47.69% vs 32.73%.
  • Ablations: an open-loop variant gets 11.71% and a variant without visual feedback 14.72%. Removing memory gives 27.56%. Replacing the planner with oracle reference actions raises the score to 38.82%.
  • Human study: 18 tasks, 4 raters; WorldGuide ranks first.
Critical strengths
  1. A clear closed-loop formulation, with controlled ablations that support its main mechanisms. The planner decides both the next action and when to stop, based on generated visual state, and the executor is trained on the actions the planner proposes. This is a clean, well-motivated formulation compared with pairing a planner with a frozen video model. The within-model ablations (Table 4, with Wilson intervals and tests in Table 16) show large, significant effects:

    • visual feedback for the planner: 14.72 → 33.33, p<0.001;
    • executor memory: 27.56 → 33.33 on WorldGuide Bench and 29.78 → 47.69 on Video-CraftBench.

    On Video-CraftBench, where every model gets only the goal, the lead over MiniMax-H3 is large and significant (p<0.001). A human study with high rater agreement (Kendall's W = 0.96) gives the same ranking.

  2. WorldGuide Bench fills a real gap in data. As Table 1 shows, existing instructional datasets do not pair atomic action captions with their clips plus an explicit completion signal at this scale and breadth (59K videos, 245 tasks). Releasing the annotations, splits, prompts and code would give the community a training and evaluation resource for goal-directed video world models.

Critical weaknesses
  1. The headline comparison does not isolate the claimed contribution, and the paper's own ablation cuts against it.

    • WorldGuide's executor is fine-tuned on in-distribution data: the same 245 tasks and the same Gemini-written caption style as the test set. All baselines in Table 2 are zero-shot.
    • The fair reference point is the same backbone. Zero-shot HunyuanVideo-1.5 given the ground-truth plan reaches 12.00%. The fine-tuned executor given the same ground-truth plan, executed open-loop and stopping after the last reference action (the "oracle ContextPlanner", Table 4), reaches 38.82%. That is higher than full closed-loop WorldGuide at 33.33% (p=0.011).
    • So most of the gain over baselines comes from in-domain executor fine-tuning (+26.8 points). With a good plan, open-loop execution beats closing the loop. This is hard to square with the central message that "closing the loop is what matters" (Conclusion).
    • The +21.6-point "closed-loop vs open-loop" result compares against a weak open-loop variant: a planner trained for next-step prediction is asked to emit the whole plan at once. The necessary baseline is missing: the same fine-tuned executor plus a strong open-loop planner, such as a frontier VLM or the ContextPlanner retrained to output full plans. Fine-tuning at least one competing generator on WorldGuide Bench would also help.
    • The methodological novelty on its own is modest: plain SFT of the planner, sequential teacher-forced training rather than joint or on-policy training, and an adopted memory scheme. So the paper's case rests on the empirical claim above, and that claim is not established.
  2. The evaluation is circular and only partly validated, and some metrics conflict with the claims.

    • The same model family (Gemini) filters the data, writes the training targets that the executor learns to render, and judges outputs against a Gemini-written reference plan with Gemini 3.5 Flash. This favours a model trained to reproduce Gemini-style captions and state changes, and zero-shot baselines are disadvantaged.
    • About 17% of annotations are judged incorrect (Table 9), which adds noise to both training targets and reference plans.
    • The paper claims that visual feedback and learned stopping produce "less redundant procedures". Yet WorldGuide has among the worst Repeat/Skip scores: 6.06 vs 3.00 for HunyuanVideo-1.5 on WorldGuide Bench, and 17.08 on Video-CraftBench, worse than every model except LTX-2.5. Its open-loop variant scores 0.49. Rollouts that run longer, until DONE or the step budget, may also inflate the chance that the final-state Task Success check passes. Rollout lengths are not compared with the baselines' fixed lengths.
    • Please report:
      • human or second-family judgments of Task Success on a larger sample, so judge agreement can be measured;
      • generated-length statistics;
      • a test split of unseen tasks, to show the system generalises beyond held-out videos of seen tasks.
Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer BDB2 (1/3)

We thank the reviewer for the thorough review and for recognizing our closed-loop formulation and WorldGuide Bench. We address each point below with new experiments.

Q1:

(i) In-distribution fine-tuning. Neither WorldGuide nor any baseline is trained on Video-CraftBench. There, all models receive only the initial image and goal, and the judge evaluates sub-goal completion without any reference plan (App. E.1), so no Gemini-written captions are involved at input or evaluation. WorldGuide still achieves the highest Task Success, 47.69% vs. 32.73% for the 4× larger MiniMax-H3 (Table 3). On WorldGuide Bench, the zero-shot baselines receive the ground-truth action plan, while WorldGuide receives only the goal and must plan for itself (Table 2). The strong closed-loop baselines (PhysAgent, TempAct, Bernini), which also plan from the goal, remain far below on both benchmarks. Closing the loop is therefore not enough. It must be coupled with an Executor trained to realize the planned actions.

(ii) Oracle vs. WorldGuide. The oracle is given the ground-truth actions and their length, so it never plans or decides when to stop. We include it to measure how much error comes from planning, so it is an upper bound rather than a deployable alternative (Sec. 4.5). WorldGuide does both from the goal alone and still reaches 33.33%, compared with 38.82% for the oracle. In Limitation Sec. 5 discusses how a stronger planner and tighter coupling can close this gap. With the same planner and Executor, closed-loop WorldGuide reaches 33.33% versus 11.71% open-loop (Table 4). The loop helps even the zero-shot Executor, raising goal-only HunyuanVideo-1.5 from 3.16% to 9.90% on WorldGuide Bench and from 24.15% to 28.43% on Video-CraftBench. We will revise the Conclusion to state this clearly.

Gain from Executor training. The same-backbone gain from 12.00% to 38.82% is the contribution we claim, not a confound. Coupling planning with an Executor trained to realize the planned actions is the central idea of the paper (Sec. 1, Fig. 1). Our new control confirms this inside the closed loop. With the same ContextPlanner and rollout, the zero-shot Executor reaches 9.90% while our trained Executor reaches 33.33% on WorldGuide Bench, and 28.43% versus 47.69% on Video-CraftBench (Rebuttal Table 1).

\begin{array}{lccccc|ccccc} Variant & Aesth. & Imag. & Dynamic & Smooth. & Consist. & Align. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline WorldGuide Bench \\ HunyuanVideo-1.5(Goal Only) & 43.64 & 66.28 & 90.82 & 96.52 & 19.26 & 56.97 & 18.68 & 34.85 & 1.03 & 3.16 \\ HunyuanVideo-1.5(Oracle Plan) & 42.43 & 65.57 & 88.84 & 96.00 & 18.91 & 62.28 & 43.53 & 80.00 & 3.00 & 12.00 \\ HunyuanVideo-1.5 + ContextPlanner & 38.24 & 64.46 & 85.82 & 95.42 & 15.34 & 60.56 & 40.22 & 78.58 & 18.26 & 9.90 \\ WorldGuide & 49.36 & 64.77 & 98.06 & 98.96 & 31.13 & 73.85 & 62.85 & 92.42 & 6.06 & 33.33 \\ \hline Video-CraftBench \\ HunyuanVideo-1.5(Goal Only) & 45.30 & 67.62 & 99.66 & 98.91 & 21.91 & 63.94 & 14.66 & 64.97 & 3.06 & 24.15 \\ HunyuanVideo-1.5 + ContextPlanner & 48.54 & 69.88 & 99.26 & 98.82 & 20.38 & 69.62 & 26.28 & 82.74 & 6.34 & 28.43 \\ WorldGuide & 40.19 & 64.63 & 99.65 & 98.86 & 22.09 & 71.80 & 57.42 & 94.45 & 17.08 & 47.69 \\ \hline \end{array}

Rebuttal Table 1. Zero-shot vs. trained Executor on the same backbone. Rows with “+ ContextPlanner” and WorldGuide share the same planner and closed loop.

(iii) Open-loop vs. closed-loop. This is addressed in (ii). The oracle plan and WorldGuide use the same Executor and memory, so the oracle is better only because it is given the ground-truth plan, which are never available real time setting. Without them, open-loop execution drops to 11.71% while closing the loop reaches 33.33%, and the loop raises even the zero-shot Executor from 3.16% to 9.90% (WorldGuide Bench) and from 24.15% to 28.43% (Video-CraftBench). Both parts of our claim therefore hold. Executor training makes a good plan executable, and closing the loop lets that plan be predicted from progress when it is not given.

AuthorsOpenReview ↗

Response to Reviewer BDB2 (2/3)

(iv) The open-loop variant asks the planner for a full plan, which differs from its next-step training. The w/o-visual variant (Table 4) avoids this mismatch. It uses the planner in its trained step-by-step mode but cannot see the generated clips, so it plans blind to execution. It reaches 14.72% compared with 33.33% for WorldGuide.

A frontier VLM planner alone is also not enough without a trained Executor. We evaluated CollabVR on Video-CraftBench, where Gemini 3.5 Flash acts as planner and verifier with retries over VBVR-Wan2.2. It reaches 16.33% Task Success compared with 47.69% for WorldGuide, even though its planner shares a model family with our judge (Rebuttal Table 2). A frontier VLM full-plan baseline with our Executor is a fair request, and we will add it in the final version.

\begin{array}{lcccccccccc} Model & Aesth. & Imag. & Dynamic & Smooth. & Consist. & Align. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline CollabVR & 40.30 & 62.50 & 88.10 & 99.00 & 19.20 & 74.26 & 26.95 & 91.35 & 0.34 & 16.33 \\ WorldGuide (ours) & 40.19 & 64.63 & 99.65 & 98.86 & 22.09 & 71.80 & 57.42 & 94.45 & 17.08 & 47.69 \\ \end{array}

Rebuttal Table 2. Comparison with CollabVR on Video-CraftBench under the Table 3 protocol.

Fine-tuning another generator is not feasible during rebuttal, as our Executor took 42 days on 64 GPUs (App. D.2). Instead, we ran zero-shot HunyuanVideo-1.5 with the same ContextPlanner and loop. It reaches 9.90%, while our trained Executor reaches 33.33%, which shows how much the Executor training adds.

(v) Novelty. The simplicity is deliberate. The novelty is in the formulation, not in new training components. WorldGuide formulates procedural video generation as closed-loop task execution, where one planner decides both the next action and when the task is complete from what has actually been generated (Sec. 3.1). This requires three things that no prior work combines: a planner grounded in generated progress, an Executor trained on the same atomic actions that planner emits (Sec. 3.3), and stopping learned from visual progress (Sec. 4.4). WorldGuide Bench is the first dataset that provides this supervision with explicit completion labels (Table 1).

Each design choice serves this formulation. Sequential training with a frozen planner keeps the planning–execution interface in natural language (App. A), so every step is interpretable, failures can be traced to planning or execution, and plans can be evaluated independently (Table 4 oracle, Table 5). We adopt the memory scheme and state this clearly (Sec. 3.4).

The new experiments support the empirical claim. In the same closed loop, Executor training lifts Task Success from 9.90% to 33.33% (with memory) on WorldGuide Bench (Rebuttal Table 1). The planner recovers from injected failures using visual evidence alone, completing 46/50 (WorldGuide Bench) and 47/50 (Video-CraftBench) tasks (Reviewer jYyZ, Q3). It stops by itself in 94% of rollouts, with only 6% hitting the step cap. On Video-CraftBench, CollabVR, a frontier VLM loop over an untrained Executor, reaches 16.33% versus our 47.69% (Rebuttal Table 2).


Q2:

(i) Judge circularity. We re-evaluated the strongest models with a judge from a different model family, GLM-5.3 Flash, using the same prompt and protocol (Rebuttal Table 3). The ranking is unchanged, and WorldGuide remains best on both benchmarks. Its margin over MiniMax-H3 actually grows under the non-Gemini judge, from 3.4 to 6.1 points on WorldGuide Bench and from 15.0 to 16.7 points on Video-CraftBench. So the gain is not an artifact of Gemini judging Gemini-style outputs.

On Video-CraftBench, no Gemini-written captions or reference plans are used at all. Models receive only the image and goal, and the judge evaluates the goal directly (App. E.1). On WorldGuide Bench, the baselines are given the Gemini-written reference plan itself as input, so they receive the same caption style. The independent human study also ranks WorldGuide first (App. F).

\begin{array}{l|cccc|cccc} \hline & Gemini 3.5 Flash & & & & GLM-5.3 Flash & & & \\ Model & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline WorldGuide Bench & & & & & & & & \\ MiniMax-H3 & 57.99 & 86.79 & 4.06 & 29.90 & 59.24 & 89.92 & 3.57 & 30.82 \\ TempAct* & 9.11 & 29.68 & 0.50 & 3.98 & 23.61 & 35.28 & 5.71 & 7.96 \\ WorldGuide (ours) & 62.85 & 92.42 & 6.06 & 33.33 & 66.49 & 93.46 & 5.52 & 36.91 \\ \hline Video-CraftBench & & & & & & & & \\ MiniMax-H3 & 28.64 & 88.55 & 2.16 & 32.73 & 47.78 & 90.36 & 5.10 & 36.39 \\ TempAct* & 7.57 & 51.26 & 0.34 & 24.47 & 9.18 & 58.52 & 1.70 & 31.97 \\ WorldGuide (ours) & 57.42 & 94.45 & 17.08 & 47.69 & 60.48 & 94.51 & 14.28 & 53.06 \\ \hline \end{array}

Rebuttal Table 3. Video-judge metrics under Gemini 3.5 Flash and GLM-5.3 Flash, with identical prompts and protocol.

AuthorsOpenReview ↗

Response to Reviewer BDB2 (3/3)

(ii) Annotation noise. The 17% comes from a strict audit. A step counts as wrong if any one of its action, objects, or result is wrong, and the three annotators agree within 1.3 points on every criterion (App. B.6, Table 9). It's cover all 245 tasks, and each task is trained on many demonstrations (about 235 per task on average, App. B.4), so no single incorrect annotation dominates what the models learn for a task.

For evaluation, the reference plan only guides the judge. The judge accepts equivalent actions, different step sizes, and valid reorderings (App. E.2), and Task Success only checks whether the final video shows the task completed (Eq. 13). Our stopping statistics in (iv) confirm this: rollouts shorter or longer than the reference still succeed 21.8% and 41.6% of the time. All models are scored against the same reference plans, so any noise affects them equally. Video-CraftBench also uses none of our annotations, and WorldGuide’s lead there is even larger, at 47.69% versus 32.73% for MiniMax-H3 (Table 3).

(iii) Repeat/Skip. The "less redundant" claim in Sec. 4.4 compares WorldGuide with its own w/o-visual variant (Table 5, 20.19 → 15.73), not with the baselines, and we will make this clear in the manuscript. Repeat/Skip only counts repeated actions that add no progress, while skipped steps are penalized in Plan Accuracy instead (Eq. 12, App. E.2). A model that does little therefore scores low almost automatically. The open-loop variant has a Repeat/Skip of 0.49 with a Plan Accuracy of only 22.87 (Table 4), and HunyuanVideo-1.5 has a Repeat/Skip of 3.00 with a Plan Accuracy of 43.53 (Table 2), while WorldGuide reaches a Plan Accuracy of 62.85. Part of WorldGuide's Repeat/Skip also comes from recovery, where the planner repeats a step whose generated clip shows it was not completed (Reviewer jYyZ, Q3).

(iv) Rollout length. The unstarred baselines on WorldGuide Bench always run the reference length because they are given the reference plan, while WorldGuide decides its own length. The table compares WorldGuide’s step count with the reference for all 978 judged rollouts on WorldGuide Bench (Video-CraftBench has no reference step counts).

\begin{array}{lcccc} Planner stop & Share & Videos & Successes & Success rate \\ \hline Correct stop (= reference) & 10.0\% & 98 & 30 & 30.6\% \\ Early stop (< reference) & 28.6\% & 280 & 61 & 21.8\% \\ Late stop (> reference) & 55.3\% & 541 & 225 & 41.6\% \\ Hit 80-step cap & 6.0\% & 59 & 10 & 16.9\% \\ \hline \textbf{Total} & 100\% & 978 & 326 & 33.33\% \\ \end{array}

Rebuttal Table 4. Stopping behavior and Task Success on WorldGuide Bench.

Longer rollouts cannot inflate Task Success, because a rollout passes only if its final video shows the task completed (Eq. 13). The longest rollouts, those that hit the 80-step cap, succeed only 16.9% of the time. The other 919 rollouts stop by predicting DONE, and 316 of them (34.4%) end with the task visibly complete. Longer rollouts succeed more often because the planner redoes steps that visibly failed, but each success still needs a completed final state. Step count alone also does not define a stopping error, since a task can be completed in more or fewer atomic steps than the reference and the judge accepts different granularities and valid alternative procedures (App. E.2). This is why shorter rollouts still succeed 21.8% of the time.

Unseen tasks. Our test split holds out unseen demonstrations of the 245 known tasks (App. B.4). Video-CraftBench is an external, fully egocentric benchmark with its own initial images, objects, and scenes, and none of its data is used in training. All models receive only the image and goal there, and WorldGuide leads with 47.69% Task Success, compared with 32.73% for MiniMax-H3 and 16.33% for CollabVR.

We hope these new results resolve your concerns, and we are happy to provide any further details you may need.

Reviewer_KAEAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorldGuide is capable of generating process videos based on an initial image and a task goal. Within the system, a planner observes the previously generated video segments to select the next action, while an executor generates the corresponding video segment. The planner also determines when the generation process should conclude. The paper proposes a dataset for training these two modules and evaluate the system using two benchmarks, with ablation studies and human evaluations.

Critical strengths
  1. This task is well-motivated. In procedural generation, actions may not always be executed as intended; therefore, it is logical to verify the generated output before determining the subsequent operation.
  2. Ablation studies strongly support the effectiveness of visual feedback. Performance degrades significantly when the planner cannot observe the generated segments. External benchmarks and human evaluations further corroborate these findings.
  3. This dataset represents a valuable contribution. By mapping task goals to specific action segments and completion signals, it is suitable for both planning and video generation tasks.
Critical weaknesses
  1. The experiments fail to isolate and evaluate the specific gains provided by the "learned executor." Keeping the planner fixed and comparing pretrained and fine-tuned executors would help establish how much this part contributes.
  2. The evidence for knowing when to stop is less complete. The mere fact that text-based planning is effective does not prove that the generated video has successfully achieved the intended goal.
  3. The generalization capability is limited. Although the primary test set includes new demonstrations for the tasks used during training, the external benchmarks are heavily skewed toward tasks such as "assembling" or "folding." It remains unclear how the proposed method would perform when faced with entirely novel processes.
Questions for authors
  1. Did you conduct any experiments comparing a pre-trained executor with a fine-tuned executor, while keeping the planner fixed?
  2. How often does the system stop too early or continue after completing the tas
  3. To what extent do the tasks in the training dataset overlap with the tasks in Video-CraftBench?
5 comments
AuthorsOpenReview ↗

Response to Reviewer KAEA

We thank the reviewer for the positive feedback and for recognizing the value of our formulation and WorldGuide Bench. We address each question below.

Q1. Pretrained vs. trained Executor. Yes. We ran this control with the planner fixed and only the Executor changed (Rebuttal Table 1).

With our ContextPlanner and the same closed-loop rollout, replacing the pretrained HunyuanVideo-1.5 with our trained Executor raises Task Success from 9.90% to 33.33% on WorldGuide Bench, and from 28.43% to 47.69% on Video-CraftBench. With the action sequence fixed to the ground-truth plan, the same change raises Task Success from 12.00% to 38.82% (Table 4, oracle). Plan Accuracy improves in the same way in both settings (40.22 → 62.85 and 43.53 → 65.42). The learned Executor is therefore responsible for a large, consistent gain, whether the actions come from our planner or from the ground truth.

\begin{array}{lccccc|ccccc} Variant & Aesth. & Imag. & Dynamic & Smooth. & Consist. & Align. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline WorldGuide Bench \\ HunyuanVideo-1.5 (Goal Only) & 43.64 & 66.28 & 90.82 & 96.52 & 19.26 & 56.97 & 18.68 & 34.85 & 1.03 & 3.16 \\ HunyuanVideo-1.5 (GT Plan) & 42.43 & 65.57 & 88.84 & 96.00 & 18.91 & 62.28 & 43.53 & 80.00 & 3.00 & 12.00 \\ WorldGuide (GT Plan, Oracle) & 52.44 & 66.72 & 97.68 & 99.49 & 33.37 & 77.14 & 65.42 & 94.28 & 2.04 & 38.82 \\ HunyuanVideo-1.5 + ContextPlanner & 38.24 & 64.46 & 85.82 & 95.42 & 15.34 & 60.56 & 40.22 & 78.58 & 18.26 & 9.90 \\ WorldGuide & 49.36 & 64.77 & 98.06 & 98.96 & 31.13 & 73.85 & 62.85 & 92.42 & 6.06 & 33.33 \\ \hline Video-CraftBench \\ HunyuanVideo-1.5 (Goal Only) & 45.30 & 67.62 & 99.66 & 98.91 & 21.91 & 63.94 & 14.66 & 64.97 & 3.06 & 24.15 \\ HunyuanVideo-1.5 + ContextPlanner & 48.54 & 69.88 & 99.26 & 98.82 & 20.38 & 69.62 & 26.28 & 82.74 & 6.34 & 28.43 \\ WorldGuide & 40.19 & 64.63 & 99.65 & 98.86 & 22.09 & 71.80 & 57.42 & 94.45 & 17.08 & 47.69 \\ \hline \end{array}

Rebuttal Table 1. Pretrained vs. trained Executor with the planner fixed.


Q2. Stopping on generated videos. We measured stopping on the generated videos, not on the predicted text plans. Gemini marks a video as successful only if its final frames show the task completed (Eq. 13). Rebuttal Table 2 compares WorldGuide's number of steps with the reference number of steps on the WorldGuide Bench test set. Video-CraftBench is not included because it has no reference steps.

\begin{array}{lcccc} Planner stop & Share & Videos & Successes & Success rate \\ \hline Correct stop (= reference) & 10.0\% & 98 & 30 & 30.6\% \\ Early stop (< reference) & 28.6\% & 280 & 61 & 21.8\% \\ Late stop (> reference) & 55.3\% & 541 & 225 & 41.6\% \\ Hit 80-step cap & 6.0\% & 59 & 10 & 16.9\% \\ \hline \textbf{Total} & 100\% & 978 & 326 & 33.33\% \\ \end{array}

Rebuttal Table 2. Stopping statistics on the WorldGuide Bench test set.

Only 59 rollouts (6.0%) reach the 80-step cap. The other 919 stop by predicting DONE, and 316 of them (34.4%) end with the task visibly complete in the generated video. Step count alone does not tell whether a stop was too early or too late, because a task can be completed in more or fewer atomic steps than the reference, and the judge accepts different granularities and valid alternative procedures (App. E.2). Rollouts shorter than the reference still succeed 21.8% of the time, and longer ones 41.6%. Whether the planner stopped correctly is therefore decided by the final visible state, which is exactly what Task Success measures. When a step visibly fails, the planner continues and redoes it instead of stopping (Reviewer jYyZ, Q3).


Q3. Overlap with Video-CraftBench. Both datasets include block building and paper folding, but the overlap is only at the category level, and the data itself is very different. WorldGuide Bench is collected from YouTube tutorials that mix egocentric and exocentric views with varied quality and settings. Video-CraftBench is entirely egocentric, recorded at high resolution in its own setups. The block tasks also differ in kind. Our training data mostly shows LEGO-style construction, while Video-CraftBench asks the model to arrange simple blocks into figures such as a horse or a house (compare Fig. 13 from WorldGuide Bench with Fig. 16 from Video-CraftBench). None of its videos, images, or annotations are used in training, and models receive only the initial image and goal, with no reference plan. WorldGuide therefore has to plan from unseen viewpoints, objects, and initial states, and it still leads with 47.69% Task Success, compared with 32.73% for MiniMax-H3. Evaluation on entirely new procedures is not covered by our current benchmarks. Our test split holds out unseen demonstrations of known tasks (App. B.4).

We hope these results answer your questions, and we would be glad to clarify anything further concerns.

AuthorsOpenReview ↗

Follow-up on Reviewer Engagement with Rebuttal

Dear Reviewer KAEA,

We sincerely appreciate the opportunity to address your comments. We understand you may have been busy, but we would be grateful if you could consider re-engaging with the discussion. We hope our responses and new experiments address your questions, and we would greatly welcome any additional feedback you might have. If these results address your main concerns, we would be grateful if you could consider raising your score.

Thank you for your time and efforts in supporting the review process.

Authors

Dear Reviewer KAEA,

As only a few hours remain in the discussion period, we wanted to follow up once more. If there are any remaining concerns or additional results that would help you raise your score, please let us know, and we will be happy to address them as soon as possible.

Thank you again for your time and efforts in supporting the review process.

Authors

Reviewer_jYyZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

WorldGuide generates goal-directed procedural videos through a planner–executor loop with visual feedback. It trains both modules on shared step-level demonstrations and uses learned completion prediction and compressed visual memory. Experiments on WorldGuide Bench and Video-CraftBench report improved task success over the evaluated baselines.

Critical strengths

The framework is well motivated for goal-directed procedural video generation. It uses generated visual feedback to choose the next action and decide when to stop. The method shows promising task-level performance

Critical weaknesses

W1. The differences and contributions to the prior work are not clearly explained.

  • The paper's main contribution is a learned closed loop (Secs 1-2.2). CollabVR already has progressive planning, visual feedback and completion decisions. SPIRAL also directly fine tunes its video generator and that point is missed in Sec 1.
  • WorldGuide emphasizes training/fine-tuning the semantic planner and the video frame generator (the executor), which is intuitive and natural. It doesn't convincingly demonstrate any methodological advance over prior works. The novelty is therefore very questionable.

W2. The evidence does not clearly isolate each component's contribution.

  • The reported gains are due to task-specific training, closed loop planning and visual memory. The ablations do confirm the contribution of various components, though they don't isolate the effect of the Executor fine-tuning from the Planner, feedback loop and memory which are all held constant (Secs 4.2-4.5; Tables 4-5).
  • The results do not clearly show how much of the improvement is due to direct Executor training. This provides very little support for the central claim of "planning coupled with learned execution".
Questions for authors

Q1. What is WorldGuide’s core new contribution relative to the closest work?

  • CollabVR already does step-by-step planning, uses visual feedback and has a mechanism for determining when a task is complete. SPIRAL also trains its planner and fine tunes its video generator via direct supervision (SFT prior to RL). The Introduction and section 2.2 don't do enough to highlight these similarities. This is particularly true of the claim that sets SPIRAL apart from direct execution supervision.
  • Which of the following is the key new contribution: (i) supervised training of the completion decision, (ii) using the Planner as the Executor's text encoder or (iii) training both modules on the same demonstrations? This work need to be very clear as to how these designs differ from the prior art.
  • Why aren't CollabVR and SPIRAL included in the experimental comparison? Are there any comparable results to back up the claimed advantage of the proposed methods over these? (Tables 2-3; App. E.1)

Q2. How much independent gain comes from training the Executor?

  • The benefits of the full system come from targeted training, closed-loop planning and visual memory all working together. The ablations that have been done so far look at some of the individual components but don't isolate the benefit from Executor fine-tuning (Table 4). The closest comparison is zero-shot HunyuanVideo-1.5 versus the oracle (Tables 2 and 4). That comparison also involves a change to memory, text conditioning and the rollout protocol.
  • Is there a control that fixes the Planner (or the action sequence) and the closed loop procedure and just changes whether or not the Executor is fine tuned? The original Executor has no memory branch so memory would have to be turned off in both runs. This result is directly relevant to the core claim of 'coupling planning with learned execution'.
  • Is "using the same demonstrations" primarily a matter of how the data is structured, or is it a necessary coupling design? What evidence currently supports the latter? (

Q3. How does the training supervision support responses to execution failures?

  • Both modules are trained solely on real demonstrations (§3.2; App. D). The checks in App. B, the completeness filtering and the human audit all apply only to these source videos. The planner therefore never sees the Executor's own failures, partial completions or recoveries or the next action labels in such cases.
  • Which of the results show that the planner is adapting its actions to actual execution deviations rather than just following the typical step order of the demonstrations? Can the visual feedback ablation distinguish between these two cases? In that version the action history is also restricted to the last three actions (Eq. 1) so the planner loses the progress information as well as the visual feedback. (§3.2; Table 4)

Q4. Is the completion decision independently verified?

  • The completion label in training comes from the final step of each demonstration (App. B.2-B.3). At deployment the Executor may produce the final action without in fact completing the task. Can the model correctly decide to continue in this scenario?
  • Are there any stats on early stopping, late stopping and correct stopping as measured on the generated videos? The text-based Plan success in Table 5 doesn't really answer that. App. E.3 does suggest that stopping errors are visible in Task Success and Repeat/Skip, though those metrics are a mixture of stopping errors and execution errors. How many rollouts actually hit the 80 step cap (App. D.3)? The completion decision is a fundamental claimed capability so it needs some direct evidence.
5 comments
AuthorsOpenReview ↗

Response to Reviewer jYyZ (1/2)

We thank the reviewer for the careful reading and insightful comments. We address each point below with new experiments.

Q1. Core contribution relative to CollabVR and SPIRAL

(i) Similarities. SPIRAL does fine-tune its generator jointly (sec. 2.2). The key difference is how each system decides what to do next and when to stop. SPIRAL predicts the whole action plan once, at the start of generation. A separate critic checks each step and retries it if it fails, but the plan itself changes only after repeated failures, and generation ends after the last planned step. CollabVR is training-free. Gemini acts as both planner and verifier, with retries, over an off-the-shelf generator, and its completion check is a prompted, zero-shot flag. WorldGuide's planner instead observes every generated clip and decides the next atomic action, or that the task is complete (Sec. 3.1–3.2, 4.4).

(ii) Key contribution. It is (1) The formulation. WorldGuide treats procedural video generation as closed-loop task execution, where one planner decides both the next action and when the task is complete from what has actually been generated (Sec. 3.1, Sec. 4.4). (2) A training recipe for this closed-loop architecture. The planner also serves as the Executor's text encoder, so the Executor learns the same actions the planner predicts. Each actions is an interpretable language instruction, so users can easily follow what the model is doing (Sec. 3.3, App. A). (3) WorldGuide Bench, the first public dataset that provides this supervision with explicit completion labels (Table 1).

(iii) Experimental comparison. SPIRAL's code and checkpoints are not publicly available, so a direct comparison is not possible. We initially omitted CollabVR due to its cost (2–12 Gemini calls per sample even with a 3-step cap) and compared against public closed-loop systems (Bernini, TempAct, PhysAgent) and MiniMax-H3 (Tables 2–3). We have now evaluated CollabVR on Video-CraftBench, which neither method is trained on, under the Table 3 protocol with Gemini 3.5 Flash as planner and verifier over VBVR-Wan2.2. WorldGuide reaches 47.69% Task Success versus 16.33% for CollabVR (Rebuttal Table 1).

\begin{array}{lcccccccccc} Model & Aesth. & Imag. & Dynamic & Smooth. & Consist. & Align. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline CollabVR & 40.3 & 62.50 & 88.1 & 99.0 & 19.2 & 74.26 & 26.95 & 91.35 & 0.34 & 16.33 \\ WorldGuide (Ours) & 40.19 & 64.63 & 99.65 & 98.86 & 22.09 & 71.80 & 57.42 & 94.45 & 17.08 & 47.69 \\ \end{array}

Rebuttal Table 1. Comparison with CollabVR on Video-CraftBench under the Table 3 protocol.


Q2. Independent gain from training the Executor

(i & ii) Control. We fix the ContextPlanner and closed loop, disable memory in both runs, and change only the Executor: zero-shot HunyuanVideo-1.5 (with its original text encoder) versus our fine-tuned Executor ("w/o Mem", Table 4). All other settings are identical (Rebuttal Table 2).

Training the Executor raises Task Success from 9.90% to 27.56% and Plan Accuracy from 40.22 to 55.36 on WorldGuide Bench. On Video-CraftBench, Plan Accuracy rises from 26.28 to 39.62 and Task Success from 28.43% to 29.78%. The memory branch, which the base model lacks, is learned only through Executor fine-tuning (App. D.2) and lifts Task Success to 47.69%. The pretrained model also shows why training is needed. With it, our planner nearly matches the ground-truth plan (9.90% vs. 12.00%) and triples its goal-only result (3.16%), yet the pretrained Executor fails to realize most planned atomic actions (Sec. 1, Fig. 1).

\begin{array}{l|ccccc|ccccc} Variant & Aesth. & Imag. & Dynamic & Smooth. & Consist. & Align. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline WorldGuide Bench \\ HunyuanVideo-1.5(Goal Only) & 43.64 & 66.28 & 90.82 & 96.52 & 19.26 & 56.97 & 18.68 & 34.85 & 1.03 & 3.16 \\ HunyuanVideo-1.5(Oracle Plan) & 42.43 & 65.57 & 88.84 & 96.00 & 18.91 & 62.28 & 43.53 & 80.00 & 3.00 & 12.00 \\ HunyuanVideo-1.5 + ContextPlanner & 38.24 & 64.46 & 85.82 & 95.42 & 15.34 & 60.56 & 40.22 & 78.58 & 18.26 & 9.90 \\ WorldGuide (w/o Mem) & 38.70 & 55.35 & 88.35 & 98.14 & 19.70 & 64.74 & 55.36 & 91.12 & 4.39 & 27.56 \\ \hline Video-CraftBench \\ HunyuanVideo-1.5(Goal Only) & 45.30 & 67.62 & 99.66 & 98.91 & 21.91 & 63.94 & 14.66 & 64.97 & 3.06 & 24.15 \\ HunyuanVideo-1.5 + ContextPlanner & 48.54 & 69.88 & 99.26 & 98.82 & 20.38 & 69.62 & 26.28 & 82.74 & 6.34 & 28.43 \\ WorldGuide (w/o Mem) & 38.82 & 63.81 & 99.63 & 98.64 & 20.21 & 73.77 & 39.62 & 92.51 & 4.10 & 29.78 \\ \hline \end{array}

Rebuttal Table 2. Pretrained vs. fine-tuned Executor with the same ContextPlanner and closed loop, memory off in both.

AuthorsOpenReview ↗

Response to Reviewer jYyZ (2/2)

(iii) Same demonstrations. It is a coupling design, not just data structuring. The Executor is trained on the same atomic actions, encoded by the planner itself (Sec. 3.3, App. A), so the planner's outputs are exactly the Executor's training inputs. The control above supports this. Given identical planner outputs, the zero-shot Executor, never trained on these actions, reaches 9.90% Task Success versus 27.56% for ours on WorldGuide Bench. Learning the planner’s own atomic actions is what lets the Executor realize them.


Q3. Response to execution failures

(i) Training on real demonstrations teaches the planner to predict the next action from the visual state, not the action history alone (Sec. 3.2). A failed step leaves the scene as if the step never happened, a state the planner has learned to act on, so it requests the missing step again without ever seeing failures in training. To test this, we take 50 successful rollouts per benchmark and inject a failure at step 5. The action history is unchanged (steps 3, 4, 5, still claiming step 5 was done), but the step-5 clip is replaced with the step-4 clip (clips 3, 4, 4), and the rollout continues. Since the history is identical, recovery can come only from the visual evidence.

WorldGuide recovers and completes 46/50 tasks on WorldGuide Bench and 47/50 on Video-CraftBench, with Plan Accuracy nearly unchanged (Rebuttal Table 3). Repeat/Skip rises because the failed step appears again in the video. The planner therefore responds to the actual generated state rather than the demonstration order, despite training only on real demonstrations.

\begin{array}{lccccc|ccccc} Variant & Aesth. & Imag. & Dynamic & Smooth. & Consist. & Align. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline WorldGuide Bench \\ WorldGuide* & 36.2 & 60.7 & 93.6 & 95.2 & 18.9 & 64.1 & 96.6 & 98.8 & 14.0 & 92.0 \\ WorldGuide & 36.7 & 60.5 & 94.0 & 95.6 & 19.2 & 64.4 & 97.6 & 100.0 & 4.0 & 100.0 \\ \hline Video-CraftBench \\ WorldGuide* & 40.2 & 65.9 & 99.2 & 98.2 & 20.8 & 73.6 & 97.4 & 98.4 & 12.0 & 94.0 \\ WorldGuide & 40.8 & 66.6 & 100.0 & 98.9 & 21.4 & 75.1 & 100.0 & 100.0 & 8.0 & 100.0 \\ \hline \end{array}

Rebuttal Table 3. Failure injection on 50 originally successful rollouts per benchmark. WorldGuide* replaces the step-5 clip with the step-4 clip while keeping the action history unchanged.

(ii) The failure-injection test in (i) shows that the planner reacts to what actually happened, not to a fixed step order. With identical action histories, the video is the only signal that step 5 failed, and the planner still recovers in 46/50 and 47/50 cases. The w/o-visual ablation (Table 4) shows the visual observation is importent. We used K=3 because longer clip histories exceed GPU memory, and a compact visual memory in the planner caused a vision–language alignment problem that would require large-scale pretraining.


Q4. Completion decision

(i) The planner predicts DONE like any other action, from the generated clips (App. B.3). If the final action was not actually completed, the video still shows unfinished work, so the planner keeps going instead of stopping. Our failure-injection test in Q3 shows this behavior. When the history says a step was done but the video shows it was not, the planner redoes the step and completes 46/50 and 47/50 tasks. Rollouts that continue past the reference length also have the highest success rate (41.6%, Rebuttal Table 4).

(ii) Rebuttal Table 4 compares the planner’s step count with the reference on the WorldGuide Bench test set (Video-CraftBench has no reference step counts).

\begin{array}{lcccc} Planner stop & Share & Videos & Successes & Success rate \\ \hline Correct stop (= reference) & 10.0\% & 98 & 30 & 30.6\% \\ Early stop (< reference) & 28.6\% & 280 & 61 & 21.8\% \\ Late stop (> reference) & 55.3\% & 541 & 225 & 41.6\% \\ Hit 80-step cap & 6.0\% & 59 & 10 & 16.9\% \\ \hline \textbf{Total} & 100\% & \textbf{978} & \textbf{326} & \textbf{33.33\%} \\ \end{array}

Rebuttal Table 4. Stopping statistics on WorldGuide Bench. Success is judged from the final visible state of each generated video.

Only 59 rollouts (6.0%) hit the 80-step cap. The other 919 stop by predicting DONE, and 316 of these (34.4%) end with the task visibly complete. Step count alone cannot define a stopping error, because a task can be completed in more or fewer atomic steps than the reference, and the judge accepts different granularities and valid alternative procedures (App. E.2). Shorter rollouts still succeed 21.8% of the time and longer ones 41.6%. Whether the planner stopped correctly is therefore decided by the final visible state, which is what Task Success measures (Tables 2–4).

We hope these new results address your concerns and clarify our contributions, and we would be happy to answer any further questions.

AuthorsOpenReview ↗

Follow-up on Reviewer Engagement with Rebuttal

Dear Reviewer jYyZ,

We sincerely appreciate the opportunity to address your comments. We understand you may have been busy, but we would be grateful if you could consider re-engaging with the discussion. We believe that some of the concerns raised could benefit from further clarification, and we would greatly welcome any additional feedback you might have.

Thank you for your time and efforts in supporting the review process.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.