- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper treats procedural video generation (cooking, origami, assembly) as closed-loop task execution. Given only an initial image and a task goal, WorldGuide loops as follows:
- A ContextPlanner (fully fine-tuned Qwen2.5-VL-7B) predicts the next atomic action, or a DONE token. It sees the goal and the last K=3 generated clips and actions.
- An Executor (fine-tuned HunyuanVideo-1.5, conditioned on the frozen planner's action embeddings) renders that action as a short clip.
- The new clip updates the planner's context and the Executor's hierarchical compressed memory. That memory schedule is adopted from YUME/FramePack.
The two modules are trained one after the other, with teacher forcing, on WorldGuide Bench. This is a new set of about 59K YouTube instructional videos (245 tasks, 27 categories), filtered and annotated by Gemini 2.5 Flash into 1–5 s atomic action clips with an explicit completion step. A human audit rates about 83% of annotations correct. The test split holds out 4 videos per task, so it covers unseen videos of seen tasks.
Evaluation is mostly by a Gemini 3.5 Flash judge (Task Success, Plan Accuracy, Order, Repeat/Skip) plus VBench metrics:
- WorldGuide Bench (980 videos): 33.33% Task Success. MiniMax-H3 gets 29.90%, but receives the ground-truth action plan.
- Video-CraftBench (294 samples, goal-only for all models): 47.69% vs 32.73%.
- Ablations: an open-loop variant gets 11.71% and a variant without visual feedback 14.72%. Removing memory gives 27.56%. Replacing the planner with oracle reference actions raises the score to 38.82%.
- Human study: 18 tasks, 4 raters; WorldGuide ranks first.
Critical strengths
-
A clear closed-loop formulation, with controlled ablations that support its main mechanisms. The planner decides both the next action and when to stop, based on generated visual state, and the executor is trained on the actions the planner proposes. This is a clean, well-motivated formulation compared with pairing a planner with a frozen video model. The within-model ablations (Table 4, with Wilson intervals and tests in Table 16) show large, significant effects:
- visual feedback for the planner: 14.72 → 33.33, p<0.001;
- executor memory: 27.56 → 33.33 on WorldGuide Bench and 29.78 → 47.69 on Video-CraftBench.
On Video-CraftBench, where every model gets only the goal, the lead over MiniMax-H3 is large and significant (p<0.001). A human study with high rater agreement (Kendall's W = 0.96) gives the same ranking.
-
WorldGuide Bench fills a real gap in data. As Table 1 shows, existing instructional datasets do not pair atomic action captions with their clips plus an explicit completion signal at this scale and breadth (59K videos, 245 tasks). Releasing the annotations, splits, prompts and code would give the community a training and evaluation resource for goal-directed video world models.
Critical weaknesses
-
The headline comparison does not isolate the claimed contribution, and the paper's own ablation cuts against it.
- WorldGuide's executor is fine-tuned on in-distribution data: the same 245 tasks and the same Gemini-written caption style as the test set. All baselines in Table 2 are zero-shot.
- The fair reference point is the same backbone. Zero-shot HunyuanVideo-1.5 given the ground-truth plan reaches 12.00%. The fine-tuned executor given the same ground-truth plan, executed open-loop and stopping after the last reference action (the "oracle ContextPlanner", Table 4), reaches 38.82%. That is higher than full closed-loop WorldGuide at 33.33% (p=0.011).
- So most of the gain over baselines comes from in-domain executor fine-tuning (+26.8 points). With a good plan, open-loop execution beats closing the loop. This is hard to square with the central message that "closing the loop is what matters" (Conclusion).
- The +21.6-point "closed-loop vs open-loop" result compares against a weak open-loop variant: a planner trained for next-step prediction is asked to emit the whole plan at once. The necessary baseline is missing: the same fine-tuned executor plus a strong open-loop planner, such as a frontier VLM or the ContextPlanner retrained to output full plans. Fine-tuning at least one competing generator on WorldGuide Bench would also help.
- The methodological novelty on its own is modest: plain SFT of the planner, sequential teacher-forced training rather than joint or on-policy training, and an adopted memory scheme. So the paper's case rests on the empirical claim above, and that claim is not established.
-
The evaluation is circular and only partly validated, and some metrics conflict with the claims.
- The same model family (Gemini) filters the data, writes the training targets that the executor learns to render, and judges outputs against a Gemini-written reference plan with Gemini 3.5 Flash. This favours a model trained to reproduce Gemini-style captions and state changes, and zero-shot baselines are disadvantaged.
- About 17% of annotations are judged incorrect (Table 9), which adds noise to both training targets and reference plans.
- The paper claims that visual feedback and learned stopping produce "less redundant procedures". Yet WorldGuide has among the worst Repeat/Skip scores: 6.06 vs 3.00 for HunyuanVideo-1.5 on WorldGuide Bench, and 17.08 on Video-CraftBench, worse than every model except LTX-2.5. Its open-loop variant scores 0.49. Rollouts that run longer, until DONE or the step budget, may also inflate the chance that the final-state Task Success check passes. Rollout lengths are not compared with the baselines' fixed lengths.
- Please report:
- human or second-family judgments of Task Success on a larger sample, so judge agreement can be measured;
- generated-length statistics;
- a test split of unseen tasks, to show the system generalises beyond held-out videos of seen tasks.
Questions for authors
See weaknesses.
5 comments
Response to Reviewer BDB2 (2/3)
(iv) The open-loop variant asks the planner for a full plan, which differs from its next-step training. The w/o-visual variant (Table 4) avoids this mismatch. It uses the planner in its trained step-by-step mode but cannot see the generated clips, so it plans blind to execution. It reaches 14.72% compared with 33.33% for WorldGuide.
A frontier VLM planner alone is also not enough without a trained Executor. We evaluated CollabVR on Video-CraftBench, where Gemini 3.5 Flash acts as planner and verifier with retries over VBVR-Wan2.2. It reaches 16.33% Task Success compared with 47.69% for WorldGuide, even though its planner shares a model family with our judge (Rebuttal Table 2). A frontier VLM full-plan baseline with our Executor is a fair request, and we will add it in the final version.
\begin{array}{lcccccccccc} Model & Aesth. & Imag. & Dynamic & Smooth. & Consist. & Align. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline CollabVR & 40.30 & 62.50 & 88.10 & 99.00 & 19.20 & 74.26 & 26.95 & 91.35 & 0.34 & 16.33 \\ WorldGuide (ours) & 40.19 & 64.63 & 99.65 & 98.86 & 22.09 & 71.80 & 57.42 & 94.45 & 17.08 & 47.69 \\ \end{array}
Rebuttal Table 2. Comparison with CollabVR on Video-CraftBench under the Table 3 protocol.
Fine-tuning another generator is not feasible during rebuttal, as our Executor took 42 days on 64 GPUs (App. D.2). Instead, we ran zero-shot HunyuanVideo-1.5 with the same ContextPlanner and loop. It reaches 9.90%, while our trained Executor reaches 33.33%, which shows how much the Executor training adds.
(v) Novelty. The simplicity is deliberate. The novelty is in the formulation, not in new training components. WorldGuide formulates procedural video generation as closed-loop task execution, where one planner decides both the next action and when the task is complete from what has actually been generated (Sec. 3.1). This requires three things that no prior work combines: a planner grounded in generated progress, an Executor trained on the same atomic actions that planner emits (Sec. 3.3), and stopping learned from visual progress (Sec. 4.4). WorldGuide Bench is the first dataset that provides this supervision with explicit completion labels (Table 1).
Each design choice serves this formulation. Sequential training with a frozen planner keeps the planning–execution interface in natural language (App. A), so every step is interpretable, failures can be traced to planning or execution, and plans can be evaluated independently (Table 4 oracle, Table 5). We adopt the memory scheme and state this clearly (Sec. 3.4).
The new experiments support the empirical claim. In the same closed loop, Executor training lifts Task Success from 9.90% to 33.33% (with memory) on WorldGuide Bench (Rebuttal Table 1). The planner recovers from injected failures using visual evidence alone, completing 46/50 (WorldGuide Bench) and 47/50 (Video-CraftBench) tasks (Reviewer jYyZ, Q3). It stops by itself in 94% of rollouts, with only 6% hitting the step cap. On Video-CraftBench, CollabVR, a frontier VLM loop over an untrained Executor, reaches 16.33% versus our 47.69% (Rebuttal Table 2).
Q2:
(i) Judge circularity. We re-evaluated the strongest models with a judge from a different model family, GLM-5.3 Flash, using the same prompt and protocol (Rebuttal Table 3). The ranking is unchanged, and WorldGuide remains best on both benchmarks. Its margin over MiniMax-H3 actually grows under the non-Gemini judge, from 3.4 to 6.1 points on WorldGuide Bench and from 15.0 to 16.7 points on Video-CraftBench. So the gain is not an artifact of Gemini judging Gemini-style outputs.
On Video-CraftBench, no Gemini-written captions or reference plans are used at all. Models receive only the image and goal, and the judge evaluates the goal directly (App. E.1). On WorldGuide Bench, the baselines are given the Gemini-written reference plan itself as input, so they receive the same caption style. The independent human study also ranks WorldGuide first (App. F).
\begin{array}{l|cccc|cccc} \hline & Gemini 3.5 Flash & & & & GLM-5.3 Flash & & & \\ Model & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline WorldGuide Bench & & & & & & & & \\ MiniMax-H3 & 57.99 & 86.79 & 4.06 & 29.90 & 59.24 & 89.92 & 3.57 & 30.82 \\ TempAct* & 9.11 & 29.68 & 0.50 & 3.98 & 23.61 & 35.28 & 5.71 & 7.96 \\ WorldGuide (ours) & 62.85 & 92.42 & 6.06 & 33.33 & 66.49 & 93.46 & 5.52 & 36.91 \\ \hline Video-CraftBench & & & & & & & & \\ MiniMax-H3 & 28.64 & 88.55 & 2.16 & 32.73 & 47.78 & 90.36 & 5.10 & 36.39 \\ TempAct* & 7.57 & 51.26 & 0.34 & 24.47 & 9.18 & 58.52 & 1.70 & 31.97 \\ WorldGuide (ours) & 57.42 & 94.45 & 17.08 & 47.69 & 60.48 & 94.51 & 14.28 & 53.06 \\ \hline \end{array}
Rebuttal Table 3. Video-judge metrics under Gemini 3.5 Flash and GLM-5.3 Flash, with identical prompts and protocol.
Response to Reviewer BDB2 (3/3)
(ii) Annotation noise. The 17% comes from a strict audit. A step counts as wrong if any one of its action, objects, or result is wrong, and the three annotators agree within 1.3 points on every criterion (App. B.6, Table 9). It's cover all 245 tasks, and each task is trained on many demonstrations (about 235 per task on average, App. B.4), so no single incorrect annotation dominates what the models learn for a task.
For evaluation, the reference plan only guides the judge. The judge accepts equivalent actions, different step sizes, and valid reorderings (App. E.2), and Task Success only checks whether the final video shows the task completed (Eq. 13). Our stopping statistics in (iv) confirm this: rollouts shorter or longer than the reference still succeed 21.8% and 41.6% of the time. All models are scored against the same reference plans, so any noise affects them equally. Video-CraftBench also uses none of our annotations, and WorldGuide’s lead there is even larger, at 47.69% versus 32.73% for MiniMax-H3 (Table 3).
(iii) Repeat/Skip. The "less redundant" claim in Sec. 4.4 compares WorldGuide with its own w/o-visual variant (Table 5, 20.19 → 15.73), not with the baselines, and we will make this clear in the manuscript. Repeat/Skip only counts repeated actions that add no progress, while skipped steps are penalized in Plan Accuracy instead (Eq. 12, App. E.2). A model that does little therefore scores low almost automatically. The open-loop variant has a Repeat/Skip of 0.49 with a Plan Accuracy of only 22.87 (Table 4), and HunyuanVideo-1.5 has a Repeat/Skip of 3.00 with a Plan Accuracy of 43.53 (Table 2), while WorldGuide reaches a Plan Accuracy of 62.85. Part of WorldGuide's Repeat/Skip also comes from recovery, where the planner repeats a step whose generated clip shows it was not completed (Reviewer jYyZ, Q3).
(iv) Rollout length. The unstarred baselines on WorldGuide Bench always run the reference length because they are given the reference plan, while WorldGuide decides its own length. The table compares WorldGuide’s step count with the reference for all 978 judged rollouts on WorldGuide Bench (Video-CraftBench has no reference step counts).
\begin{array}{lcccc} Planner stop & Share & Videos & Successes & Success rate \\ \hline Correct stop (= reference) & 10.0\% & 98 & 30 & 30.6\% \\ Early stop (< reference) & 28.6\% & 280 & 61 & 21.8\% \\ Late stop (> reference) & 55.3\% & 541 & 225 & 41.6\% \\ Hit 80-step cap & 6.0\% & 59 & 10 & 16.9\% \\ \hline \textbf{Total} & 100\% & 978 & 326 & 33.33\% \\ \end{array}
Rebuttal Table 4. Stopping behavior and Task Success on WorldGuide Bench.
Longer rollouts cannot inflate Task Success, because a rollout passes only if its final video shows the task completed (Eq. 13). The longest rollouts, those that hit the 80-step cap, succeed only 16.9% of the time. The other 919 rollouts stop by predicting DONE, and 316 of them (34.4%) end with the task visibly complete. Longer rollouts succeed more often because the planner redoes steps that visibly failed, but each success still needs a completed final state. Step count alone also does not define a stopping error, since a task can be completed in more or fewer atomic steps than the reference and the judge accepts different granularities and valid alternative procedures (App. E.2). This is why shorter rollouts still succeed 21.8% of the time.
Unseen tasks. Our test split holds out unseen demonstrations of the 245 known tasks (App. B.4). Video-CraftBench is an external, fully egocentric benchmark with its own initial images, objects, and scenes, and none of its data is used in training. All models receive only the image and goal there, and WorldGuide leads with 47.69% Task Success, compared with 32.73% for MiniMax-H3 and 16.33% for CollabVR.
We hope these new results resolve your concerns, and we are happy to provide any further details you may need.
Response to Reviewer BDB2 (1/3)
We thank the reviewer for the thorough review and for recognizing our closed-loop formulation and WorldGuide Bench. We address each point below with new experiments.
Q1:
(i) In-distribution fine-tuning. Neither WorldGuide nor any baseline is trained on Video-CraftBench. There, all models receive only the initial image and goal, and the judge evaluates sub-goal completion without any reference plan (App. E.1), so no Gemini-written captions are involved at input or evaluation. WorldGuide still achieves the highest Task Success, 47.69% vs. 32.73% for the 4× larger MiniMax-H3 (Table 3). On WorldGuide Bench, the zero-shot baselines receive the ground-truth action plan, while WorldGuide receives only the goal and must plan for itself (Table 2). The strong closed-loop baselines (PhysAgent, TempAct, Bernini), which also plan from the goal, remain far below on both benchmarks. Closing the loop is therefore not enough. It must be coupled with an Executor trained to realize the planned actions.
(ii) Oracle vs. WorldGuide. The oracle is given the ground-truth actions and their length, so it never plans or decides when to stop. We include it to measure how much error comes from planning, so it is an upper bound rather than a deployable alternative (Sec. 4.5). WorldGuide does both from the goal alone and still reaches 33.33%, compared with 38.82% for the oracle. In Limitation Sec. 5 discusses how a stronger planner and tighter coupling can close this gap. With the same planner and Executor, closed-loop WorldGuide reaches 33.33% versus 11.71% open-loop (Table 4). The loop helps even the zero-shot Executor, raising goal-only HunyuanVideo-1.5 from 3.16% to 9.90% on WorldGuide Bench and from 24.15% to 28.43% on Video-CraftBench. We will revise the Conclusion to state this clearly.
Gain from Executor training. The same-backbone gain from 12.00% to 38.82% is the contribution we claim, not a confound. Coupling planning with an Executor trained to realize the planned actions is the central idea of the paper (Sec. 1, Fig. 1). Our new control confirms this inside the closed loop. With the same ContextPlanner and rollout, the zero-shot Executor reaches 9.90% while our trained Executor reaches 33.33% on WorldGuide Bench, and 28.43% versus 47.69% on Video-CraftBench (Rebuttal Table 1).
\begin{array}{lccccc|ccccc} Variant & Aesth. & Imag. & Dynamic & Smooth. & Consist. & Align. & Plan Acc. & Order & Repeat/Skip\downarrow & Task Succ. \\ \hline WorldGuide Bench \\ HunyuanVideo-1.5(Goal Only) & 43.64 & 66.28 & 90.82 & 96.52 & 19.26 & 56.97 & 18.68 & 34.85 & 1.03 & 3.16 \\ HunyuanVideo-1.5(Oracle Plan) & 42.43 & 65.57 & 88.84 & 96.00 & 18.91 & 62.28 & 43.53 & 80.00 & 3.00 & 12.00 \\ HunyuanVideo-1.5 + ContextPlanner & 38.24 & 64.46 & 85.82 & 95.42 & 15.34 & 60.56 & 40.22 & 78.58 & 18.26 & 9.90 \\ WorldGuide & 49.36 & 64.77 & 98.06 & 98.96 & 31.13 & 73.85 & 62.85 & 92.42 & 6.06 & 33.33 \\ \hline Video-CraftBench \\ HunyuanVideo-1.5(Goal Only) & 45.30 & 67.62 & 99.66 & 98.91 & 21.91 & 63.94 & 14.66 & 64.97 & 3.06 & 24.15 \\ HunyuanVideo-1.5 + ContextPlanner & 48.54 & 69.88 & 99.26 & 98.82 & 20.38 & 69.62 & 26.28 & 82.74 & 6.34 & 28.43 \\ WorldGuide & 40.19 & 64.63 & 99.65 & 98.86 & 22.09 & 71.80 & 57.42 & 94.45 & 17.08 & 47.69 \\ \hline \end{array}
Rebuttal Table 1. Zero-shot vs. trained Executor on the same backbone. Rows with “+ ContextPlanner” and WorldGuide share the same planner and closed loop.
(iii) Open-loop vs. closed-loop. This is addressed in (ii). The oracle plan and WorldGuide use the same Executor and memory, so the oracle is better only because it is given the ground-truth plan, which are never available real time setting. Without them, open-loop execution drops to 11.71% while closing the loop reaches 33.33%, and the loop raises even the zero-shot Executor from 3.16% to 9.90% (WorldGuide Bench) and from 24.15% to 28.43% (Video-CraftBench). Both parts of our claim therefore hold. Executor training makes a good plan executable, and closing the loop lets that plan be predicted from progress when it is not given.