acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Canvas: Spatial-Temporal Visual Reasoning for Embodied Video Generation

Abstract

Video generation models are increasingly used as visual world models for embodied AI, but they are not explicitly trained for spatial-temporal reasoning required by multi-object, multi-step manipulation tasks. Such models may generate visually plausible videos while manipulating the wrong object, misordering subtasks, or placing objects at incorrect targets. To this end, we propose Reasoning Canvas, a dual-stream video diffusion framework that introduces an explicit auxiliary Thought stream for spatial-temporal structure. The Thought stream renders object-following bounding boxes and step-wise text prompts on a black canvas, providing a lightweight representation of what should happen before high-fidelity pixels are fully synthesized. A Timestep-Bias Reasoning Routing (TBRR) schedule resolves this simpler Thought stream earlier than the RGB stream, enabling subsequent RGB denoising to condition on a more stable plan. We also construct HierCanvas Dataset, a RoboCasa-derived hierarchical manipulation dataset with task→subtask→step programs, bounding boxes, and rendered step prompts. Experiments show that Reasoning Canvas yields consistent gains in visual quality, instruction alignment, and tracking, with stronger robustness to ambiguous instructions and OOD task compositions, validating ahead-of-time visual reasoning as an effective interface for video generation based world models.

Primary area:
generative models
Keywords:
Video Generation, Video Reasoning, Embodied World Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mBTAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Reasoning Canvas augments a video diffusion model with a generated Thought stream containing object/gripper bounding boxes and rendered step captions. A timestep-biased schedule resolves this auxiliary stream before RGB denoising is complete, and two-stage training, Thought guidance, and Thought Forcing are used to make RGB generation follow it. The paper constructs the hierarchically annotated HierCanvas dataset and evaluates simulation and real-robot videos, unseen task compositions, vague instructions, and an external robotics video benchmark. It reports improved visual and trajectory metrics over adapted RGB-only baselines and analyzes spatial/textual Thought variants, guidance strength, and ground-truth Thought as an upper bound.

Critical strengths
  1. Co-generating an interpretable spatial-temporal condition is a useful design for manipulation video generation. The main comparisons use model-generated Thought rather than future ground truth, and the spatial/textual ablations, guidance sweeps, and separately identified GT-Thought upper bound provide evidence that this channel meaningfully influences RGB generation (Tables 6, 10, 12, and 15).

  2. The empirical study extends beyond one simulated setting: it includes real-robot validation, unseen compositions, and an external robotics benchmark. The real-world comparison uses identical training and validation episode IDs across methods, and the paper explicitly acknowledges the absence of downstream control validation and the gap between generated and ground-truth Thought (Appendix A.3 and Section 5).

Critical weaknesses
  1. Capacity and training-budget differences confound attribution. Section 3.2 describes a shared DiT with stream-specific embeddings/modulation, whereas Appendix E, Table 17 states that every parameterized DiT module is replicated into two independent copies. Reasoning Canvas also receives 24 epochs of Thought pretraining, and Thought Forcing adds another training budget to an RC-ST checkpoint. Capacity/compute-matched RGB baselines and a continued-training control without Forcing are needed to isolate the proposed components.

  2. The evaluation split descriptions are inconsistent. Appendix A.2.2/Table 8 defines 274 training/ID task classes and 37 held-out OOD classes, yet describes the main 311-example evaluation as an ID subset containing one episode from each of all 311 classes. Its actual membership and relationship to the 185-example OOD evaluation remain unclear. This is a consequential protocol ambiguity, rather than evidence of leakage, and prevents a confident interpretation of the generalization results.

  3. Thought Forcing is specified differently across the paper. Section 3.5 starts from randomly noised ground-truth Thought, whereas Appendix E.2 specifies pure noise and noisy-RGB cross-stream context; the latter also appears inconsistent with Section 3.4's Thought-only attention description. Initialization and information flow define what this intervention learns, so these differences must be reconciled for the ablation to be interpretable and reproducible.

  4. Evidence for the specific ahead-of-time reasoning and planning claims is weaker than evidence for useful auxiliary conditioning. The default tau=900 is worse than synchronized tau=0 on mATE (10.52 vs. 9.43), while the best tau=600 improves it to 9.06. Table 11 provides useful semantic evaluation, but its average changes only from 2.657 to 2.714 and instruction grounding is unchanged. Direct evidence of correct object selection and ordered subtask completion would strengthen the planning interpretation. These attribution and protocol issues make me lean toward weak rejection.

Questions for authors
  1. Which modules actually share weights, what are the parameter counts and training/inference costs, and how does performance compare with a capacity/compute-matched RGB baseline and an RC-ST model continued for the same extra training budget without Thought Forcing?

  2. Can the authors provide the task/episode manifests for the main 311-example and 185-example OOD evaluations, identify the checkpoint/configuration used in each table, and reconcile the 274-versus-311 ID task counts? Please also clarify the actual Thought Forcing initialization and attention mask.

  3. Can the authors quantify correct-object selection and ordered subtask completion on a representative sample, with human or rule-based validation and uncertainty? Table 11 provides useful VLM evidence, but its small changes and unchanged instruction-grounding score do not yet strongly support the broader planning claim; trajectory similarity alone does not resolve it.

Reviewer_2iZzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing methods for embodied video generation often produce incorrect objects, misorder subtasks, or place objects at incorrect targets. To address these limitations, this paper proposes Reasoning Canvas, a dual-stream framework that introduces an auxiliary thought stream for explicit spatial-temporal planning. Unlike text-only or RGB frame-based chain-of-thought, the thought stream uses color-coded bounding boxes on black canvases to represent the gripper and manipulated entity, together with step-wise text for temporal planning. The two streams share a single diffusion backbone, while a Timestep-Bias Reasoning Routing schedule is also introduced to allow the thought stream to complete denoising earlier. The RGB stream then uses the resolved plan and the initial frame as guidance to generate the video through flow matching. Moreover, this paper constructs a dataset, HierCanvas. Extensive experiments show that Reasoning Canvas consistently improves video generation across visual quality, instruction alignment, and tracking accuracy.

Critical strengths
  1. The proposed thought stream explicitly models both spatial grounding and temporal planning while enabling faster denoising than the RGB stream.

  2. The newly introduced dataset provides a useful resource for future research.

  3. Extensive experiments demonstrate improvements in visual quality, instruction alignment, and tracking accuracy, alongside robustness to ambiguous instructions and generalization to out-of-distribution task compositions.

Critical weaknesses
  1. The caption of Figure 2 is unclear and does not adequately explain how the components interact or how information flows between them.

  2. The design of the Thought generator is insufficiently explained. Could a multimodal large language model be used to directly predict this structured spatial-temporal plan?

  3. The framework appears tailored to gripper pick-and-place tasks. Given its reliance on a trained Thought generator, its ability to generalize to broader embodied scenarios remains unclear.

Questions for authors

See weaknesses.

Reviewer_ydEvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reasoning Canvas, a method that jointly generates an RGB video and a Thought video containing bounding boxes and step-wise rendered text to guide RGB frame generation. This is motivated by the observation that video models struggle to generate accurate manipulation videos. The Thought video uses a black canvas with bounding boxes and text to represent the gripper, manipulated object, and current subtask. The Thought stream finishes denoising earlier, allowing RGB generation to benefit from the clearer Thought stream. Experiments are conducted to validate these design choices.

Critical strengths

I feel the main idea is quite interesting. First generating a symbolic visual representation and then use it to guide complex and detailed RGB frames sounds reasonable to me. The paper is generally well-written and easy to follow, and the reported improvements are encouraging.

Critical weaknesses

Despite the interesting idea, I feel some specific design choices are not sufficiently justified:

  1. Based on my understanding, the video model is used to reason (split a task into subtasks) and generate the subtask descriptions as image pixels. In my view, a more natural alternative is to use an LLM/VLM to plan the task first, then use the resulting descriptions to condition the generation of Thought frames. I feel this could improve the model’s performance on complex tasks. It would be helpful to compare an LLM-based planner with the current purely generation-based method.

  2. I am a bit concerned about the generated subtask descriptions in the Thought frames, as latent video diffusion models like Wan are known to struggle with generating text. I wonder how readable the captions are and how important their readability is for RGB generation. The paper discusses imperfect Thought through Thought Forcing and the ground-truth comparisons, but it remains unclear how text-rendering errors specifically affect RGB generation. It would be helpful to conduct an experiment to evaluate this. I am interested in seeing these results.

Questions for authors

See the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.