acceptodds
Under review as a conference paper at ICLR 2027

PhysPlan: Grounded Physical State Reasoning and Graph-Guided Optimization for Physically Plausible Video Generation

Abstract

Video diffusion models (VDMs) synthesize photorealistic content, yet they often fail to follow the course that a physical phenomenon should take within a given scene. Recent training-free methods let a vision-language model (VLM) plan the phenomenon and guide a frozen VDM toward the plan; however, such plans are derived from the prompt and consumed as whole keyframes or trajectories, which leaves unspecified where the consequences land in the observed scene and turns incidental visual details into optimization targets. We observe that a phenomenon specified in words unfolds as sparse, local changes to the physical state of the observed scene. Building on this observation, we present PhysPlan, a training-free image-to-video framework that represents a phenomenon as a grounded state graph and uses this graph to decide what, where, and when the guidance constrains. Grounded Physical State Reasoning decomposes the phenomenon into physical deltas, each stating the new state of every changing object together with the physical rule that produces it, and translates each delta into graph edits, verified by deterministic checks, that leave all other objects unchanged. Graph-Guided Test-Time Optimization renders a keyframe for each state, measures the denoised estimates only along the properties selected by the edits, and concentrates the update on the edited objects. Experiments on PhyGenBench and Physics-IQ show that PhysPlan improves physical plausibility over foundational and training-free guided baselines while preserving visual quality.

Primary area:
generative models
Keywords:
Video Diffusion Models, Physical Reasoning, State Graphs, Test-Time Optimization, Physics-Aware Video Generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_1ZymOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PhysPlan is a training-free framework for physically plausible image-to-video generation. A VLM parses the input scene into a graph and plans qualitative object-state changes, which are implemented as typed graph edits and rendered into target keyframes. During video sampling, the edits select manually designed appearance, area, position, depth, and instance-count losses, whose gradients update the video latent while all model weights remain frozen.

Experiments on PhyGenBench and Physics-IQ show improvements over the base model and Frame Guidance. Ablations and a human study provide additional evidence of practical effectiveness.

Critical strengths
  • The matched-keyframe comparison supports the usefulness of selective guidance. PhysPlan and Frame Guidance use the same backbone, keyframes, and anchors, yet PhysPlan improves Physics-IQ from 30.3 to 38.2. This indicates that its benefits extend beyond obtaining better reference images from an external image editor.

  • The system is evaluated from several complementary perspectives. Benchmark scores, component ablations, human preferences, failure attribution, and runtime measurements provide a reasonably comprehensive account of its practical behavior.

Critical weaknesses
  • The physical representation and guidance are heavily hand-designed, limiting their generality. The framework expresses phenomena through discrete objects, a fixed attribute/relation vocabulary, and predefined mappings from edits to visual losses. These constraints naturally capture changes in appearance, size, location, and presence, but provide limited control over continuous dynamics, quantitative physical relationships, or phenomena better represented as fields or distributed materials. The acknowledged weakness in fluid dynamics illustrates this limitation. Moreover, the deterministic checks establish graph consistency rather than physical validity. The evidence therefore supports a useful mechanism for enforcing selected qualitative outcomes more strongly than a general advance in physical reasoning.

  • The methodological advance is limited relative to the complexity of the inference pipeline. Physical planning and target appearance are supplied by pretrained VLM and image-editing models, while the video generator is steered through established latent-optimization machinery and manually constructed visual objectives. The main contribution is consequently the organization and selection of these components, with limited evidence of a broadly applicable new reasoning or optimization principle. This concern is reinforced by evaluation on only one video backbone and substantial inference overhead: approximately 366 seconds and 64 GB per video, versus 100 seconds and 26 GB for the base model. Training-free operation is useful, but does not by itself establish sufficient methodological significance.

Questions for authors

My main concern is an inherent limitation of the formulation: physical guidance is expressed through a hand-designed graph-edit vocabulary and corresponding visual proxy losses, while continuous dynamics between anchors remain unconstrained by explicit physical relationships. What classes of physical phenomena can this representation meaningfully capture, and which fall outside its scope? Please clarify how these representational limitations affect the claimed generality of the method.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1Zym (Part 1/2)

Thank you for the careful assessment of the scope of PhysPlan, and for noting that the matched comparison with Frame Guidance shows a benefit beyond better reference images from the image editor. Revisions are highlighted in blue in the updated PDF. Your points concern which phenomena the representation can capture (W1, Q1) and the methodological contribution and cost (W2).

W1 & Q1: Clarifying the Scope and Generality of the Representation

Below, we state the scope of PhysPlan precisely, show where it matters, and explain how the representation can be extended.

What PhysPlan targets. PhysPlan enforces the qualitative outcome of a phenomenon in the observed scene: which objects change, into what state, where, and in what order. This is the design stated in the submission, in which the VLM judges only which objects change, how, and in what order, without estimating quantities (Sections 2 and 3.1). PhysPlan therefore does not model continuous dynamics, quantitative relations, or field quantities. In the revised manuscript, the abstract, introduction, and conclusion state this scope, and the contribution is described as enforcing grounded qualitative outcomes in a frozen VDM.

Why this target matters. Many failures that PhyGenBench and Physics-IQ measure are wrong qualitative outcomes, such as an object that vanishes without a product, an object that duplicates, or a flame that keeps burning under water (Figure 4), and these are the outcomes that the graph controls.

Scope (Q1). Table R2 (response to Reviewer xrzv) reports the gain of PhysPlan on each physical law defined by PhyGenBench (Meng et al., 2025). The gains are largest for laws whose outcome is a discrete change of object state, such as a change of color, a phase change, or a collapse (e.g., Atmospheric Pressure +0.49 and Color +0.48), and smallest for laws whose outcome is a light pattern or a medium without a clear object boundary (e.g., Dispersion +0.02 and Tyndall Effect +0.04) or continuous motion (Gravity +0.12). Fields, waves, and quantitative laws are outside the representation.

Continuous phenomena. Part of this limitation lies in how coarsely continuous phenomena are planned rather than in the representation itself. In Table R3 (response to Reviewer xrzv), one added instruction in the stage-1 prompt splits a continuous change into successive intermediate states: the VLM then plans 6.3 instead of 2.6 events per video, and Fluid Dynamics on Physics-IQ improves from 31.8 to 33.9, close to VLIPP (34.1), at 24% more runtime.

Why the vocabulary is fixed. The attribute keys and relation types are fixed, but their values are open-vocabulary (Table 5), so a key such as surface can take values like wet, frosted, or charred. Fixing the keys is what makes an edit checkable and measurable: a key either exists or not, and each key is assigned a loss term. The mapping from edit types to loss terms follows from what each edit changes: appearance, area, location, depth order, or presence. Table 4 of the submission supports this mapping, since each term matters most in the categories it is designed for. Extending the vocabulary, for example with particle sets or fields for liquids, requires a new key and a new measurement, without changing the rest of the pipeline.

Physical validity. As in our response to Reviewer zzZk (W1, P1), the deterministic checks are designed to verify the structure of the reasoning chain rather than physical laws; these structural checks encode basic physical invariants, such as that objects cannot appear or vanish without a source, while the physical content of each change comes from the qualitative rule that the VLM gives. The revised wording makes this distinction explicit.

In summary, PhysPlan targets grounded qualitative outcomes rather than continuous or quantitative dynamics. This scope is now stated explicitly, and it covers the qualitative failures that current VDMs most visibly make.

AuthorsOpenReview ↗

Response to Reviewer 1Zym (Part 2/2)

W2: Clarifying Methodological Contribution and Cost

Our contribution. PhysPlan is a training-free interface between physical reasoning and a frozen VDM. Its plan is a state graph grounded in the observed frame and changed only by verified typed edits, and this graph decides what each guidance term measures, where the update acts, and when each state should hold. The tools that execute the plan, a VLM, an open-vocabulary segmenter, a depth estimator, a feature encoder, and the latent optimization of Frame Guidance, are standard and widely used across vision; our contribution is the rule that couples them. The other reviewers identify this coupling as the strength of the design: one state graph determines the state changes, loss types, regions, and temporal anchors (Reviewer zzZk), which is a sensible improvement over treating every detail of a generated keyframe as a target (Reviewer xrzv). Prior planners instead derive the plan from the prompt and pass it to the VDM as whole keyframes or trajectories (VLIPP, Chain-of-Events, CausalMotion), and prior guidance matches whole frames (Frame Guidance). Because PhysPlan requires no training data or fine-tuning, it also applies to a new backbone without the per-backbone training that methods such as PhysHPO and PhyGDPO need.

Same tools, smaller gains without the graph. The same tools appear in prior work, but without a grounded plan that decides what each one measures, they yield smaller gains (Table 2). VLIPP uses a VLM planner and Grounded-SAM-2 with a CogVideoX-based generator and reaches 0.60 on PhyGenBench and 34.6 on Physics-IQ. Frame Guidance uses our keyframes and the same latent optimization on the same base, yet stays at 0.51 and 30.3, i.e., it does not improve its base on PhyGenBench (0.52); as the reviewer notes, this matched comparison shows that the benefit of PhysPlan extends beyond better reference images from the image editor. PhysPlan reaches 0.77 and 38.2. Training-based methods improve their T2V base models on PhyGenBench by 0.16 (PhysHPO) and 0.15 (PhyGDPO), whereas PhysPlan improves its I2V base by 0.25 without any training, although T2V and I2V results are not directly comparable.

The components couple rather than stack. The one-factor ablations in Table R6 (response to Reviewer zzZk) show that the gain comes from how the graph connects the components, not from adding them. With identical components, building G_0 from the prompt instead of the observed frame costs 7.1 points, and regenerating the graph instead of editing it costs 7.3 (rows i-a and ii). With the same keyframes and region weighting, replacing the graph-selected terms by whole-frame L2 costs 3.7 points (row iii-a). Removing any single factor costs 1.3 to 7.3 points, and these drops sum to 21.6, about twice the total gain over the base model (11.1), so the factors depend on each other rather than adding independently.

Beyond our original setting. Under the same VLM and generator weights, PhysPlan outperforms VLIPP, and it remains ahead with a different VLM (Table R1, response to Reviewer xrzv). On two more backbones, PhysPlan improves Physics-IQ by a similar margin (Table R4).

Cost. The comparison that isolates our cost is Frame Guidance, which uses the same optimization: PhysPlan requires 18% more runtime than Frame Guidance (Table 14 of the submission), and 9% and 11% more on LTX-Video-I2V and Wan2.2-TI2V-5B (Table R4), mainly for the feature encoder, instance matching, and the additional terms. Peak memory is likewise close to that of Frame Guidance (64 vs. 58 GB on CogVideoX-I2V-5B). Most of the runtime is backpropagation through the denoiser, which every loss-guided method shares.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer 1Zym,

As the discussion period closes on November 18 (AoE), we would like to kindly check whether our response has addressed your concerns: the scope of the representation, now stated explicitly and supported by the per-law analysis (Tables R2–R3), the two additional backbones (Table R4), and the cost relative to Frame Guidance. If so, we would be grateful if you would consider updating your score. We are happy to clarify anything further.

Reviewer_puYoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

PhysPlan is a training-free image-to-video method. A VLM turns the input frame into a scene graph and breaks the prompted phenomenon into a few events, each written as checked graph edits so that unedited objects stay unchanged. An image editor renders a keyframe for each state, and a frozen VDM is guided toward these keyframes at keyframes, only on the edited objects and properties.

Critical strengths
  1. The problem is reasonable and meaningful.
  2. The method is simple and intuitive, with some interesting design details, such as checked graph edits and losses chosen by edit type.
  3. The visual results are good on several cases and better than the compared methods.
Critical weaknesses
  1. Related work and baselines are insufficient. Recent physics-based methods such as 3DPhysVideo, PhysAgent and NEWTON are not cited, and their visual results look better than the examples on the project page. Besides the base I2V models, the paper only runs Frame Guidance and takes VLIPP’s numbers from its paper. No simulation-guided method is compared.
  2. The method relies on the image editor. Appendix C attributes 25% of failures to keyframes, but rendered keyframes are never checked, and no other editor is tested.
Questions for authors
  1. Could the authors compare with recent physics-based methods, especially simulation-guided ones such as 3DPhysVideo and PhysAgent? Comparing mainly with general-purpose methods is not enough to show effectiveness.
  2. How robust is the method when the image editor hallucinates?
6 comments
AuthorsOpenReview ↗

Request to correct review comments for PhysPlan

Dear Reviewer puYo,

Thank you for taking the time to review our submission. We would like to ask for your help in clarifying the written feedback so that we can respond appropriately.

The current review discusses 3D smoke reconstruction, ScalarFlow/Plume, and a vorticity loss. However, our paper, PhysPlan, focuses on training-free image-to-video generation using grounded state graphs. These topics seem unrelated to our work, so we wonder whether some comments from another submission may have been inadvertently included in the review.

We also came across comments at https://openreview.net/forum?id=6S505RgVo2 that appear to discuss PhysPlan's methodology and experiments. We thought this might be helpful in identifying the intended feedback.

We would appreciate if Reviewer puYo could update the written comments to ensure they correspond to our paper. This would allow us to better understand and respond to the reviewer's actual concerns during the discussion.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Response to Reviewer puYo (Part 1/2)

Thank you for your updated review and for pointing us to recent physics-based methods. Revisions are highlighted in blue in the updated PDF. Your points concern the related work and baselines (W1, Q1) and the dependence on the image editor (W2, Q2), for which we add a matched VLIPP re-run, two more backbones, two open-weight editors, controlled keyframe corruptions, and a keyframe verification step.

W1 & Q1: Comparing with Recent Physics-Based Methods

We now discuss 3DPhysVideo, PhysAgent, and NEWTON in Section 2 of the revised manuscript, and we also strengthen the baselines that the reviewer found insufficient.

Stronger baselines. Instead of quoting VLIPP from its paper, we now re-run it under matched settings (Table R1, response to Reviewer xrzv): with the same VLM and generator weights, PhysPlan outperforms VLIPP by 0.14 on PhyGenBench and 3.2 on Physics-IQ. We also apply PhysPlan to LTX-Video-I2V and Wan2.2-TI2V-5B, where it improves Physics-IQ by 9.9 and 9.8 points (Table R4).

Related methods. 3DPhysVideo reconstructs the 3D scene from a single image with an I2V model, runs a physics solver on the reconstruction, and guides the same I2V model with the simulated point cloud. PhysAgent writes a physics program for a simulator and repairs it in a loop of simulation, verification, and targeted correction. NEWTON trains a planner with Flow-GRPO that calls keyframe generation, scientific computation, and prompt refinement around a frozen generator. Simulation-guided methods produce accurate motion for the materials and interactions that their solvers support.

Difference from PhysPlan. PhysPlan makes a different trade-off: it needs no 3D reconstruction, simulator, or training, and it covers changes of object state that solvers do not model, such as burning, melting into a puddle, or a flame quenched in water (Figure 4), while being weaker on continuous dynamics (response to Reviewer xrzv, W2).

Quantitative comparison. NEWTON reports results on PhyGenBench, which allows an indirect comparison: its trained planner raises LTX-Video-2B from 0.51 to 0.56 as a text-to-video method, whereas the training-free PhysPlan raises CogVideoX-I2V-5B from 0.52 to 0.77. The generators, input modality, and judging setup differ, so the absolute numbers are not directly comparable, but the gain over each base model is substantially larger for PhysPlan. 3DPhysVideo and PhysAgent are evaluated on their own scenes and benchmarks (GPT-based scores and VideoPhy, and a benchmark of events with specified forces and timing), which test precisely specified, simulated dynamics that PhysPlan deliberately does not model.

Visual quality. The examples on our project page are 720×480 outputs of CogVideoX-I2V-5B, so part of the difference in visual quality comes from the generator; NEWTON, for example, also reports results with Veo-3.1. Since PhysPlan transfers across backbones (Table R4), it directly benefits from stronger generators such as Wan2.2-TI2V-5B.

AuthorsOpenReview ↗

Response to Reviewer puYo (Part 2/2)

W2 & Q2: Assessing the Dependence on the Image Editor

We address the dependence on the image editor in three ways: other editors, controlled corruptions of the keyframes, and a verification of every rendered keyframe.

Other editors. We replace Gemini 3 Pro Image with two open-weight editors, Qwen-Image-Edit and FLUX.1 Kontext [dev], keeping the event chains, instructions, and guidance fixed (Table R9). With either editor, PhysPlan stays well above its base model (0.52 and 27.1) and above VLIPP (0.60 and 34.6) on both benchmarks, and the score decreases only moderately compared with Gemini 3 Pro Image.

Table R9: PhysPlan with different image editors. Event chains, instructions, and guidance are fixed.

Image editorPhyGenBench ↑Physics-IQ ↑
Gemini 3 Pro Image (submission)0.7738.2
Qwen-Image-Edit0.7336.8
FLUX.1 Kontext [dev]0.6935.9

Robustness to hallucinated keyframes (Q2). Since PhysPlan measures only the properties selected by the edits, errors on objects that no edit names do not enter its objective, whereas the whole-frame L2 loss of Frame Guidance matches them. To test this, we corrupt the keyframes in three controlled ways and run both methods with the same corrupted keyframes (Table R10). Errors on objects that no edit names lower PhysPlan by 1.1–1.3 points, against 2.3–2.8 for Frame Guidance. A shifted edited object affects both methods, PhysPlan slightly more (1.8 vs. 1.4), since its location term measures exactly this property.

Table R10: Physics-IQ with corrupted keyframes. Changes are relative to uncorrupted keyframes.

CorruptionFrame GuidanceChangePhysPlanChange
None30.3–38.2–
Unrelated object inserted27.5−2.837.1−1.1
Attribute of an unedited object changed28.0−2.336.9−1.3
Edited object shifted28.9−1.436.4−1.8

Keyframe verification. As the reviewer notes, Appendix C attributes 25% of the failures to keyframes. We therefore add a check that applies the GPSR checks to each rendered keyframe and re-renders a keyframe that fails it, with at most three attempts (Table R8; details in our response to Reviewer zzZk, Q3). The check rejects 23% of the keyframes at the first attempt and raises PhysPlan to 0.78 on PhyGenBench and 39.0 on Physics-IQ, at about 9% additional time per video.

Tables R9 and R10 and the keyframe verification are included in the revised manuscript (highlighted in blue).

Reviewer_zzZkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PhysPlan, a framework for physically plausible video generation that requires no additional training. The method first constructs an object state graph from the initial image and decomposes the physical process into incremental edits that undergo structural checks. It then generates corresponding keyframes and selects object-level losses, optimization regions, and temporal anchors based on the edits to guide a frozen video diffusion model. The paper reports improvements in physical plausibility on PhyGenBench and Physics-IQ and validates the method through visual quality evaluations, ablation studies, and a human preference study.

Critical strengths
  1. The interface between planning and generation constraints is well designed. The same state graph determines object state changes, loss types, regions of application, and temporal anchors, allowing the planning results to concretely influence the generation process.
  2. The experiments cover a broad range of evaluations and include valuable controlled comparisons. The paper covers two physics benchmarks, visual quality, human preferences, failure analysis, and cost evaluation. In particular, the comparison with Frame Guidance using the same keyframes and anchors helps assess the role of object-level guidance.
Critical weaknesses
  1. The claims regarding novelty and the guarantees provided by the mechanisms are overstated, and the existing ablations are insufficient to support the core explanations. The proposed design can only validate the structure of graph edits, not physical laws. The count loss also cannot constrain the position or appearance of unedited objects when all instances are matched. Key ablations simultaneously change the graph representation, grounding, and loss design, and therefore have not independently established the contribution of each core design choice.
  2. It is unclear how object identities are transferred from the graph to visual instances. The graph uses persistent object IDs, but segmentation relies primarily on categories, while matching relies primarily on positions. Multiple objects of the same category, occlusion, and crossing trajectories may lead to identity confusion, causing edits to affect the wrong object.
Questions for authors
  1. When all instances are matched and the count loss is zero, what mechanism prevents changes to the position or appearance of unedited objects? Is there a direct evaluation of object preservation?
  2. How are multiple objects of the same category mapped to their respective persistent IDs in the graph? How are identity and lineage consistency maintained during occlusion, crossing trajectories, and object splitting?
  3. Are the generated keyframes checked for consistency with the current scene? In the demo of dry leaves burning on the project page, the generated video contains grass that is absent from the initial image. I suspect this may have been introduced by the keyframes.
6 comments
AuthorsOpenReview ↗

Response to Reviewer zzZk (Part 1/4)

Thank you for your detailed and careful review, down to checking the examples on our project page. Revisions are highlighted in blue in the updated PDF.

W1 & Q1: Restating the Novelty, Clarifying the Guarantees of the Checks, Preserving Unedited Objects, and Disentangling the Ablations

(P1) Novelty. Individual components of PhysPlan build on existing work: event chains over scene graphs (Wang et al., 2026), VLM planning (Yang et al., 2025a), and loss-based guidance (Jang et al., 2026). Our contribution is how they are connected, which the reviewer also identifies as the main strength of the paper: one state graph, grounded in the observed frame and changed only through verified typed edits, determines the state changes, loss types, regions, and temporal anchors. In contrast, Chain-of-Events builds its graphs from the prompt and conditions the VDM on whole keyframes, VLIPP passes trajectories, and Frame Guidance matches whole frames. Two results show that this connection, rather than any single component, produces the gain. First, the same tools yield smaller gains without it: Frame Guidance uses our keyframes and the same latent optimization but reaches 30.3 on Physics-IQ, and VLIPP, which also uses a VLM planner and Grounded-SAM-2, reaches 34.6, against 38.2 for PhysPlan. Second, the one-factor ablations in (P3) show that the components depend on each other rather than adding independently. In the revised manuscript, Section 1 states the contribution in these terms instead of claiming novelty for the individual components.

What the checks guarantee. The deterministic checks are designed to verify the structure of the reasoning chain rather than physical laws, and we make this distinction explicit in the revision (Abstract, Section 1, Section 3.1). These structural checks encode basic physical invariants: objects cannot appear or vanish without a source (lineage), no object changes without a cause stated in the delta (coverage), and support and containment remain physically realizable (consistency). Whether a stated cause is correct, such as ice above its melting point turning into water, is judged by the VLM through the qualitative rule that each delta must give. Appendix C of the submission shows that both parts work in practice: 13% of the edit sets violate a check at the first attempt, most often coverage, and 97% pass after retries, while reasoning errors cause only 11% of the failed videos.

(P2) Preservation of unedited objects. The count term is designed to keep the identity of objects rather than their state: it penalizes an object only when an instance is lost or duplicated, and it is zero once all instances are matched. Preserving the position and appearance of unedited objects is therefore not the role of this term but of two other mechanisms. First, the VDM is conditioned on I_0, so unedited objects follow the observed frame unless guidance moves them. Second, region weighting reduces the update outside the regions of the edited objects, so the guidance changes unedited objects as little as possible. Region weighting acts in the layout stage, where the positions and shapes of objects are set, while the detail stage uses the unweighted gradient (Algorithm 1). In the revised manuscript, we correct the description of the count term in Appendix A.5 and state the role of each term explicitly.

Direct evaluation (Q1). For every event, we measure how much each object that the event does not edit changes in the generated video between the previous keyframe position and the current one (from frame fi−1f_{i-1} to frame fif_i, with f0f_0 the first frame). This covers both objects that are never edited and objects that are edited only in other events, and it compares each object with its own state just before the event rather than with I0I_0 (Table R5). For the base model and Frame Guidance, we use the same event chains and keyframe positions, so that all methods are measured on the same objects and intervals. PhysPlan changes unedited objects the least on all three metrics, and removing region weighting increases their change, which confirms that region weighting contributes to their preservation.

Table R5: Preservation of objects that an event does not edit, on Physics-IQ, measured between consecutive keyframe positions of the generated video. Drift is the displacement of the object centroid divided by the image diagonal (×10⁻²); LPIPS is computed inside each object's mask at the earlier keyframe position.

MethodDrift ↓Mask IoU ↑LPIPS ↓Physics-IQ ↑
CogVideoX-I2V-5B1.400.620.8727.1
Frame Guidance0.370.800.2830.3
PhysPlan w/o region weighting (λ = 1)0.320.870.1836.0
PhysPlan (full)0.240.940.1338.2
AuthorsOpenReview ↗

Response to Reviewer zzZk (Part 2/4)

W1 & Q1 (Continued): Restating the Novelty, Clarifying the Guarantees of the Checks, Preserving Unedited Objects, and Disentangling the Ablations

(P3) Disentangled ablations. Ablation (i) changes grounding (whether G_0 is built from the observed frame or from the prompt alone), representation, and loss together, so the 2.2-point estimate for grounding in the submission compared two settings that differ in both grounding and representation. In the revised manuscript, we replace this estimate with ablations that change one factor at a time (Table R6): an ungrounded graph built from the prompt alone, which isolates grounding; whole-frame L2 with region weighting, which separates the loss from the region weighting; and GPSR without the deterministic checks.

Table R6: Ablations on Physics-IQ that change one factor at a time. New rows are marked with *.

SettingGrounded G_0Graph editsLossRegion weightingPhysics-IQ ↑
(i) Prompt-derived planNoNoWhole-frame L2No28.1
(i-a)* Ungrounded graphNoYesGraph-selectedYes31.1
(ii) Full-graph regenerationYesNo (graph diff)Graph-selectedYes30.9
(iii) Whole-frame L2YesYesWhole-frame L2No30.3
(iii-a)* Whole-frame L2 + region weightingYesYesWhole-frame L2Yes34.5
(iv) w/o region weightingYesYesGraph-selectedNo36.0
(ix)* w/o deterministic checksYesYes, uncheckedGraph-selectedYes36.9
PhysPlan (full)YesYesGraph-selectedYes38.2

Results. Each design choice contributes on its own. Building G_0 from the prompt alone costs 7.1 points, comparable to regenerating the full graph instead of applying verified edits (7.3 points, ablation (ii)), followed by the graph-selected loss (3.7, row iii-a), region weighting (2.2, row iv), and the deterministic checks (1.3, row ix). The 2.2-point estimate in the submission therefore understated the contribution of grounding.

Coupling rather than stacking. These results also support the novelty argument in (P1). The drops from removing a single factor sum to 21.6 points, about twice the total gain of PhysPlan over its base model (11.1). The factors therefore reinforce each other rather than adding independently: grounding is effective because the edits keep the graph tied to the observed scene, and the graph-selected loss is effective because the graph decides what it measures and where. This interdependence is what the state graph provides, and it is the contribution that we now state in Section 1.

Tables R5 and R6 and the revised claims are included in the revised manuscript (highlighted in blue).

AuthorsOpenReview ↗

Response to Reviewer zzZk (Part 3/4)

W2 & Q2: Transferring Identities Between the Graph and Visual Instances

Below, we describe how identities are transferred in the submission, the case that it does not cover, and how we extend it.

Pipeline in the submission. In I_0 and in every keyframe, Grounded-SAM-2 is prompted with the category of each node together with its attributes in the graph (e.g., 'small, clear ice cube'), and the instance with the highest confidence is assigned to the node; if this confidence is below a threshold, the node is treated as not visible. In the preview, instances are paired with keyframe instances by centroid distance (Eq. 6). When each category occurs once in the scene, which holds for 81% of the scenes across both benchmarks, this mapping is unambiguous. When several objects of the same category also have similar attributes, however, the prompt does not distinguish them, so two nodes can be assigned the same instance, and an edit to one object can be measured and guided on the other. This case is uncommon in both benchmarks, but it matters for multi-object scenes, so we extend the pipeline as follows.

Node-level grounding. We ground every node to its own instance. (1) When parsing I_0, the VLM outputs a bounding box for each node; the node's mask is the Grounded-SAM-2 mask with the highest IoU with that box, and nodes of the same category are assigned one-to-one, so two nodes never share an instance. (2) In each keyframe, the node is prompted with the referring expression that we already produce for the image editor, which adds its relations in G_0 to its attributes (e.g., 'the ice cube on the tray'), and a spawned object is searched near its source, following its lineage. (3) The reference feature of each node is computed from its own mask, and the matching cost adds the cosine distance between instance features to the centroid distance, so two objects of the same category in different places or with different appearance are not exchanged.

Occlusion, crossing, and splitting. Matching is performed independently at each keyframe, so identities are not propagated through the frames between keyframes: objects that cross between two keyframes are matched again at the next one, where the feature term distinguishes them by appearance. An object hidden in a keyframe falls below the confidence threshold and therefore has no keyframe instance there, so its terms are skipped at that keyframe (Appendix A.6) rather than filled in or suppressed. Splitting is represented in the graph itself: once an object separates, its parts are spawned as new nodes with lineage to the original (Appendix A.1), and each part is grounded as above.

Evaluation. Because few scenes in the two benchmarks contain same-category objects with different expected outcomes, we construct a targeted set of 20 scenes, each with two or more objects of the same category whose expected outcomes differ (e.g., 'the ice cube on the left melts'), with initial frames generated as for PhyGenBench. On this set, we measure how often an edit acts on the wrong instance and how often identities are swapped between keyframes (Table R7). Node-level grounding more than halves wrong-instance edits (45% → 20%) and reduces identity swaps (35% → 20%). This analysis is included in Appendix C of the revised manuscript (highlighted in blue).

Table R7: Identity transfer on a targeted set of 20 scenes, each with two or more objects of the same category whose expected outcomes differ. Values are the percentage of scenes with at least one error.

Identity transferWrong-instance edits (%) ↓Identity swaps (%) ↓
Attribute prompt + top-1 instance (submission)4535
Node-level grounding + feature and centroid matching2020
Reviewer_xrzvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhysPlan is a training-free framework for physically plausible image-to-video generation. It grounds a physical event in an object state graph, represents changes through checked graph edits, and uses those edits to select the properties and regions constrained during test-time optimization. Experiments on PhyGenBench and Physics-IQ show improvements over the base generator and whole-frame guidance, supported by component ablations and a user study.

Critical strengths
  1. The graph provides a useful connection between physical planning and video control: it identifies what should change while retaining objects and properties that should remain stable. This is a sensible improvement over treating every detail of a generated keyframe as a target.
  2. The gains are substantial, and the comparison with Frame Guidance uses the same backbone and keyframes. The improvement from 30.3 to 38.2 on Physics-IQ, together with the human evaluation and quality measurements, supports the practical value of selective guidance.
Critical weaknesses
  1. The advance over existing physical planners is less convincing than the advance over whole-frame guidance. VLIPP already uses VLM-derived physical plans, while Frame Guidance supplies the underlying optimization approach. Their integration through a grounded graph is useful, but the reported VLIPP results do not provide a controlled comparison of complete planning systems with comparable reasoning and generation resources.
  2. The representation is better suited to discrete changes in object state than to continuous dynamics. Motion between anchor frames remains largely dependent on the video model; the weaker fluid-dynamics result relative to VLIPP illustrates this boundary. The authors acknowledge this limitation, but it narrows the physical capability added by the method.
  3. The proposed method is evaluated only with CogVideoX-I2V-5B. This establishes its usefulness for that generator, but leaves its applicability to other video backbones uncertain. Testing a second backbone, particularly one with stronger baseline physical performance, would clarify whether graph-guided optimization provides a broadly useful benefit.
Questions for authors
  1. How does PhysPlan compare with an existing physics-aware planner such as VLIPP using the same video backbone and a comparable VLM setup? Which types of physical interaction benefit most from the graph formulation?
7 comments
AuthorsOpenReview ↗

Response to Reviewer xrzv (Part 1/4)

Thank you for your constructive review, and for recognizing the grounded graph as a sensible improvement over treating every detail of a generated keyframe as a target. Revisions are highlighted in blue in the updated PDF. Your points concern the comparison with VLIPP under matched settings (W1, Q1), the scope of the graph formulation (W2, Q1), and other video backbones (W3), and we ran the experiments you suggested for each.

W1 & Q1: Comparing with VLIPP under a Controlled Setup

Since the submission reported VLIPP’s numbers from its paper, the two systems also differed in the planning VLM, the generator weights, and how the plan reaches the VDM. To isolate the contribution of the planner, we re-ran VLIPP and PhysPlan under matched settings (Table R1).

Setup. All runs generate 49 frames at 720x480 with the same seed. VLIPP's initial frames for PhyGenBench are not released, so every method uses the Gemini-generated initial frames that we use for all I2V methods in Table 2. We vary three factors: (1) the planning VLM, GPT-4o as in VLIPP or Gemini 3 Flash as in PhysPlan; (2) the generator weights, the base CogVideoX-I2V-5B or the Go-with-the-Flow weights used by VLIPP; and (3) how the plan is executed.

Executing each plan. VLIPP conditions the VDM through noise warped along its planned trajectories, while PhysPlan optimizes the latent at test time. Since PhysPlan has no trajectories along which to warp noise, it samples standard Gaussian noise on the Go-with-the-Flow weights and applies graph-guided optimization.

Table R1: Controlled comparison with VLIPP. Rows vary the planning VLM, the generator weights, and the plan execution. PhyGenBench (PCA) and Physics-IQ averages; runtime per video on one H100.

PlannerVLMGenerator weightsPlan executionPhyGenBench ↑Physics-IQ ↑Time (s)
VLIPP (reported)GPT-4oGo-with-the-FlowWarped noise0.6034.6–
VLIPP (re-run)GPT-4oGo-with-the-FlowWarped noise0.6134.3128
VLIPP (re-run)Gemini 3 FlashGo-with-the-FlowWarped noise0.6434.9121
PhysPlanGPT-4oCogVideoX-I2V-5BGraph-guided (ours)0.7338.0370
PhysPlanGemini 3 FlashGo-with-the-FlowGaussian noise + graph-guided0.7838.1364
PhysPlan (submission)Gemini 3 FlashCogVideoX-I2V-5BGraph-guided (ours)0.7738.2366

Reproduction. Our re-run closely matches the published numbers (0.61 vs. 0.60; 34.3 vs. 34.6) despite the different PhyGenBench initial frames, so the differences below are not artifacts of our reproduction.

Effect of the plan. With the same VLM and generator weights (Gemini 3 Flash, Go-with-the-Flow), PhysPlan outperforms VLIPP by 0.14 on PhyGenBench and 3.2 on Physics-IQ. VLIPP asks the VLM for quantitative trajectories, i.e., bounding-box coordinates over time, which VLMs estimate unreliably (Li et al., 2025), and these trajectories only describe where objects move. PhysPlan instead asks the VLM only for qualitative judgments and plans how the observed scene changes, including changes of state (e.g., ice turning into water) and passive objects that the prompt does not mention (e.g., the tray under the ice). Changing the VLM has a small effect, and PhysPlan with GPT-4o still outperforms every VLIPP setting, so the gain comes from what is planned, not from which VLM plans it.

Cost. PhysPlan takes about 3× the runtime of VLIPP (366 vs. 121 s) because it optimizes the latent at test time; its runtime is comparable to Frame Guidance, which also optimizes at test time (310 s, Table R4).

Table R1 and the setup are included in Appendix B of the revised manuscript (highlighted in blue), and the comparison with VLIPP in Section 4.2 is revised accordingly.

AuthorsOpenReview ↗

Response to Reviewer xrzv (Part 2/4)

W2 & Q1: Clarifying the Scope of the Graph Formulation

By design, PhysPlan specifies the state of the scene at the keyframes and leaves the motion between them to the VDM. To make this scope precise, we show below which physical interactions benefit from the graph formulation, test whether the Fluid Dynamics gap depends on the number of keyframes, and describe how we clarify the scope in the revision.

Which interactions benefit (Q1). To answer this question with an established taxonomy, Table R2 breaks down the PhyGenBench results by the physical laws defined in the benchmark (Meng et al., 2025). The gains are largest for laws whose outcome is a discrete change of object state, such as a change of color, a phase change, or a collapse (e.g., Atmospheric Pressure, Color, and Solid Pressure), and smallest for laws whose outcome is a light pattern or a medium without a clear object boundary (e.g., Dispersion +0.02, Tyndall Effect +0.04, and Interference & Diffraction +0.10). Among mechanical laws, the smallest gain is on Gravity (+0.12), whose outcome is continuous motion rather than a change of state.

Link to Fluid Dynamics. This pattern matches the Fluid Dynamics category of Physics-IQ, which measures continuous motion more directly and where the gain of PhysPlan is the smallest among Solid Mechanics, Fluid Dynamics, and Optics (+2.0). This difficulty is not specific to PhysPlan. In the results reported by VLIPP (Yang et al., 2025, Table 2), trajectory planning improves Solid Mechanics by +11.9 (30.4→42.3) but Fluid Dynamics by only +4.3 (29.8→34.1). VLIPP also notes that events not representable by bounding-box trajectories are beyond its scope. Fluids have no stable object boundary, so neither object-level states nor box trajectories describe them fully. Recent benchmarks make the same observation: a fluid scene combines several constraints, such as flow, boundary interaction, and continuity, and a generated video can show plausible flow while making liquid appear or disappear (PhyGround, 2026). We analyze this further in the next comment.

Table R2: PhyGenBench results per physical law, using the law labels of the official PhyGenBench prompts (Meng et al., 2025). Domain rows repeat Table 2 of the submission; law rows are computed from the same videos. Base: CogVideoX-I2V-5B.

Domain / physical lawPromptsBasePhysPlanGain
Mechanics (all)400.480.81+0.33
Gravity50.590.71+0.12
Buoyancy50.480.90+0.42
Solid Pressure50.520.92+0.40
Atmospheric Pressure50.390.88+0.49
Elasticity50.460.70+0.24
Friction50.480.85+0.37
Surface Tension50.460.68+0.22
Stress50.490.86+0.37
Optics (all)500.690.78+0.09
Reflection100.730.89+0.16
Refraction100.710.82+0.11
Tyndall Effect100.690.73+0.04
Dispersion50.640.66+0.02
Interference & Diffraction100.620.72+0.10
Direct Radiation50.730.83+0.10
Thermal (all)300.430.74+0.31
Solidification50.470.77+0.30
Melting50.480.86+0.38
Liquefaction50.430.69+0.26
Boiling50.400.79+0.39
Deposition50.410.59+0.18
Sublimation50.410.72+0.31
Material (all)400.410.75+0.34
Color50.420.90+0.48
Hardness50.430.74+0.31
Solubility50.500.62+0.12
Combustibility50.410.72+0.31
Flame Reaction50.370.75+0.38
Acidity and Alkalinity50.350.73+0.38
Redox Property50.410.78+0.37
Dehydration Property50.400.76+0.36
AuthorsOpenReview ↗

Response to Reviewer xrzv (Part 3/4)

W2 & Q1 (Continued): Clarifying the Scope of the Graph Formulation

Finer decomposition of continuous phenomena. In PhysPlan, the number of events, and therefore of keyframes, is decided by the VLM during phenomenon decomposition, so a continuous phenomenon such as a pour or a splash is often planned as only a few states. To test whether a finer plan reduces the gap, we add one instruction to the decomposition prompt (p_delta, Appendix A.3): “When the state of an object changes continuously, such as pouring, spreading, or splashing, split the change into successive intermediate states.” Since the instruction targets continuous phenomena, we evaluate it on the Fluid Dynamics category of Physics-IQ (Table R3).

Result. With the finer instruction, the VLM plans 6.3 instead of 2.6 events per video, and Fluid Dynamics improves from 31.8 to 33.9, close to VLIPP (34.1), at 24% more runtime. Most of the gap on fluids therefore comes from how coarsely continuous phenomena are planned. The remaining limitation concerns phenomena that are not changes of object state at all, which we now state explicitly.

Table R3: Finer decomposition of continuous phenomena through the stage-1 prompt, on the Fluid Dynamics category of Physics-IQ, the category that the instruction targets. Events and time are averaged over its 45 videos; time on one H100.

SettingEvents per videoFluid Dynamics ↑Time (s)
PhysPlan (submission)2.631.8324
PhysPlan + finer decomposition6.333.9507
VLIPP (reported)–34.1–

Revisions. In the revised manuscript (highlighted in blue), we make the scope of our claims more precise in the abstract and the introduction, clarifying that the graph formulation is most effective for phenomena that unfold as changes in the state of discrete objects. Section 5 now also discusses classes that lie outside this representation, such as fields, waves, and quantitative laws. We see combining the event chain with trajectory planning between keyframes, as in VLIPP, as a natural extension and discuss it as future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.