acceptodds
Under review as a conference paper at ICLR 2027

EnactPhys: Executing Evolving Physical Processes in Video Diffusion

59 views

Abstract

Physical control in video generation requires translating user-specified parameters into corresponding motion responses. These responses arise through an evolving physical process in which parameter effects depend on the current state. Existing parameter-conditioning approaches leave this intermediate process largely implicit. Motivated by this task structure, we introduce **EnactPhys**, a video diffusion architecture for **state-mediated physical-process execution**. In our rigid-body instantiation, object states evolve through parameter-conditioned updates and inter-object interactions. We also introduce **PhysDelta**, a benchmark that jointly measures the **accuracy** and **selectivity** of physical control in simulation and matched real-world experiments. EnactPhys leads all four PhysDelta control metrics, improving over the strongest baseline for each metric by **15.0%–37.7%**, while maintaining competitive video quality and physical plausibility. It achieves the highest Verified Score on Physics-IQ (Solid Mechanics) among the evaluated methods, and ranks first in Invariance Score and second in Dynamical Score on MORPHEUS. State intervention experiments further show that learned state transformations transfer to held-out inputs and steer generated motion toward that induced by the target parameters, while keeping the original physical parameters fixed. Project page: https://enactphys.github.io.

Primary area:
generative models
Keywords:
Video Diffusion, Physical Dynamics, Controllable Generation, Object-Centric Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4XFVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at physical control in video generation: you give the model physical parameters like force, mass, gravity, friction, and restitution, and the generated motion should change the way physics says it should. The key observation is that a parameter’s effect depends on what is happening in the scene at that moment, and the authors argue existing parameter-conditioned methods leave this state-dependent process implicit. They propose EnactPhys, which adds an object-state branch to a pretrained Wan2.2 video DiT. The branch keeps a latent state for each object over time, injects force, gravity, and mass through per-object temporal attention, and injects friction and restitution only through gated messages between objects in contact, then writes the updated states back into the video features. Training uses about 110K synthetic MuJoCo/Blender videos, with per-frame masks and contact labels as extra supervision.

Besides the framework, the authors introduce PhysDelta, a benchmark that keeps the same initial image, changes one parameter at a time, and checks both whether the motion changes as expected and whether things that should stay the same actually do, such as pre-impact motion or other objects’ motion. It has a simulated split with numerical ground truth and a real-image split scored on direction only. EnactPhys beats all baselines, including same-backbone, same-data variants, on all four PhysDelta control metrics while keeping video quality on par, and does well on Physics-IQ and MORPHEUS.

Critical strengths
  1. The paper identifies a real and well-motivated gap in physical control for video generation. Its core observation is that a parameter’s effect depends on the evolving scene state, for example restitution only matters once a collision happens. Framing the goal as both accuracy and selectivity gives a clear and useful way to define what correct physical control means.

  2. I think the PhysDelta evaluation protocol is a useful contribution that other work could adopt (for example, counterfactual analysis?). It keeps the same initial image and changes one parameter at a time, which makes generated motions directly comparable across parameter settings. It also explicitly checks whether unaffected phases and non-target objects stay unchanged, a property that is important for controllable world models.

  3. The main comparison is well controlled and appears well. The Wan(Text-only) and Wan(ControlNet) baselines share the same backbone, training data, and step budget as EnactPhys, so the comparison isolates the effect of how parameters are injected. The gap over these variants is large, especially on Object Control. EnactPhys maintains strong generation quality alongside its control gains. Video quality and artifact rates are on par with the strongest fine-tuned baselines, and the model also performs well on external benchmarks such as Physics-IQ and MORPHEUS.

  4. I think the explicit object-level states make the model easier to probe and manipulate. The state-intervention experiments show that editing the learned states steers the generated motion while the input parameters are held fixed, which indicates that these states carry causal information about the motion.

Critical weaknesses
  1. I think the writing currently holds the paper back somewhat. The same results and claims appear several times across the abstract, introduction, results, and conclusion, and it would help to use that space to explain what actually goes wrong when the state-dependent process is left implicit. The framing around "executing physical processes" also feels a bit stronger than the mechanism. In my view, the most convincing story in the paper is that contact-gated, role-separated parameter pathways drive the selectivity gains, and the paper could benefit from putting that at the center.

  2. It is somewhat difficult to separate the architecture's contribution from the additional supervision. EnactPhys uses mask supervision, contact-label supervision for the gates, and extra motion losses, and I could not tell whether the baselines were trained with the same losses. Could the authors clarify whether the All-to-All ablation retains any contact supervision? An ablation that removes gate supervision, and a version of the Wan baselines trained with the same motion losses, would make the attribution clearer.

  3. The contact-window and message-intervention experiments are useful, but the core motivation could be tested more directly through a quantitative test involving multiple physical parameters. The paper argues that parameter effects depend on the scene state, while PhysDelta varies one parameter at a time and the composed-event results are shown qualitatively. If possible, a small quantitative test, for example where force determines whether a collision occurs and the effect of restitution is measured accordingly, would nicely support the main claim.

Questions for authors

Please refer to the Weaknesses section; I would appreciate it if the authors could address those points in the rebuttal.

I am open to adjusting my score based on the authors’ response.

Reviewer_LbhfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EnactPhys designs a conditioning branch for fine-grained control over physical parameters, which fuses input parameters with object-centric video features via temporal attention and inter-object message-passing before injecting them back into a frozen video diffusion backbone. Complementing the model, the authors introduce PhysDelta, a benchmark for evaluating the accuracy/selectivity of the fine-grained physical control. Quantitative results show that EnactPhys outperforms existing baselines, while qualitative results demonstrate smooth responses to parameter variations and different parameter combinations.

Critical strengths
  • The proposed benchmark addresses a meaningful, new task, providing systematic evaluation of fine-grained physical controllability with detailed metrics defined alongside.
  • Incorporating relational physical parameters through inter-object message passing provides a structural inductive bias which seems to be beneficial for modeling inter-object, contact-dependent dynamics.
Critical weaknesses
  • Inter-object message passing seems to be a key distinction from prior methods, enabling more precise control of contact interactions. While the ablations show improvements over all-to-all attention (is this one attention layer across object and time, or do you also do first temporal then cross-object attention layers?), it remains unclear whether this structure learns meaningful relational dynamics or mainly improves through additional explicit control signal injection.

  • Despite focusing on rigid-body dynamics, some qualitative results have noticeable deformation, interpenetration, and visual artifacts (for humans). These issues are not acknowledged in the limitations and seem inconsistent with the high physical-plausibility scores, raising concerns about the sensitivity of the VLM-based evaluator.

  • Fairness of baseline comparisons may be affected by differences in conditioning interfaces, as unsupported physical controls are provided through structured text. In particular, Phantom may be disadvantaged by text-based conditioning, making it difficult to isolate architectural improvements from differences in control inputs. Additionally it would be great if the qualitative comparisons included explicit physical-control baselines (Force-prompting, PhyCo), which may help better understand the reported gains.

Questions for authors
  • Is there a particular masking/dropout strategy applied for sampling different parameter combos during training? Is there a systematic evaluation on performance for held-out parameter combinations versus those seen during training?

  • Is EnactPhys responsive to time-varying control? The paper mentioned windowed injection of parameters, is EnactPhys capable of modeling parameters changes over consecutive windows (e.g. sliding across a high friction to low friction surface)?

  • EnactPhys claims that the "object states" in its control branch capture dynamics and investigates this through speed decoding in Sec. 5.2.3. How do these representations compare to latents from baselines such as Phantom, which also claims that its V-JEPA2-based embeddings capture dynamics? A comparison would make the claim more convincing.

  • As mentioned in the weakness section, it is slightly unclear what the structural bias from inter-object message passing brings over simple inter-object attention. Is the structural bias in EnactPhys capable of extending inter-object modeling to a greater number of objects in a zero-shot manner?

  • What is the "structured-text format" used to prompt baselines?

Reviewer_DLwNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that physical control in video generation is not a static conditioning problem but an evolving process: whether and how a parameter (e.g. restitution) affects motion depends on the current scene state (e.g. whether contact has occurred). Existing parameter-conditioning methods leave this intermediate process implicit inside the generator; this paper proposes to make it explicit.

Critical strengths
  1. The "selectivity" axis of the benchmark genuinely fills a gap. Existing physical benchmarks (WorldModelBench / Physics-IQ / PhyGround / MORPHEUS) are almost all plausibility- or reference-based, and do not test whether, after changing one parameter, the things that should not change actually stayed unchanged. PhysDelta splits invariance (unaffected motion phases) and object selectivity (non-target objects) into independent metrics, and the positioning in Table 1 holds up. On Object Control, Wan2.2 scores only 2.76 while EnactPhys reaches 72.64 — the axis is highly discriminative, and current models genuinely cannot do this.
  2. The controlled comparisons are better constructed than in most comparable papers. The three Wan2.2 baselines (Text-only / ControlNet / reproduced Phantom) use the same initialization, the same 109,800 training videos, the same 6,000 steps, the same resolution, the same output length, and paired seeds. This is considerably cleaner than the many papers that compare directly against other groups' released checkpoints.
Critical weaknesses
  1. The improvement is concentrated on the authors' own, distribution-matched benchmark, and the external evidence is weak without being acknowledged as such. On PhysDelta the margins are large (15-38% relative). Off PhysDelta they largely vanish:
  • Physics-IQ Verified (Solid Mechanics): 48.16 versus Phantom 46.22 and PhyCo 44.87, a +1.94 point margin with no error bars, despite three seeds being available.
  • MORPHEUS: Dynamical Score 0.6781, clearly behind MiniMax H3 at 0.8286; Invariance 0.7111 versus 0.7035, essentially a tie.

Meanwhile PhysDelta-Sim uses simulated or scene-constrained synthetic initial images, and the training corpus is itself MuJoCo + Blender Eevee plus PhyCo data and its simulation pipeline. The evaluation is therefore substantially in-distribution with respect to training.

What makes this more serious is what was not evaluated. Table 1 itself lists What-If World, Cronos and PhysWeep, the benchmarks closest to this paper's task in that they are intervention-based and, for What-If World, explicitly score unaffected behavior. None of them is evaluated. The two external benchmarks that are used (Physics-IQ, MORPHEUS) are reference-based, that is, the least aligned with the paper's own stated contribution.

As it stands the evidence supports large gains on in-distribution rigid-body parameter control, not the broader claim of state-mediated physical controllability. Either evaluate on What-If World or Cronos, or narrow the claim accordingly.

  1. There is no ground-truth ceiling for PhysDelta, so the metric scale is uncalibrated.

The Sim magnitude score maps pixel-space measurement ratios onto physics-law-predicted ratios via

N(r) = (r - 1) / (r + 1)
S_mag = 1 - (r_measured - r_expected)^2 / (r_measured^2 + r_expected^2)

This mapping is sensitive to camera geometry, perspective and scene scale. The paper never reports what score the MuJoCo ground-truth rendered videos themselves receive under PhysDelta. Without that number, 56.91 is uninterpretable: if the ground truth scores near 100 the metric is tight and there is clear headroom; if the ground truth scores around 70 the metric is dominated by measurement noise and the ranking is far less meaningful.

Relatedly, N(r) is noise-dominated as r approaches 1 (small intended changes), and the denominator r_measured^2 + r_expected^2 amplifies error when both ratios are near zero. Please report a noise floor, for example the distribution of S_mag across different seeds under identical parameters.

This is a single additional experiment, but I consider it necessary for the benchmark contribution to stand.

  1. Parameter Control conflates generation failure with incorrect control direction.

PhysDelta-Real PC is a directional accuracy, for which chance is 50%. Yet Wan2.2-TI2V-5B scores 15.25, far below chance, because unreadable responses contribute zero (Section C.3). The metric is therefore effectively

PC = P(response readable) x P(direction correct | readable)

which entangles controllability with raw generation quality and motion magnitude. Please report the readable-response rate and the conditional directional accuracy separately.

The same concern applies to S_object = c * d_target / (d_target + d_other): a model that generates nearly static video drives d_other towards zero and is structurally favored, with c only partially compensating. Please report the absolute magnitudes of d_target and d_other, not only their ratio, and state the repeat-derived tracking threshold numerically.

4 The two representation-level claims lack their essential controls.

(a) The linear probe has no backbone control. R^2 = 0.811 for decoding speed from object states is reported, but not what the same ridge probe achieves on object-region-pooled features from the frozen Wan backbone. Video features already encode motion; without this baseline, "object states encode speed" carries no information about what the state branch adds.

(b) The affine state transformation has no random-direction control. The 94.6% trajectory-MSE reduction should be compared against applying a random affine map of matched norm. The experiment is also somewhat circular: the reference condition is the parameter-intervention output, and the affine map is fitted precisely between states under the two parameter values. A random-direction control would establish that the effect is specific rather than a generic consequence of perturbing the state.

These two controls are what I would most like to see in the rebuttal.

  1. Positioning relative to object-centric and graph-based physics modeling is missing.

The pipeline Reader -> per-object temporal update -> typed gated pairwise messages (contact / restitution / friction) -> Writer is, structurally, an Interaction Network (Battaglia et al., 2016) or Graph Network Simulator (Sanchez-Gonzalez et al., 2020) grafted onto a DiT, with a read/write interface in the tradition of object-centric latent dynamics (Slot Attention, SAVi, SlotFormer, C-SWM). None of this literature is cited.

This does not make the work unoriginal, but the absence of positioning makes "explicitly modeling object states and interactions" read as a novel idea, when it has been standard practice in the object-centric community for years. In my reading the actual delta is: bidirectional read/write coupling of an object-centric latent simulator with a frozen large-scale video DiT, with per-frame masks and contact labels supervising the spatial routing and interaction gating. That is a defensible and interesting contribution, and the paper should state it explicitly rather than leaving the reader to extract it. Additional physics-grounded image-to-video work (PhysDreamer, PhysMotion) would also strengthen Section 2; PhysGen is already cited.

  1. Capacity is not matched or reported, and input availability differs across methods.
  • No trainable parameter counts, FLOPs or inference-time measurements are given. EnactPhys adds eight modules; Wan (Text-only) adds almost nothing. All-to-All State Attention partially serves as a capacity-matched control, but the paper does not state whether its parameter count matches EnactPhys. Please add a parameter and compute table.

  • EnactPhys requires first-frame object masks and explicit object/relation parameter binding at inference, whereas CogVideoX-5B-I2V and MiniMax H3 receive only structured text. This is not disqualifying, since the ControlNet baseline also receives spatial maps, but Table 2 should include a column specifying the input modality available to each method (text / spatial map / mask / relation binding), otherwise the comparison is not like-for-like.

  • The inference-time mask requirement is a practical limitation and belongs in Section I, which currently discusses only deformable bodies, fluids and long-video generation.

W7. (Major) The human verification protocol is confounded.

Appendix H states that the reviewer watched the paired videos "alongside their physical conditions, automatic motion readouts, and scores, and judged whether the automatic assessment agreed with the visible motion". Showing the automatic score while asking whether the automatic score is correct invites confirmation bias; the reported 97.19% agreement is therefore weak evidence for metric validity.

Similarly, the Physical Plausibility agreement (91.67%, Cohen's kappa = 0.8082) rests on a single annotator, so no inter-rater reliability is available. Please use blinded presentation with scores withheld, at least two independent annotators, and report inter-rater kappa.

Separately, judging penetration, deformation or disappearance from 16 frames arranged in a 4x4 grid is temporally coarse; the prompt itself concedes that "later absence alone is not proof of disappearance", which amounts to acknowledging a systematic blind spot. At minimum, FAIL cases should be re-checked on the full video.

Questions for authors

see the above weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.