acceptodds
Under review as a conference paper at ICLR 2027

PHASEPLAY: CONTROLLABLE 3D SCENE SYNTHESIS WITH THERMAL-DRIVEN PHASE TRANSITIONS

Abstract

Generating physically grounded videos of material transitions remains a significant challenge because both the geometry and appearance of objects evolve dynamically over time. Current approaches often face a trade-off: pre-trained diffusion models provide realistic appearance but frequently suffer from physical hallucinations and offer limited controllability for users, while traditional physics-based simulators excel at motion but struggle to capture complex visual state changes. To address this, we introduce PhasePlay, a framework for action-conditioned 3D scene dynamics that models realistic phase transitions by integrating specialized Material Point Method (MPM) and Position-Based Dynamics (PBD) solvers with an AI-assisted reasoning module. Our methodology combines the structural rigor of classical solvers with the adaptive capacity of AI to maintain precise control over both physical movement and evolving appearance. While we recognize the inherent complexity of full-scale multi-physics, evaluations against state-of-the-art baselines, such as Sora-2 and WonderPlay, demonstrate that PhasePlay achieves superior visual quality and physical fidelity across diverse interactive scenarios.

Primary area:
generative models
Keywords:
Action-Conditioned Video Generation, Physics-grounded, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_f1P3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work deals with the problem of generateing visually impressive and physically plausible videos of phase transitions from image(s). The authors build a pipeline, combining the abilities of vlm reasoning, physical simulation, and video diffusion models: vlm decides the physical process and final visual results, physical simulators update the physical states based on the reconstructed 3D scenes, video diffusion models improve the visual details to make the results more realistic. The main contributions are the integration of approximate phase-transition simulation and the temporal Gaussian mapping mechanism. Authors compare their methods with several video generation baseline models and WonderPlay, qualitative and quantitative results are competitive.

Critical strengths

Video generation of phase transitions is a hard problem. The presented results in this work demonstrate that the proposed pipeline can deal with the phase change and produce visually convincing results.

Critical weaknesses
  1. The claimed physical grounding is weak from the thermal part. The paper neglects the modeling of thermal effects like temperature evolution and heat transfer. Consequently, it's difficult for the method to model spatially varying transitions and boundary conditions, such as melting that begins near a localized heat source.
  2. The ablation of the method is sufficient. Authors didn't show the intermediate results of each pipeline components, e.g., the generated rgb before vdm refinement and how the gaussian temporal updater makes effects.
  3. The video model baselines authors compared in the paper are weak. The authors should test several much better methods, like Runway4.5, Seedance2.5.
Questions for authors
  1. Could the authors clarify the amount of manual configuration and tuning required to produce the reported results? In particular, which simulation parameters are inferred automatically, and which must be specified or adjusted for each scene?
  2. The anonymous link in the paper is expired. Could the authors provide a working link to the video results?
  3. Regarding the writing, the authors spend a lot of efforts connecting this method with "World model", which i see as irrelevant, what this work tries to deal with is just a specific video generation task.
  4. Although the solver formulation includes a temperature field in 3.2, the described phase-transition mechanisms mainly use time-dependent schedules for dynamic parameters like stiffness and damping. Could the authors clarify how temperature is updated and are these time-dependent schedules prescribed per scene or derived from thermal conditions?
  5. How does the method handle the gap between the simulated geometry and the AI-generated target state? Since optimal transport establishes spatial correspondences, could the authors discuss or demonstrate the method’s robustness to such mismatches?”
5 comments

Thank you for your comments! Please first check out Our "Reply to the major comments from all reviewers". We address each point from you below:

For weakness #1, We agree that PhasePlay does not model heat transfer or spatially varying thermal fields. The phase-change mechanisms are phenomenological approximations, and we have revised the abstract and introduction to state this. As mentioned in the "reply to all reviewers", This simplified design is deliberate, because it lets users without substantial physics knowledge operate the pipeline. The reasoning module handles the parameter choices, and the video refinement stage handles appearance. We agree that some scenarios need finer control over the physics. The Limitations section notes this, and the field-valued extension (replacing the scalar λ with a spatially varying field) is a natural direction for users with sufficient physics knowledge. We consider this extension as a less common scenario, which is not considered in this work. However, we have modified the conclusion to admit this limitation.

For weakness #2, We agree that intermediate outputs help readers assess each component. The added Appendix G now shows, for each example, the input and inpainted background, the depth map, the SAM mask candidates, the object multiview, the simulated flow magnitude, the Gaussian-updated render, and the final video. These results show that the predictions at some stages are sometimes incorrect. However, the correct candidate results is always shown in the output pools for user to select. Hence the pipeline can provide reliable results as long as each stage ouput is confirmed by users.

For weakness 3, We have added Seedance-2.5 in Appendix C, with qualitative results in Figure 4. Seedance-2.5 renders the wood ignition and ocean wave examples smoothly, but shows physical and thermodynamic inconsistencies in the other cases, including butter melting and the ice-related scenes. This still show that exisitng state-of-art is not sufficinet / reliable for generating physics-plausible video.

For the Q1, please check out the main reply further, The reasoning module generates the simulation hyperparameters for each scene with AI reasoning automatically.

For Q2, We apologise. the link is activated now.

For Q3, We agree that the task studied here is physics-aware video generation from a single image with user-specified actions. We mention world models in the related work because they also predict future states from action inputs, which partially falls in this category. We will consider reducing the length or removing in the final version.

For Q4, we apologize the confusion, we tried to make the framework more general in theory and we admit that is our writing incorrectness that cuase the confusion. In practice, the calculation is simplified: the AI reasoning module takes the stated thermal condition T(t) and generates the schedule parameters for each scene / physics, completely by AI reasoning. Then the particle states are updated based on the schedule parameters and other input such as forces.

For your Q5, Optimal transport establishes the correspondence between simulated particles and the target Gaussians. Appendix G shows cases where the shape changes less, and the mismatch is acceptable there. However, we want to highlight that the final state is just a reference for how the new generated particles should look like in RGB color, in the end the final video is still largely controlled by the simulation results, as shown in the Appendix G. So the model will not get confused under this case.

Thanks for you reply! According to you reply and revised manuscript, I have 2 follow-up questions:

  1. Fidelity of the final video to the simulated motion. In some of the intermediate results in Appendix G like Fig.11/12, the motion in the final video appears to differ noticeably from that in the simulation render. To what extent does the video diffusion model override the simulated dynamics during refinement?

  2. Reliability of AI-proposed simulation parameters. The response states that the reported examples used AI-proposed parameters without manual adjustment. Since several parameters jointly determine the simulated behavior, and their effective values in an approximate solver may differ from real-world material values, I doubt the reliablity of a one-shot vlm inference. Could the authors report the acceptance criterion or the first-attempt success rate across the evaluated scenes?

Thanks so much for your follow-up questions!

For your Q1, as we state in the paper, we use the same pipeline as WonderPlay for final video refinement. The coarse render and the simulated flow reach the video generator through two separate control paths:

  1. Motion path (flow-warped noise). The pixel-space flow F, rendered from the particle velocities, warps a Gaussian noise sequence into structured noise N(F). This noise is fused with random noise, and Go-with-the-Flow uses it as the starting noise, so the simulated motion is imposed on the generated frames.
  2. Appearance path (SDEdit). The coarse video Ṽ, rendered from the Gaussian-updated particles, is noised to timestep s₁ and then denoised jointly over all frames from that point, following WonderPlay Eq. (4): V_{s₁} = α_{s₁} Ṽ + √(1 − α²_{s₁}) N(F). The render is the starting point of denoising, not a conditioning signal applied throughout, and a larger s₁ gives the generator more freedom to change it.

We totally agree with your observations, but since the render is just the starting point of denoise, it is ok that the rendered video is not what we expect for the final video. This is the expected behaviour of the appearance path, and it is controlled by s₁. By your comment we think that you are referring to the ocean wave example. If you check out our Figure 3 - the predicted final state of this case, we want to hightlight that the predicted final state (Figure 3e) is the main source of the mismatch: the generator is pulled toward an appearance that the final-state prediction did not capture well. We will attribute it to the “very off” prediction of the final stage by AI, instead of our pipeline failure. If the users provide better “expected appearance of the ice”, the results will be better.

For your Q2, we really appreciate your questioning about reliablity of a one-shot vlm inference. Indeed, we consider that to let the model create fully controlled and satisfied physics-grounded video in one single shot, there is quite a long path to go. We apologize if any sentence in our paper makes you think that is our claim. As we mention in the main reply, our user interviews indicate that “realistic” or acceptable physics-grounded video is highly subjective and varies from person to person. For this reason, we believe it is still currently difficult to quantify an acceptance rate in a strict and scientific way, based on the state-of-arts.

However, all the results in Figure 3 is actually the best of three runs per model, with slightly different prompts. We already reported the best results we got from each model. For our pipeline, if we consider our shown video is “acceptable”, roughly our success rate of the first attempt on average is about 66% (one run will be off and not good per example). However, the results have shown that our model indeed can provide better control on the final video creation, compared with similar methods / generative AI models without our pipeline, considring all the shown results of the baseline is not “acceptable” with the success rate of 0% in three runs.

We hope these can answer your questions well, and please feel free to ask us more questions if there are still any confusions!

Reviewer_7WMtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhasePlay aims to generate action-conditioned dynamic scenes involving material phase transitions from a single image. The framework combines scene reconstruction, visual reasoning, extensions to MPM/PBD physics solvers, and diffusion-based video refinement. The paper evaluates the framework across ten scenarios and reports improved results.

Critical strengths

The paper addresses a meaningful problem: generating material transitions requires coordinated changes in motion, geometry, and appearance. Extending existing physics solvers and combining their outputs with appearance guidance is a reasonable approach to this problem.

Critical weaknesses
  1. The discussion of existing methods relies predominantly on earlier work and does not sufficiently cover recent approaches to physics-informed video generation. For example, “Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment” explores another approach to incorporating physical information into video generation, while “Physical Simulator In-the-Loop Video Generation” also integrates a physical simulator into the video generation pipeline. My concern is not the publication years themselves, but whether the paper adequately establishes its distinctive contribution relative to these approaches. The authors should clarify these distinctions.

  2. A central contribution of the paper appears to be the extension of existing solvers to more general scenes. This is a valid direction, but the boundary between general mechanisms and scene-specific engineering remains unclear. Appendix A provides example configurations, yet it is difficult to determine how these configurations are derived from user inputs or model predictions, how much manual tuning they require, and how well they transfer to new scenes.

  3. The pipeline depends on several upstream components, including reconstruction, visual reasoning, and target-state appearance prediction. Errors in material identification, transition prediction, or target appearance could affect subsequent simulation and video generation. The paper does not sufficiently examine how these errors propagate through the pipeline. The selected successful examples therefore provide limited evidence of reliable performance across varied inputs.

  4. The manuscript presents sampled frames and an anonymous project link, but I could not access the linked page and therefore could not inspect playable videos. For a video generation paper, static frames are insufficient to assess temporal coherence, motion quality, transition continuity, and flickering. This substantially limits my ability to evaluate the claimed advantages. Accessible supplementary videos, including complete comparisons under matched input conditions, are needed.

Questions for authors
  1. What is the principal new contribution relative to recent physics-informed video generation methods? Please clarify how the proposed approach differs from these methods.

  2. How much manual configuration is required for a new scene? Which material parameters and transition schedules are inferred automatically, and which require scene-specific tuning?

  3. How does the pipeline behave when visual reasoning or target-state appearance prediction is incorrect? What evidence do you have regarding robustness and error propagation under these conditions?

  4. Can you provide an accessible alternative link to the complete comparison videos? Access to these videos would affect my assessment of both the demonstrated quality and the reliability of the framework.

3 comments

Thank you for your comments! Please first check out Our "Reply to the major comments from all reviewers". We address each point from you below:

For the weakness 1, In the revision, we have added Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment and Physical Simulator In-the-Loop Video Generation to the related work. And we would like to also clarify how each differs from PhasePlay.

  1. Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment is a black-box video generation model with predefined domain knowledge. Like our benchmarked baseline VLIPP, it does not give users control over the physics simulation, and it aims to produce more physically grounded video through learned reasoning.
  2. Physical Simulator In-the-Loop Video Generation shares our use of scene reconstruction. It does not include an AI reasoning component that introduces content absent from the input scene, such as a phase change. Its demonstrated scope is limited to undeformable object manipulation.
  3. Our work is trying to combine pure AI reasoning and physics simulation in-the-loop method to provide a more reliable solution.

For the weakness #2, The AI reasoning module generates the simulation hyperparameters for each scene automatically. The three phase-change mechanisms are the same across all scenes. Appendix A lists the values used for each scene, which is completely generated by our AI reasoning mechanism.

For weakness #3, We agree that the original submission did not measure error propagation (as explained in the comment reply to all the reviewer, we assume that user check and confirm at each stage.). The revision adds intermediate results for each example in Appendix G, including the SAM candidates, object multiview, simulated flow magnitude, Gaussian-updated render, and final video. The pipeline presents candidate outputs at each stage, and the user reviews them before the final video is produced. We acknowledge that predictions at some stages are sometimes incorrect, and a systematic study of error propagation remains future work. However, we also want to highlight that this work focus on controllability, hence, we only need to make sure that users confirm each step output to be correct, and we can ouput the final satisfactory video.

For your Q1, same as weakness #1, It is a novel pipeline as a combination of AI reasoning and physics-simulation-based video generation, with user control over each stage of the pipeline. For Q2, As stated in our response to Weakness 2, the reasoning module infers the simulation hyperparameters for each scene. In all examples in the paper, we used the AI-proposed values without manual adjustment. For Q3, Appendix G shows cases where we have incorrect intermediate prediction: a. the first mask candidate is not the intended object; b. Figure 11 shows artifacts in the wave example, etc. However, the correct output is always shown in the candidate pools for the user to select. We admit that the final output video relies on the user reviewing each stage’s candidate outputs to catch such errors before rendering. But since correct candidate results always exists, we consider the final correct output will always be generated. For Q4, the link is activated now.

Thank you for your rebuttal and clarifications. However, my concern about error accumulation remains insufficiently addressed. Relying on users to inspect intermediate outputs does not establish how reliably upstream errors can be detected and corrected, or how they affect the final results. I therefore maintain my original score.

Thanks for your feedbacks! We respect your decision, but we would think that this is a common limitations of all "simulation-in-the-loop" method, and we believe that none of the exisitng methods combining simulation with pre-trained model can guarantee that no errors will happen in the pipeline.

On the other hand, the pure data-driven video model also seems unable to guarantee that no errors propagate through the diffusion process. And that is our motivation: to ground the un-controlled generation with the simulation framework for complex physics simulation. Compared with complete black box, "user visualize and inspect the intermediate results" seems more reasonable considering current state-of-art.

Reviewer_Qcw9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PhasePlay generates dynamic 3D scenes and videos from a single image, focusing on material phase changes. It combines an AI-generated final-state image, approximate MPM/PBD simulation, and a video diffusion model. Its main additions are scheduled changes to material behavior and a Gaussian updater that blends initial and final appearances using optimal transport.

Critical strengths
  1. The paper addresses a useful problem: generating material changes that affect both motion and appearance. This extends single-image animation beyond moving or bending objects and could be useful for visual effects.
  2. The Gaussian updater links moving particles to initial and final appearances and handles newly added particles. This offers a practical way to change material appearance during simulation, though its benefit needs further testing.
Critical weaknesses
  1. The claims are stronger than the method supports. Phase changes use scheduled stiffness changes, particle emission, and velocity damping, but the paper does not explain how temperature controls these changes. This weakens its main claim of physically consistent, heat-driven simulation.
  2. The ablations only vary update intervals and timesteps. They do not test the separate effects of final-state images, phase-change rules, or optimal transport. The source of the reported gains therefore remains unclear.
Questions for authors

How does temperature control phase changes, and which results show benefits from the simulator beyond final-state appearance guidance?

1 comment

Thank you for your comments! Please first check out Our "Reply to the major comments from all reviewers". We address each point from you below:

For your pointed weakness #1, We agree that the manuscript overstates the physics. PhasePlay is a phenomenological approximation, not a heat-transfer solver, and we have revise the abstract and introduction to say so. In PhasePlay, the AI-based physics reasoning module takes the input image, the text prompt, and the stated thermal condition, and generates the simulation hyperparameters to drive the particle-based simulator. The user adjusts only the parameter that sets how fast each transition occurs.

For your pointed weakness #2, in terms of the ablation studies that you are interested in, we have attached more intermediate results in Appendix G, as mentioned in the main reply. In particular, for the factors that you are insterested in, we consider that:

  1. Final-state images effect. Without the predicted final state, the Gaussian temporal updater has no final-state anchor, so only the initial-state mapping remains. The pipeline then reduces to the original WonderPlay setting, which is already reported in Table 1 and Figure 3.
  2. Phase-change rules. The AI reasoning module sets all phase-change parameters. The simulator’s rules convert those parameters into particle motion, and the timestep analysis in Section 4.3 and Appendix F shows that simulator output strongly affects video quality. Consider everything is generated automatically, we do not intend to run ablation studies on them.
  3. Optimal transport. Nearest-neighbour assignment does not preserve correspondence theoretically when shape and volume change substantially, as in phase transitions, so we use OT. However, for this mapping mechanism this will be a big topic to evaluate all possible options, hence we only revise conclusion to mention more indepth research needed for this.

For your question, we consider that our comparison with Google flow, that use initial state and final state to make video, can show that: with final state prediction is not good enough. To “interpolate” well between first state and final state, the physics simulation is still required.

We hope that our answer can solve most of your comments, and we appreciate any more questions and comments about our work!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Reply to the major comments from all reviewers

As the title indicates, PhasePlay focuses on controllability. We believe the realism and physical plausibility of AI-generated video are largely subjective and depend on the viewer, since different people have different expectations of how a physical process should look. Our main contribution is therefore to give users control over the video generation pipeline, so they can inspect and adjust the physical parameters before rendering rather than accepting or rejecting the output afterward. This differs from video generation models, which are black boxes: PhasePlay is a more transparent pipeline that can be combined with different stronger generative models.

We also do not intend the framework to be fully manual. Our methodological contribution is a straightforward: AI reasoning + approximated physics framework. A detailed physics engine that closely models real-world physics would require users to have substantial physics knowledge and to set many parameters. With AI reasoning do the most of the hyper-parameter choice under our approximated physics framework, our pipeline needs only three user-facing parameters, which control how fast each transition occurs: the transition rate λ for fluid-solid transitions, the particle injection rate per timestep for fluid/solid-to-air transitions, and the damping coefficient η for air-to-fluid/solid transitions. All other parameters are proposed by the AI reasoning module and can also be changed by the user. In all examples in the paper, we used the AI-proposed values without manual adjustment. We acknowledge that the original manuscript did not make this clear, and we have revised Section 3.2 and Appendix A to describe how parameters are set.

We also want to explain to two main concerns raised across the reviews:

  1. Comparison with stronger video models. As mentioned before, we want to show the performance improvements using our pipeline. Consider that our AI reasoning part simply use Chatgpt-image-1.5, we do not select strong commercial video generative model as our benchmark method. However, to show that "physics consistency of generated video" is still the topic that is not fully solved, We have added a comparison with Seedance-2.5 (as a representative of strong commercial model) on the same scenarios in Appendix C in the revision.
  2. Reliability of pipeline components. We did not originally study the reliability of each component, because the pipeline design assumes user oversight at each stage. However, we agree that intermediate results are useful for readers to assess the pipeline, and we have added them to the appendix G: including all the intermediate results: impainted background, depth estimation, SAM detection, object multi-view, simulted particle flows, rendered video of Gaussian updating, and the final video output.

FInally, We apologise that the anonymous link, https://anonymous.4open.science/r/PhasePlay-388A, was not accessible during review. It has been reactivated. We hope the revised manuscript and these additional results address the major concerns raised, and we will reply to each reviewer to address the specific comments from different reviewers.

Dear Reviewers and Area Chairs,

As the discussion period nears its close, we would like to thank all reviewers again for your constructive feedback and summarize the core positioning of our work again in this comment:

  1. Motivation (Physics-grounded video generation remains unsolved): To further validate the motivation of our work, we have added qualitative comparisons against a strong commercial baseline, Seedance-2.5, in Appendix C (Figure 4). While strong generative models produce visually smooth frames, they still struggle with physical and thermodynamic consistency during complex phase transitions (such as melting and phase-change dynamics), confirming that physics-grounded video generation remains an open challenge.

  2. Research Scope (A model-agnostic controllable pipeline, not a new backbone): We wish to clarify a potential misconception regarding our scope: rather than training a stronger standalone video generation backbone, our focus is on proposing a controllable, model-agnostic pipeline that effectively orchestrates existing pre-trained AI models with approximate physics simulation. Hence we do not use very strong models for our benchmarks in our first manuscript. Because our pipeline decouples physical dynamics from appearance refinement, stronger video generation models can be directly plugged into the final refinement stage.

  3. Novelty (Bridging simulation-in-the-loop and AI reasoning beyond rigid bodies): To the best of our knowledge, our method is the first to bridges pure AI physics reasoning and physics-simulator-in-the-loop approaches for the rough video generation. By combining AI-driven physical reasoning with phenomenological MPM/PBD extensions and temporal Gaussian updating, PhasePlay extends controllable physics-based generation beyond simple rigid-body motion to complex, non-rigid material phase transitions.

  4. Completeness of Revision: Many of the concerns raised in the initial reviews stemmed from missing intermediate visualizations, unclear descriptions of parameter inference, and the temporarily inactive project link. In our revision, we have resolved these issues by: (a) reactivating the anonymous repository and playable comparison videos, (b) clarifying our phenomenological formulation and automated parameter selection in the Abstract, Section 3.2, and Appendix A, and (c) adding full stage-by-stage intermediate results in Appendix G.

We sincerely hope this summary and our updated manuscript help clarify the positioning and contributions of our work, and hope that all the reviewers could spend some time again to re-evaluate our submission in light of its research novelty, technical formulation, and practical usage contribution, and we welcome any additional feedback or follow-up questions before the discussion window closes. Thanks very much again for your time and efforts in evaluating this work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.