acceptodds
Withdrawn from ICLR 2027

PhysiPlay: Learning Multi-Agent Physics-Based Game Simulation in World Space

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent work showed that multi-object future dynamics can be generated as a single denoising process directly over 3D world-space mesh vertex trajectories, without a latent space or hand-imposed physical priors such as rigidity. Such models are bidirectional: all conditioning is fixed before sampling, so they cannot respond to actions that arrive during a rollout. We present PhysiPlay, which keeps this world-space formulation and makes it interactive. PhysiPlay adapts a bidirectional teacher into a block-causal generator that produces four-frame chunks from a fixed-size window of recent history and a persistent geometry reference, and binds keyboard actions to individual scene objects through global per-frame modulation and object-local FiLM. Actions therefore directly drive the controlled objects and propagate through contact to other objects. We demonstrate PhysiPlay on two-player 3D pong with balls spanning six materials and multiple geometries, conditioned on physical-property vectors. Because the output is explicit geometry, physical quantities and behaviors such as rigidity, momentum, and deformation can be directly inspected or measured; we quantitatively evaluate trajectory accuracy, rigidity, and momentum over rollouts three times longer than the bidirectional teacher’s training window and ablate the causal adaptation. By operating in view-invariant world coordinates with persistent object representations, PhysiPlay takes a step toward learned interactive simulators for multi-agent control.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
learned simulation, action-conditioned world models, causal diffusion, physical AI

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_cFBtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhysiPlay, which is an action-conditioned generative simulator that directly predicts vertex trajectories of meshes in 3D space. The model is initialized from PhysiFormer, with additional action conditioning injected through global AdaLN and local FiLM to control actuated objects. PhysiPlay uses a chunk causal autoregressive generation strategy to support streaming generation and interactively take in commanded actions of the actuated objects during inference. To improve rollout stability, training incorporates ground-truth, noised ground-truth, and self-generated histories, along with geometric perturbations. Experiments compare PhysiPlay with video generators and 3D trajectory diffusion models and show improvements on selected metrics.

Critical strengths
  1. Compared to video models, the exploicit 3D mesh representation provides a persistent state of complete objects, and allows the model to maintain geometric consistency across viewpoints.
  2. PhysiPlay makes modifications on top of the base model to support flexible object conditioning and interactive streaming generation.
Critical weaknesses
  1. The modifications on top of PhysiFormer have limited justification. It is unclear whether the causal adaptation provides a clear improvement over straightforward chunk-by-chunk generation using the original base model with action conditioning. It would be helpful to include a comparison to justify this adaptation.

  2. PhysiPlay compares its predictions with video generation models and 3D trajectory prediction models. However, PhysiPlay receives privileged information such as complete object states and structured actions, while video models receive only an initial frame and textual descriptions. The synthetic Pong renderings may also be out of distribution for the video models. Is there a more closely matched evaluation setting that would make this comparison more meaningful?

  3. The comparisons with 3D trajectory prediction models seem limited and exclude active paddle interactions. Furthermore, PhysiPlay underperforms PhysCtrl on two out of the three reported metrics.

  4. The ablations do not seem to ablate the different forcing mechanisms used in the paper. Could the authors include comparisons showing the effectiveness of the forcing mechanisms?

  5. Despite introducing techniques to improve long horizon rollout stability, the model still appears to have substantial trajectory divergence according to Table 3.

  6. The data is simulated at 120 Hz, so a four frame chunk is ~0.033 seconds. A 243 frame rollout represents ~2.03 seconds of simulated time (243/120), but takes 34.5 seconds to generate, which is roughly 17 times slower than real time. This seems to limit the practical use of the model for interactive simulation.

Questions for authors

Besides the weaknesses, could the authors report the number of unique training trajectories or total simulated frames used to train the model?

Reviewer_sACNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes PhysiPlay. It extends PhysiFormer, a bidirectional diffusion model over world-space mesh vertex trajectories, for interactive, action-conditioned simulation.

The method has three stages. (1) Pre-train from PhysiFormer, (2) Add a material vector for the ball and keyboard action conditioning. Action is injected using two pathways: a global adaLN with pooled action, and a local FiLM path for each paddle. (3) Distil it into a block-causal generator, which generates conditioned on 8 history frames plus the first 4 clean frames for geometry reference.

The causal training part mixes teacher forcing, diffusion forcing, self forcing, and a proposed “physics forcing”. Experiments are conducted on two-player 3D pong with 1 ball and 2 paddles. Baseline comparisons are conducted for video diffusion models in pixel space, TIE and PhysCtrl in state space, and the causal design is ablated on rollouts.

Critical strengths

The idea of extending bidirectional PhysiFormer into an interactive, causal variant is clear and well-motivated. As with PhysiFormer, the output is an explicit 3D mesh, so momentum and rigidity can be measured directly and rendered consistently. The design of adding additional action tokens and replacing additive X0/V0 with fixed reference cross-attention is reasonable and supported in Table 3. An ablation report on 3x windows besides the training horizon. The overall writing is clear and easy to understand, with an honest limitation section. An agentic trial with GPT6-Astra is provided.

Critical weaknesses
  • The task setting is a little too toyish and potentially overclaimed. Currently, it is only 1 ball + 2 paddles. Moreover, it seems gravity is not ON, while PhysiFormer enabled gravity. And “Multi-agent” in the title means two scripted paddles, which is potentially overclaimed.

  • Physics following is not shown. None of the supplementary videos shows rigid-body interaction, such as collisions between the ball and the paddle. The interaction is mostly the ball bouncing off the wall (-1~1 unit cube). There is no contact-event metric (e.g. post-impact velocity vs GT), but “actions propagate through contact” is the main claim. All quantitative results (Tables 1–3) are metal only. The 6-material claim has only qualitative videos.

  • Action following is not measured. There is no metric that paddles follow the keys, and no counterfactual (same state, different action → different ball outcome).

  • FiLM design is not justified. The paper explains why the action should be local, but not why FiLM is the right modulation. There is no comparison with alternatives and no ablation of the global vs local paths. Fig. 1 and Sec. 3.2 say “mean pool”, but Appendix C say actions are applied per frame and not pooled. This is inconsistent.

  • Forcing recipe is mixed without evidence. TF/DF/SF/physics forcing are combined with a narrative (“clean → corrupted → self-generated”) but no ablation. Physics forcing is the only new part, but it gets two sentences and no ablation. The schedule (30,030 / 980 / 4,200 / 1,120 updates) looks ad hoc.

  • Baselines are not fair or are not enough. Video models are zero-shot, while PhysiPlay is trained on this data and rendered with the GT renderer. In Table 2, PhysiPlay only wins on rigidity. Most importantly, there is no comparison with PhysiFormer itself. And RigidFormer and WorldParticle are physics transformers and are only cited, not compared.

  • Generalization is inherited, not tested. Unseen geometry, more objects, and concave shapes are never tested, even though it kept the PhysiFormer backbone.

Questions for authors
  1. Task setting and claims. Is gravity enabled in the setup? PhysiFormer was trained with gravity, so if it is disabled here, how does this affect the dynamics prior inherited in Stage 1? Also, can the authors justify the term “multi-agent” in the title when the setting has only two scripted paddles?

  2. Physics following and contact. Can the authors provide a quantitative contact-event metric (e.g., post-impact velocity or bounce direction vs. GT) specifically for ball–paddle collisions, not only ball–wall bounces? Since all quantitative results are metal-only, can the authors report per-material metrics for the other five materials to support the 6-material claim?

  3. Action following. How do the authors verify that the paddles actually follow the keyboard input? Can they report a paddle-tracking metric under the given actions, and a counterfactual experiment (same initial state, different action sequence) showing that the ball’s trajectory changes accordingly after contact?

  4. FiLM design. Why is FiLM chosen as the object-level modulation, rather than alternatives such as additive action embeddings or cross-attention to action tokens? Moreover, please also clarify the inconsistency between Fig. 1 / Sec. 3.2 (“mean pool”) and Appendix C (“applied per frame rather than pooled”).

  5. Forcing recipe. What is the empirical justification for combining TF, DF, SF, and physics forcing in a single mixture? Can the authors provide ablations for each component, especially physics forcing? Also, explain how the schedule lengths (30,030 / 980 / 4,200 / 1,120 updates) and the mixture ratios were chosen.

  6. Baselines. Why is PhysiFormer itself not compared, given that its paper already reports chunked long-horizon rollout? Can the authors compare against PhysiFormer’s chunked rollout, or the Stage 2 bidirectional teacher with a sliding window, to show that the Stage 3 causal adaptation is necessary?

  7. Generalization. Since Stage 3 substantially changes the conditioning (reference cross-attention, local temporal positions, per-paddle FiLM), can the authors test whether PhysiPlay still generalizes to unseen ball geometries, concave shapes, and more objects?

Reviewer_kTNdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Though denoising diffusion models are neural physics simulators in vertex trajectory space, past work fixes conditioning before sampling and diffuses over a fixed trajectory length, which prevents interactive generation. PhysiPlay enables online action-conditioning for more than one agent via block-causal generation, taking a step towards online, interactive neural physics simulation in vertex trajectory space. Additionally, PhysiPlay adds richer material conditioning that one can apply per object. The authors produce their own dataset of two player pong simulation, extend the architecture of the PhysiFormer backbone, and apply recent bidirectional-to-causal methods to achieve their goal. For 3D pong, the proposed model achieves better perceptual metrics on rendered outputs than video-based world models and attains better or comparable physical plausibility metrics against similar models. The second result is impressive given that PhysiPlay is more feature-rich than its baselines. The causal model reportedly generates qualitatively physically plausible outputs up to three times the original fixed 81-frame window of the bidirectional base though quantitatively Table 3 shows errors grow super-linearly. The work presents a step towards online, multi-agent 'world models' with fine-grained physics controls that complement video-based models and presents an interesting collection of methods.

Critical strengths

The paper builds a comprehensive solution spanning data collection, architecture, bidirectional to causal adaptation, and materials conditioning. The project source code will be a valuable asset for the community because of its wide coverage along the generative model stack. In particular, the collection of diffusion forcing, self forcing, and physics-forcing together could be useful for adapting physics-related diffusion models for causal generation.

If the paper is the first to produce a model with these capabilities (specifically, multi-agent online interaction in vertex space), then the work is pioneering and interesting. My assessment partially hinges on this novelty, see the first question in the questions section.

Critical weaknesses

The comparison against video diffusion baselines does not support the argument that PhysiPlay is a better simulator. The authors acknowledge that the perceptual metrics say nothing about physical plausibility. I understand that video based 'world models' are a reasonable baseline for this work, but the metrics' use here is a bit odd considering PhysiPlay does not generate video or image. Perhaps a better comparison would extract the implicit geometry in the generated videos with 3D point tracking and estimate the metrics in Table 2. Here is a paper that may help towards this end: https://spatialtracker.github.io/ (I am not affiliated with this work). Broadly, I suggest the authors design an evaluation protocol to compare against video world models for physics.

The authors examine only a single game. Demonstrating their method on games with more complex multi-player interaction or richer geometry would strengthen their claims considerably. Additionally, the supplemental material suggests that few collision events occur in the training data. Specifically, I interpret the file vdm_baseline/sample_000/GT.mp4 to depict a ground truth sample. In that file, the paddles move slowly and the ball never collides with other objects. If this sample is representative of the training distribution, it's unclear that the proposed model truly learns any meaningful interactions.

Questions for authors

Is this the first work to enable multi-agent causal simulation in 'world' state space? If so, the writing does not emphasize this. If not, how is this method distinct from its contemporaries?

What happens when we supply actions that push the paddles repeatedly in one direction? Do they move outside the region seen in training? More broadly, how does the model behave when given actions that may be out of distribution?

Reviewer_UfvWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

PhysiPlay extends PhysiFormer’s world-space vertex trajectory model to interactive, action-conditioned simulation. It combines block-causal generation with a fixed window of recent history, a persistent geometry reference, and per-object action conditioning. The method is evaluated on two-player 3D Pong with varying geometries and six ball materials.

Critical strengths

A shared explicit 3D representation. The generated geometry supports consistent rendering across viewpoints and direct measurement of physical quantities. This makes the predicted dynamics easier to inspect and evaluate.

Interactive generation with separate player inputs. The causal adaptation and per-paddle action conditioning allow both players to affect a shared scene during a rollout, extending the original fixed-horizon trajectory model toward interactive simulation.

Critical weaknesses

Limited technical novelty. While the setting is interesting, the methodological contribution beyond combining PhysiFormer with existing causal generation and action-conditioning techniques remains unclear. The paper also offers limited distinctive observations that could benefit other AI research beyond this particular setting.

Overly simple multi-agent scenarios. Although ball shapes and materials vary, the experiments are restricted to Pong with two paddles and one ball. This provides limited evidence of effectiveness across the more varied interactions and gameplay situations encountered in actual multi-agent games.

Unclear benefits of explicit 3D vertex trajectories. Explicit geometry supports physical measurements and consistent multi-view rendering, but it remains unclear why generating every vertex trajectory is necessary for game simulation. The paper does not sufficiently establish practical advantages over more compact state or latent representations.

Limited comparisons and inconsistent performance gains. The paper’s positioning relative to multi-agent, multi-view world models such as MultiWorld[1] remains unclear beyond its explicit 3D output. The video baselines are not fine-tuned for the task, and the state-space comparisons exclude multi-agent action conditioning. Moreover, Table 2 shows improved rigidity but higher position and momentum-change errors than both baselines. While internal ablations are provided, the comparative evidence does not sufficiently establish the method’s advantages over existing approaches.

[1] https://multi-world.github.io/ [2] https://research.nvidia.com/labs/sil/projects/gamma-world/

Questions for authors
  1. What evidence supports the method’s effectiveness in more varied and realistic multi-agent gameplay scenarios beyond two-player Pong?
  2. Beyond physical measurements and multi-view rendering, what practical benefits does explicit vertex trajectory generation offer over compact state or latent representations for game simulation? What evidence supports these benefits?
  3. Is there comparative evidence under matched training data and action conditioning that establishes the method’s advantage? In particular, how should the improved rigidity but higher position and momentum-change errors in Table 2 be interpreted when assessing its overall benefits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.