PhysiPlay: Learning Multi-Agent Physics-Based Game Simulation in World Space

Recent work showed that multi-object future dynamics can be generated as a single denoising process directly over 3D world-space mesh vertex trajectories, without a latent space or hand-imposed physical priors such as rigidity. Such models are bidirectional: all conditioning is fixed before sampling, so they cannot respond to actions that arrive during a rollout. We present PhysiPlay, which keeps this world-space formulation and makes it interactive. PhysiPlay adapts a bidirectional teacher into a block-causal generator that produces four-frame chunks from a fixed-size window of recent history and a persistent geometry reference, and binds keyboard actions to individual scene objects through global per-frame modulation and object-local FiLM. Actions therefore directly drive the controlled objects and propagate through contact to other objects. We demonstrate PhysiPlay on two-player 3D pong with balls spanning six materials and multiple geometries, conditioned on physical-property vectors. Because the output is explicit geometry, physical quantities and behaviors such as rigidity, momentum, and deformation can be directly inspected or measured; we quantitatively evaluate trajectory accuracy, rigidity, and momentum over rollouts three times longer than the bidirectional teacher’s training window and ablate the causal adaptation. By operating in view-invariant world coordinates with persistent object representations, PhysiPlay takes a step toward learned interactive simulators for multi-agent control.