PhysDuo: Learning Pairwise Physics for Compositional Multi-Object Simulation

Generalizing learned rigid-body dynamics across physical timesteps and object counts remains challenging. We introduce PhysDuo, a compositional point-cloud simulator that decomposes rigid-body dynamics into two complementary components: per-object self dynamics and pairwise contact response. A timestep-conditioned self-dynamics model autoregressively predicts each object's motion under external forces and floor contact, using timestep conditioning and temporally scaled motion residuals. With this model frozen, a pairwise contact model learns contact-induced corrections to the predicted motion of two objects. A linear momentum constraint couples their translational corrections, while a symmetric contact gate suppresses updates for non-interacting pairs. Both models reconstruct rigid motion from updated anchor positions and maintain angular velocity as an explicit state. During multi-object inference, the shared contact model is applied sequentially to object pairs using their latest states, allowing subsequent interactions to incorporate earlier contact responses. This composition enables the same learned self and pairwise dynamics to be reused across scenes with varying object counts without changing the model parameters. We evaluate PhysDuo across physical timesteps, rollout horizons, and object counts, and further demonstrate zero-shot composition in scenes containing substantially more objects than seen during training. Ablations examine timestep conditioning, contact modeling, and pairwise composition.