PhysGraph: Point Graphs for Physical Scene Understanding

Real-world videos mix rigid bodies, cloth, and liquids – yet existing dynamic-reconstruction methods commit to a single motion model. We present PhysGraph, a temporal interpolation and extrapolation pipeline that uses a one-shot, foundation-model-predicted material phase as a prior for selecting one of three classical motion models on a shared per-object kNN point graph: Procrustes-style SE(3) for solids, thin-plate-spline residuals for deformables, and optimal transport for liquids. The graph is reused across branches as a computational primitive – for neighborhood-aware correspondence filtering (rigid, elastic), as the elastic-kernel subgraph (TPS), and as the rewireable bipartite source-to-target structure that OT solves over (liquid). Composed with bidirectional Gaussian splatting, the resulting motions synthesize novel frames up to $26\times$ faster than per-scene-optimized baselines. On six synthetic scenes PhysGraph reduces depth error by $2.5\times$ over linear 3D interpolation; on 90 DAVIS sequences it has the highest dynamic-region recall at every stride with zero failures (\vs up to 69% for per-scene optimizers), and it leads dynamic-region recall in extrapolation as well. PhysGraph targets large temporal gaps, where per-scene optimizers become ill-conditioned and fail. The source code will be made public.