PhyTraj: Motion as a Query for Visual Physics Auditing

A generated video can look convincing while an object's motion makes the event physically implausible. Detecting such failures calls for relating motion to its visual context. We introduce PhyTraj, which uses estimated object trajectories as queries into frozen video representations for visual physics auditing. Metadata-derived object names guide tracking; a trajectory encoder turns each motion history into a query that retrieves context from VideoMAE tokens. The resulting motion–video representations are contextualized across objects and aggregated into a clip-level violation score. We evaluate this approach on 2,334 human-labeled clips with available object tracks from one video generator, spanning six physical-rule families. Labels describe the behavior visible in the rendered clips, and object compositions are held out across splits. In an exploratory representation study on 353 test clips, the full trajectory–video model achieves 0.758 AUROC, compared with 0.683 for the video-only control. A complementary study on frozen fusion features finds that simple pooling remains competitive with structured aggregation. These findings support trajectory-conditioned video representations for detecting visible physical violations, with motion serving as an object-centered interface to visual context. The artifact is available at https://anonymous.4open.science/r/PhyTraj-B046/.