acceptodds
Under review as a conference paper at ICLR 2027

PhyTraj: Motion as a Query for Visual Physics Auditing

Abstract

A generated video can look convincing while an object's motion makes the event physically implausible. Detecting such failures calls for relating motion to its visual context. We introduce PhyTraj, which uses estimated object trajectories as queries into frozen video representations for visual physics auditing. Metadata-derived object names guide tracking; a trajectory encoder turns each motion history into a query that retrieves context from VideoMAE tokens. The resulting motion–video representations are contextualized across objects and aggregated into a clip-level violation score. We evaluate this approach on 2,334 human-labeled clips with available object tracks from one video generator, spanning six physical-rule families. Labels describe the behavior visible in the rendered clips, and object compositions are held out across splits. In an exploratory representation study on 353 test clips, the full trajectory–video model achieves 0.758 AUROC, compared with 0.683 for the video-only control. A complementary study on frozen fusion features finds that simple pooling remains competitive with structured aggregation. These findings support trajectory-conditioned video representations for detecting visible physical violations, with motion serving as an object-centered interface to visual context. The artifact is available at https://anonymous.4open.science/r/PhyTraj-B046/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual physics auditing, video understanding, object trajectories, physical consistency, generated video

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xGsZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PhyTraj for detecting physical violations in generated videos. The method uses estimated object trajectories to query visual features from a frozen VideoMAE encoder, followed by object-level fusion and classification. The authors evaluate the method on 2,334 human-labeled videos covering six physical-rule categories. In exploratory experiments, the full model achieves 0.758 AUROC versus 0.683 for the video-only baseline. The problem is interesting, and the results are promising. However, I have several concerns about the novelty of the method and whether the experiments adequately support the proposed motion-query mechanism.

Critical strengths
  1. Using object trajectories to guide visual feature extraction is a reasonable approach to physical violation detection, particularly for contact and collision-related events.

  2. The paper includes useful ablations and evaluates on held-out object compositions. I also appreciate the group-level bootstrap analysis and the discussion of experimental limitations.

Critical weaknesses
  1. The method combines fairly standard components, including object tracking, temporal Transformers, cross-attention, and feature aggregation. Related work such as VAMP and PDI-Bench has already explored motion-based physics evaluation. The main distinction appears to be using trajectories as attention queries, but the paper lacks comparisons with simpler approaches such as feature concatenation, late fusion, or trajectory-only classification. It is therefore difficult to assess the benefit of the proposed design.

  2. In Table 3, the full 3D model achieves 0.758 AUROC, while shuffled trajectories and 2D trajectories both reach around 0.753. These differences are quite small. Although the shuffling experiment preserves local motion features, the results raise questions about how much the model actually benefits from temporal ordering and 3D motion information. More convincing ablations would strengthen the main claim.

  3. All videos come from Seedance 2.0 under fixed-camera settings. Holding out object compositions is useful, but generator-specific artifacts and similar event templates may still appear across splits. The method also relies on metadata-provided object names and excludes videos without valid tracks. It would be helpful to evaluate on another video generator and examine how performance changes with tracking quality.

  4. The representation ablations were added after examining results on the same test set. In the separate frozen-feature study, the AUROC improvement over VideoMAE-only is 0.045, with a 95% confidence interval of [-0.0006, 0.0865]. Also, comparisons with VideoPhy-2 and other evaluators are not entirely fair, since PhyTraj is trained on the target dataset while those methods are evaluated zero-shot. Stronger matched baselines or an independent test set would make the results more convincing.

Questions for authors
  1. How does trajectory-query attention compare with simpler fusion methods under similar model capacity? Is there evidence that the proposed attention mechanism provides benefits beyond adding trajectory features?

  2. How well does PhyTraj generalize across video generators and different levels of tracking quality? Could the authors report per-category results with uncertainty estimates to clarify where gains arise?

Reviewer_p5KrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a method called PhyTraj (Motion as a query) which focus on detecting physical artifacts in AI generated videos. Core idea of this method is retrieving visual cue from pretrained video representation using the object trajectory query. To validate this method, authors construct a dataset generated by Seedance 2.0, covering six physical rule categories. The annotations are based on observed physical behavior. The experimental results show that trajectory-video fusion improves performance over a video-only baseline and zero-shot physics baseliens.

Critical strengths

This paper shows a potential of motion query for detecting physical artifacts from AI-generated videos. But, it needs to be validated more carefully.

Critical weaknesses
  1. The proposed method does not contain any component for explicit physical modeling. The motion query is implicitly expected to retrieve physical evidence from a general video representation. This might work, but there are no experiments that explicitly validate this motion query. Furthermore, the ablation study shows that there is only a small gap between the authors' proposed 3D trajectory and simple object-centric inputs. The main experiments compare PhyTraj with VideoMAE and zero-shot baselines. Thus, these experiments do not establish that PhyTraj is better than other physics baselines or object-centric fusion methods.

  2. The reliability of the annotations and dataset is not sufficiently validated. Although two annotators labeled the videos, inter-annotator agreement is not reported. This dataset excluded the videos without usable trajectories are excluded, which may introduce selection bias. Furthermore, the evaluation is limited to a single video generator, fixed-camera settings, and metadata-provided object queries. Therefore, it remains unclear whether PhyTraj can generalize to other generators and more realistic settings.

Questions for authors

Could you broad the scope of comparison with other baselines? Such as trained physical baselines and object-centric baselines.

Reviewer_JDxtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PHYTRAJ scores whether the behavior visible in a generated clip violates a physical annotation rule. Metadata object names guide tracking. A trained trajectory encoder maps positions, velocities, accelerations, and a validity bit into object tokens. Each token cross-attends to frozen VideoMAE features, a relation encoder mixes the objects, and a head outputs one clip-level probability. The classifier does not receive the generation prompt or the rule family. There is no object-level or time-level supervision. SAM 3 and DetAny3D are fixed perception; VideoMAE stays frozen. The corpus is 2,334 clips from one generator, Seedance 2.0, under fixed-camera prompts, balanced 1,167/1,167 after filtering. Two annotators labeled 2,454 candidates and adjudicated disagreements. Only the final bit is stored, so inter-annotator agreement cannot be computed. The trajectory filter drops 96 clips, 36 plausible and 60 violating, and class balancing then drops 24 plausible clips. Splits hold out object-composition groups: train 1,401, validation 350, calibration 230, test 353 (45 groups). Families are balanced overall and not within family. Trajectory continuity is 39.0% violations (689 clips). Gravity and projectile motion is 78.3% violations (411 clips). The paper separates a pre-specified frozen-head suite from a representation suite added after those test results were seen. In the exploratory suite, full 3D fusion reaches 0.7582 AUROC against an expanded-capacity video-only control at 0.6834. The paired AUROC difference is +0.0748 with interval [0.0267, 0.1186], and the paper says exact parameter parity is not established. Shuffled 3D, which keeps local velocity and acceleration, scores 0.7526. Image-plane 2D scores 0.7525. In the pre-specified suite, fusion plus the evidence head scores 0.7622 against video-only 0.7173. The paired AUROC difference is +0.0449 with interval [-0.0006, 0.0865], which includes zero. NLL and Brier intervals for that comparison exclude zero. Eight fusion heads span only 0.7538 to 0.7622 AUROC. A linear mean/max pool scores 0.7599. Temperature scaling fit on 230 clips slightly worsens evidence NLL and Brier on the test set (0.5821 to 0.5832, and 0.1980 to 0.1983).

Critical strengths

First, the label and the split are set up more carefully than the abstract's headline. Visible behavior is the target, not prompt intent. The classifier is denied the prompt and the rule family. Composition groups do not cross splits. Uncertainty is a bootstrap over 45 test groups, not an i.i.d. clip bootstrap. The paper reports the zero-crossing AUROC interval, the shuffle result, the 2D-versus-3D AUROC tie, and the calibration regression. That reporting is a reason to trust the experimental section. Second, on the metrics where the pre-specified comparison does separate, fusion is favored, and the representation is not an artifact of one head. NLL and Brier intervals exclude zero. Evidence AURC is 0.1888 against 0.2624 for video-only. The linear mean/max head at 0.7599 is within 0.003 of the evidence head at 0.7622. The finding that simple pooling matches structured aggregation is supported by Table 4.

Critical weaknesses

First, motion-as-query is not identified. The confirmatory AUROC gain over VideoMAE is +0.0449 with an interval that includes zero. The abstract's larger gap uses a later video control whose parameter match is explicitly not established. The temporal shuffle and the image-plane tracks are statistically tied with full 3D on AUROC. That pattern fits which objects are tracked and a rough motion descriptor, not a 3D physical readout. Pair and global branches of the evidence head are also not separated from zero. This is the weakness that supports rejection. Second, the corpus cannot yet carry a physics-auditing claim. One generator, shared templates across a composition split, no agreement statistic, and evaluation only on trackable clips leave tracker failure and family or appearance shortcuts entangled with the label. Gravity is 78.3% violations and continuity is 39.0%. A model that tracks family-correlated appearance can move aggregate AUROC. The large gaps to VideoPhy-2 (0.5524), VideoScore2 (0.5422), and a single-backbone PhyGenEval-style model (0.5253) compare a system trained on 1,401 in-domain clips with zero-shot evaluators. The paper says so. Those numbers should not be read as evidence that the query recovers physical rules.

Questions for authors

Clarification: What fraction of the 2,454 candidates did the two annotators initially disagree on, and does test AUROC change if evaluation is restricted to clips they marked the same way before adjudication? Agreement is currently not estimable, and the answer could move the rating. Clarification: What do a trajectory-only head and a validity-mask-only head score on this split? If either matches fusion, the video cross-attention is not doing the work the title describes. Clarification: The pre-specified AUROC interval includes zero. Is there a second generator, or a result on the 96 clips dropped for missing tracks, on which a pre-specified AUROC interval excludes a negligible effect? I would treat a positive answer as new evidence, not as a required future-work suggestion for the current score. The first two questions are clarification requests about data and ablations that decide whether the present claim holds. The third asks for evidence the paper does not contain; I list it because the authors already mark the representation suite as unconfirmed, and a negative answer would keep the rating where it is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.