acceptodds
Under review as a conference paper at ICLR 2027

PCBench: Physical Commonsense Benchmark for Agentic Video Evaluation and Generation

Abstract

Video generators now render photorealistic clips, but do the objects in them move as physics requires? Released balls hover, pendulums stall and requested collisions never happen, yet common visual-quality metrics and vision-language judges do not directly measure conformity to explicit motion laws. We introduce PCBench, a physical-commonsense benchmark that pairs 3,093 real videos in ten phenomenon categories with 4,049 generated videos from six generator families and 945 simulations with perturbations of known severity. Four-axis annotations of generated clips separate prompt following from physical plausibility. Across all six generators, the requested phenomenon is realized less reliably than general physical plausibility. For mechanics, where motion laws are explicit and can be checked quantitatively, we propose agentic evaluation: a tool-using workflow grounds and tracks the target object, fits an executable motion program and reports an interpretable Motion-Law Conformity (MLC) score. Without fitting to human labels, the residual increases with severity across all 18 controlled perturbation pairs, and MLC has the highest point correlation with mean human ratings on 216 generated mechanics clips ( versus 0.21 for VideoScore2), although the paired interval for this difference includes zero. Because the program that scores a motion can also produce it, the same programs close the loop to generation: they render temporal controls for a video diffusion model, re-measure the output and repair measured failures. Program-controlled generation produces the requested motion where resampling rarely does, and three human raters on average favour it in all 16 matched pairs. Code is available through an anonymous link: https://anonymous.4open.science/r/pcbench-iclr27-review-code-2F15/

Primary area:
datasets and benchmarks
Keywords:
generated video, physical consistency, video evaluation, event-aware system identification, uncertainty, abstention, benchmark leakage

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WkExOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To test whether video generators actually follow physical laws rather than just rendering plausible-looking frames, the authors build PCBench, combining 3,093 real videos, 4,049 generated clips from six generator families, and 945 Unity simulations with controlled physical perturbations. A four-axis human annotation protocol scores general physical plausibility (SCS-ge) before revealing the prompt to separate it from prompt-specific phenomenon realization (SCS-sp), showing that generators fail at the requested physical event far more often than at general coherence. For seven mechanics families, the paper fits analytical motion equations to Grounded-SAM2 tracks to compute a Motion-Law Conformity (MLC) score, and reuses those motion programs in Blender to drive and re-verify Wan2.1-VACE-1.3B generation.

Critical strengths
  1. Having annotators score general physical plausibility (SCS-ge) before seeing the prompt is a clean, effective way to separate "looks like plausible physics" from "actually produced the requested physical phenomenon" (SCS-sp). Paired with the 945 controlled Unity videos across 18 motion-perturbation pairs, the benchmark provides both a clear empirical finding across six generators and a well-calibrated synthetic testbed for trajectory evaluators.

  2. The paper is unusually candid about negative results and uncertainty, and the temporal permutation test in Section 5 is a great diagnostic. Scrambling the middle control frames preserves visual appearance and endpoints while breaking dynamics; showing that trajectory forecasting catches this while VideoScore2 fails (and correlates inversely with the pass/fail gates) clearly demonstrates why VLM judges alone are insufficient.

Critical weaknesses
  1. It is unclear whether MLC reliably ranks generated clips within a given motion type, or if the pooled correlation (rho = 0.382 on the 216 generated mechanics clips) is mostly a between-family scaling artifact. In Appendix D (Tables 15, 17, and 19), within-family correlation is only 0.125 (CI includes zero), six of seven families have near-zero or negative correlations, and replacing min-max normalization with percentile ranks drops Test95 generated correlation from 0.381 to 0.048.

  2. The "agentic" framing feels overstated relative to the actual experiments. In Section 4, the motion family comes from the ground-truth category, equations are fixed, and humans manually chose the target track in 227 of 421 multi-object clips (54%). In Section 5, Blender scenes and programs are manually supplied (real-video reconstruction in Appendix E.13 failed), and on the two repair cases the agent action simply turns on full 49-frame control.

Questions for authors
  1. Why is within-family correlation near zero for most mechanics families, and does applying the coverage and motion gates from Section 5 to the benchmark clips improve within-family ranking or robustness to percentile normalization? If gating resolves this, I would be glad to take that into account in my final rating.

  2. How much does the MLC correlation on the 216 generated clips change if you replace manual track selection on the multi-object clips with an automated heuristic (such as GroundingDINO confidence or a VLM prompt)?

Reviewer_ciUPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PCBench, a benchmark for evaluating physical commonsense in video generation. The dataset contains 3093 real videos, 4049 generated videos from six video generation model families, and 945 controlled simulation videos, covering ten categories of physical phenomena. The authors propose a four-axis human evaluation protocol that distinguishes prompt adherence, specific physical plausibility, general physical plausibility, and object consistency. Beyond the dataset, the paper introduces Motion-Law Conformity (MLC), an interpretable evaluation metric that measures deviations from predefined physical motion models using object tracking and trajectory fitting. The authors further demonstrate how these executable motion programs can guide video generation and support verification and correction of generated motion.

Critical strengths
  1. PCBench provides a diverse collection of real and generated videos, along with controlled simulations, to evaluate physical consistency in video generation. I particularly appreciate the distinction between general physical plausibility and whether the specific physical phenomenon requested in the prompt is correctly reproduced.
  2. Instead of relying solely on VLM-based judgments, the proposed MLC directly measures deviations from physical motion models. The controlled perturbation experiments also show that the metric can capture different degrees of physical inconsistency, making the evaluation more interpretable.
  3. The experiments cover several video generation models and include comparisons with existing evaluation metrics and human judgments. I also find the generation experiments useful, as they show how the proposed evaluation framework can be applied to guide and improve physically consistent video generation.
Critical weaknesses
  1. Although MLC achieves the highest overall correlation with human ratings (ρ = 0.382), its within-family correlation is much lower (ρ = 0.125). This suggests that part of the overall performance may come from differences across motion categories, while its ability to distinguish the physical correctness of videos within the same category remains limited. More discussion or analysis of this gap would help clarify the effectiveness of MLC as an evaluation metric.
  2. Most generated videos have only one human annotation, while multiple ratings are mainly available for the mechanics subset. It would be helpful to provide more analysis of annotation consistency across different physical categories, especially given the subjective nature of physical plausibility judgments.
  3. Although PCBench covers ten categories of physical phenomena, MLC currently focuses on seven relatively simple motion types. It would be interesting to discuss how the evaluation could be extended to more complex physical phenomena.
  4. The controlled Unity simulations are useful for validating MLC, but it is not entirely clear how well the designed perturbations reflect the physical errors commonly found in generated videos. A more detailed comparison between synthetic perturbations and actual generation failures would help strengthen the evaluation.
Questions for authors

Please see the weaknesses above for more details. I would appreciate the authors' clarification on the following questions:

  1. What explains the large gap between the overall and within-family correlations of MLC? How could its fine-grained evaluation ability be improved?
  2. How consistent are the human annotations across different physical categories, particularly those outside the mechanics subset?
  3. How could MLC be extended to evaluate more complex physical phenomena, such as fluid dynamics and deformable objects?
  4. How representative are the controlled Unity perturbations of the physical errors observed in generated videos? Could the authors provide more analysis of this connection?
  5. How does MLC perform in a fully automatic setting, without manually selecting target trajectories?
  6. How sensitive is MLC to camera motion and viewpoint changes, given its reliance on 2D object trajectories?
Reviewer_tR5qOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PCBench, a physical-commonsense benchmark containing real, generated, and controlled videos. Its four-axis annotations distinguish whether the requested physical phenomenon is realized (SCS-sp) from general physical plausibility (SCS-ge). For mechanics, the authors propose Motion-Law Conformity (MLC), which measures how well tracked object trajectories fit predefined motion programs. The same executable programs are then used to produce temporal controls for video generation, followed by re-measurement and diagnosis-directed regeneration. The experiments include generator benchmarking, controlled perturbations, comparisons with human ratings, and matched generation studies.

Critical strengths
  1. The paper focuses on an good problem: realistic-looking generated videos may fail to exhibit the physical phenomenon requested by the prompt. PCBench is a potentially useful resource for studying these failures.
Critical weaknesses
  1. The benchmark's central findings rely heavily on a score that overlaps with prompt following. Although PCBench distinguishes requested-phenomenon realization (SCS-sp) from general physical plausibility (SCS-ge), SCS-sp penalizes videos in which the requested event is absent, even when the observed motion is physically plausible. Thus, a low SCS-sp does not necessarily indicate a violation of physical laws. This concern is reinforced by the strong correlation between PCS and SCS-sp. Without explicitly separating failures to realize the requested phenomenon from physical violations within successfully realized phenomena, the main findings may primarily reflect limitations in prompt/event following rather than physical-law compliance.
  2. The cross-generator benchmark comparisons lack consistent evaluation conditions. In Table 1, different generators are evaluated on substantially different numbers of videos (ranging from 69 to 1,467), with different prompt sets and even different category coverage. Such comparisons cannot reliably isolate differences in model capability from differences in prompt difficulty or category composition. This raises concerns about the validity of the main leaderboard and the conclusions drawn from cross-generator comparisons.
Questions for authors
  1. What evidence shows that PCBench measures physical-law compliance beyond prompt/event following? SCS-sp strongly overlaps with PCS, and failure to realize a requested phenomenon does not necessarily indicate a violation of physical laws.
  2. How can Table 1 support reliable cross-generator comparisons when the models are evaluated on different prompts, sample sizes, and category distributions? The reported differences may reflect variations in test difficulty rather than model capability, and the partial matched-prompt analysis does not fully resolve this concern.
  3. What is the fundamental justification for treating the fitting residual of an estimated 2D trajectory as a measure of physical-law compliance? A good mathematical fit does not necessarily imply physically correct motion, particularly when the underlying 3D motion, camera geometry, and physical parameters are unknown.
  4. What is the distinct contribution of the proposed generation framework beyond prior simulation-guided methods such as GPT4Motion(CVPR 2024 workshop)? The improvements appear primarily attributable to dense Blender-rendered temporal controls, while the experiments do not establish the additional value of the proposed evaluation and agentic feedback mechanisms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.