- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PVBench evaluates physical plausibility in generated videos using object-level measurements from specialized vision models, supplemented by multimodal-model judgments. It introduces 600 paired T2V/I2V prompt groups and 46 evaluation dimensions, including 25 physical dimensions. Experiments cover six models in both generation settings, with human evaluation used to assess agreement. The results highlight weaknesses in mechanics and conservation laws.
Critical strengths
- The measurement-based approach makes physical errors more specific and interpretable than a single plausibility judgment. Evaluating geometry, motion, and conservation separately is useful for identifying where a video model fails.
- The paired T2V/I2V design allows a useful comparison of how reference images affect physical behavior. The human study also provides evidence that the proposed scores broadly agree with human judgments.
Critical weaknesses
- The reliability of several physical metrics remains uncertain. Table A4 reports human correlations of only 0.125 for object permanence and 0.240 for motion continuity, with 15 samples per dimension. Since most dimensions lack physical ground truth, the overall correlation does not establish that individual metrics reliably distinguish generation errors from measurement errors.
- The evaluation does not clearly establish whether the intended interaction must occur before its physical correctness is scored. Although action adherence is measured separately, it is unclear how failed actions affect physical scores. This makes it difficult to determine whether low scores reflect weak physical modeling or failure to generate the requested event.
Questions for authors
- How do you establish that low physical scores reflect errors in the generated videos rather than failures of the perception pipeline? Beyond agreement with human ratings, what validates the accuracy of the recovered physical quantities?
- Are physical scores conditioned on the requested event actually occurring? For example, how do you distinguish a video that avoids a collision from one that completes it but violates momentum conservation, and can avoiding the interaction improve the score?
- R05 measures deformation using deviations from rigid-body behavior, with smaller deviations receiving higher scores. How does this metric handle scenes where deformation is physically expected, such as a sponge being compressed? Are the scoring criteria conditioned on the object's material properties and the intended interaction?
3 comments
Thank you for the careful response, which addresses many of my concerns. I maintain my score.
Dear Reviewer RUtJ,
Thank you for your positive assessment of PVBench and your thoughtful feedback. We are glad that our response has addressed many of your concerns. We hope that the additional validation in our rebuttal and the clarifications in the revised manuscript further strengthen your overall assessment of our work.
Thank you again for your time and constructive discussion.
Best regards,
Authors of PVBench
Q1: Independent validation of physical scoring
Thank you for asking about measurement reliability.
We distinguish unavailable measurements from measured physical anomalies. Of 6,291 records, 237 are skipped, 10 unobservable and 6,044 scored; identified measurement failures do not directly become low generator scores. Logs retain quantities and triggered rules, plus objects or intervals for track/collision checks, allowing comparison with the video.
R05 should preserve strain under rigid motion and recover known compression. With known 3D tracks of 27 points, rigid transforms give zero strain. Four-step uniaxial compression to 90% length gives mean strain 0.060156 versus the analytic 0.06015625; superimposed rigid motion preserves it. This verifies deformation response and rigid-motion invariance in these controls.
Known boxes test R23 independently of human ratings. Equal-mass symmetric approach/rebound gives analytic and measured residual 0. Changing one outgoing speed to 0.1 times the other gives analytic residual 0.2903226 and measured 0.2903, lowering the score from 5 to 3. Uniformly rescaling both trajectory sets by 0.8 preserves their residuals and scores, distinguishing momentum changes from measurement-unit changes.
On generated videos, R02 compares predicted and depth-derived normals; R20 retains interpolation errors and track changes; R21 retains detection and identity histories across occlusion. These observations help trace anomalies and compare them with depicted events. Known-input tests validate the calculations; quantifying perception recovery error additionally requires reference geometry or motion for the same videos.
On the same 125 MiniMax H3 pairs, correlation with human ratings is ρ = 0.840 for PVBench versus 0.142 for direct MLLM scoring; full-mark rates are 14.4% versus 74.4%. Known inputs test physical response, while matched human comparisons support diagnostic value on these outputs. Together they support specific, inspectable physical diagnosis.
Q2: Event-occurrence conditions in physical scoring
Thank you for asking how event occurrence is handled.
R23 uses changes in image-plane object-center distance to locate momentum comparisons. Separation must vary, with its minimum below 0.30 times its maximum. The minimum-distance time defines pre/post windows for the residual. All 213 valid records retain this time and residual.
Conservation errors change scores within the same comparison interval. Changing only one outgoing speed in the known-track test raises the residual from 0 to 0.2903 and lowers the score from 5 to 3. Identical approaches can thus receive different scores for different pre/post motion.
Approach/rebound evidence also enters scoring. Of 213 valid videos, 93 have interaction weights below 1: 29 final scores fall by 1, while 64 stay unchanged through rounding or the score floor. Event times, residuals and weights make deductions traceable.
Near misses can conserve momentum and receive high R23 scores if they pass proximity detection. Unavailable measurements are skipped, so the mean can change with included videos. Scores should therefore be read alongside the depicted event, measurement coverage and IF-3 action adherence where assigned, distinguishing requested collision completion from measured conservation.
Other event-based dimensions explicitly penalize omissions: all 632 records lacking the target phenomenon in R06/R07/R14/R25 receive the minimum score, as do all 104 R05 records without detected deformation. Event occurrence already enters scoring; per-dimension conditions will be specified in the appendix.
Q3: How R05 scores expected deformation
Thank you for your question about expected deformation.
R05 first checks for deformation, using the average Frobenius norm of Green–Lagrange strain from tracked points. Values ≤ 0.02 receive 1; magnitude bins apply above this threshold. Thus, zero recovered strain receives the minimum on compression tasks, as in all 104 such records.
Prompt U46V2-0149 asks two plates to compress and release a foam cylinder, a counterpart to sponge compression. All 4 outputs without detected deformation score 1; all 8 with deformation score 4 or 5. Scoring distinguishes remaining rigid from producing the requested compression.
R05 uses common strain intervals across materials. Where R06 is also assigned, seven material-family rubrics assess material response, including elastic recovery of compressed sponge and persistent plastic deformation of clay. The appendix will separately define deformation magnitude and material-behavior criteria.
Known trajectories test strain calculation; the foam task tests score direction on generated outputs. R05 and R06 correlate with human ratings at ρ = 0.622 and 0.867 on 15 examples each, further supporting both diagnostics.