acceptodds
Under review as a conference paper at ICLR 2027

PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements

Abstract

Recent video generation models produce visually compelling outputs, yet their physical plausibility remains poorly measured. Physical plausibility asks whether generated objects obey geometry, kinematics, dynamics, and conservation laws. Existing benchmarks rely on coarse visual-language model judgments or target embodied world models rather than video generation itself. We introduce **PVBench**, **a physical video benchmark that quantitatively measures the plausibility of generated videos across 46 dimensions organized into seven domains**, spanning: *i)* instruction and semantic fidelity, *ii)* behavioral and causal consistency, *iii)* perceptual and temporal quality, *iv)* geometric and spatial consistency, *v)* material properties and field states, *vi)* kinematic and spatiotemporal persistence, *vii)* and mechanical plausibility with conservation laws. **PVBench** comprises 600 carefully crafted prompts, of which 30% involve physical embodied agents such as robot arms, humanoids, quadrupeds, and multi-agent systems. For challenging physically tractable dimensions, it pairs each with specialized vision models that derive verifiable, physically grounded measurements, such as object-level 6D pose, metric scale, surface normals, kinematics, and numerical conservation-law residuals. It thus avoids relying solely on subjective scoring, while supporting both text-to-video and image-to-video generation. Finally, we validate the agreement between automatic metrics and human judgments through human evaluation. Meanwhile, we evaluate mainstream open-source and closed-source models and present several insights. By releasing the benchmark, prompts, and calibration protocol, we aim to foster physically grounded video generation.

Primary area:
datasets and benchmarks
Keywords:
video generation, physical plausibility, benchmark, quantitative evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RUtJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PVBench evaluates physical plausibility in generated videos using object-level measurements from specialized vision models, supplemented by multimodal-model judgments. It introduces 600 paired T2V/I2V prompt groups and 46 evaluation dimensions, including 25 physical dimensions. Experiments cover six models in both generation settings, with human evaluation used to assess agreement. The results highlight weaknesses in mechanics and conservation laws.

Critical strengths
  1. The measurement-based approach makes physical errors more specific and interpretable than a single plausibility judgment. Evaluating geometry, motion, and conservation separately is useful for identifying where a video model fails.
  2. The paired T2V/I2V design allows a useful comparison of how reference images affect physical behavior. The human study also provides evidence that the proposed scores broadly agree with human judgments.
Critical weaknesses
  1. The reliability of several physical metrics remains uncertain. Table A4 reports human correlations of only 0.125 for object permanence and 0.240 for motion continuity, with 15 samples per dimension. Since most dimensions lack physical ground truth, the overall correlation does not establish that individual metrics reliably distinguish generation errors from measurement errors.
  2. The evaluation does not clearly establish whether the intended interaction must occur before its physical correctness is scored. Although action adherence is measured separately, it is unclear how failed actions affect physical scores. This makes it difficult to determine whether low scores reflect weak physical modeling or failure to generate the requested event.
Questions for authors
  1. How do you establish that low physical scores reflect errors in the generated videos rather than failures of the perception pipeline? Beyond agreement with human ratings, what validates the accuracy of the recovered physical quantities?
  2. Are physical scores conditioned on the requested event actually occurring? For example, how do you distinguish a video that avoids a collision from one that completes it but violates momentum conservation, and can avoiding the interaction improve the score?
  3. R05 measures deformation using deviations from rigid-body behavior, with smaller deviations receiving higher scores. How does this metric handle scenes where deformation is physically expected, such as a sponge being compressed? Are the scoring criteria conditioned on the object's material properties and the intended interaction?
3 comments

Q1: Independent validation of physical scoring

Thank you for asking about measurement reliability.

  1. We distinguish unavailable measurements from measured physical anomalies. Of 6,291 records, 237 are skipped, 10 unobservable and 6,044 scored; identified measurement failures do not directly become low generator scores. Logs retain quantities and triggered rules, plus objects or intervals for track/collision checks, allowing comparison with the video.

  2. R05 should preserve strain under rigid motion and recover known compression. With known 3D tracks of 27 points, rigid transforms give zero strain. Four-step uniaxial compression to 90% length gives mean strain 0.060156 versus the analytic 0.06015625; superimposed rigid motion preserves it. This verifies deformation response and rigid-motion invariance in these controls.

  3. Known boxes test R23 independently of human ratings. Equal-mass symmetric approach/rebound gives analytic and measured residual 0. Changing one outgoing speed to 0.1 times the other gives analytic residual 0.2903226 and measured 0.2903, lowering the score from 5 to 3. Uniformly rescaling both trajectory sets by 0.8 preserves their residuals and scores, distinguishing momentum changes from measurement-unit changes.

  4. On generated videos, R02 compares predicted and depth-derived normals; R20 retains interpolation errors and track changes; R21 retains detection and identity histories across occlusion. These observations help trace anomalies and compare them with depicted events. Known-input tests validate the calculations; quantifying perception recovery error additionally requires reference geometry or motion for the same videos.

  5. On the same 125 MiniMax H3 pairs, correlation with human ratings is ρ = 0.840 for PVBench versus 0.142 for direct MLLM scoring; full-mark rates are 14.4% versus 74.4%. Known inputs test physical response, while matched human comparisons support diagnostic value on these outputs. Together they support specific, inspectable physical diagnosis.


Q2: Event-occurrence conditions in physical scoring

Thank you for asking how event occurrence is handled.

  1. R23 uses changes in image-plane object-center distance to locate momentum comparisons. Separation must vary, with its minimum below 0.30 times its maximum. The minimum-distance time defines pre/post windows for the residual. All 213 valid records retain this time and residual.

  2. Conservation errors change scores within the same comparison interval. Changing only one outgoing speed in the known-track test raises the residual from 0 to 0.2903 and lowers the score from 5 to 3. Identical approaches can thus receive different scores for different pre/post motion.

  3. Approach/rebound evidence also enters scoring. Of 213 valid videos, 93 have interaction weights below 1: 29 final scores fall by 1, while 64 stay unchanged through rounding or the score floor. Event times, residuals and weights make deductions traceable.

  4. Near misses can conserve momentum and receive high R23 scores if they pass proximity detection. Unavailable measurements are skipped, so the mean can change with included videos. Scores should therefore be read alongside the depicted event, measurement coverage and IF-3 action adherence where assigned, distinguishing requested collision completion from measured conservation.

  5. Other event-based dimensions explicitly penalize omissions: all 632 records lacking the target phenomenon in R06/R07/R14/R25 receive the minimum score, as do all 104 R05 records without detected deformation. Event occurrence already enters scoring; per-dimension conditions will be specified in the appendix.


Q3: How R05 scores expected deformation

Thank you for your question about expected deformation.

  1. R05 first checks for deformation, using the average Frobenius norm of Green–Lagrange strain from tracked points. Values ≤ 0.02 receive 1; magnitude bins apply above this threshold. Thus, zero recovered strain receives the minimum on compression tasks, as in all 104 such records.

  2. Prompt U46V2-0149 asks two plates to compress and release a foam cylinder, a counterpart to sponge compression. All 4 outputs without detected deformation score 1; all 8 with deformation score 4 or 5. Scoring distinguishes remaining rigid from producing the requested compression.

  3. R05 uses common strain intervals across materials. Where R06 is also assigned, seven material-family rubrics assess material response, including elastic recovery of compressed sponge and persistent plastic deformation of clay. The appendix will separately define deformation magnitude and material-behavior criteria.

  4. Known trajectories test strain calculation; the foam task tests score direction on generated outputs. R05 and R06 correlate with human ratings at ρ = 0.622 and 0.867 on 15 examples each, further supporting both diagnostics.

Thank you for the careful response, which addresses many of my concerns. I maintain my score.

Dear Reviewer RUtJ,

Thank you for your positive assessment of PVBench and your thoughtful feedback. We are glad that our response has addressed many of your concerns. We hope that the additional validation in our rebuttal and the clarifications in the revised manuscript further strengthen your overall assessment of our work.

Thank you again for your time and constructive discussion.

Best regards,
Authors of PVBench

Reviewer_2YYjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PVBench, a benchmark with 600 paired prompt groups for evaluating the realism of synthetically generated videos in both T2V and I2V settings. It includes 46 dimensions across 7 domains, organized into semantic, perceptual, and physical assessment layers. Of these, directly 25 dimensions belong to the physical layer.

The benchmark focuses on measuring physical plausibility through numerical quantities extracted by specialized pretrained vision models, rather than relying only on MLLM judgments. MLLMs are still used for dimensions that are difficult to measure directly. The extracted quantities are checked against physical constraints and mapped to scores. The authors evaluate six video generation models in both T2V and I2V settings and compare the automatic scores with human ratings.

Critical strengths
  1. The paper clearly motivates the limitations of relying only on MLLM judgments for evaluating physical plausibility. Measuring specific physical quantities can provide more detailed and testable information about physical errors.
  2. Supporting both T2V and I2V with paired scene descriptions is useful for comparing models under different input conditions and studying the effect of the first-frame reference image.
  3. The numerical evaluation approach is promising. The comparison in Table 7 provides evidence that, in the tested setting, these scores distinguish videos better and agree more closely with human ratings than direct MLLM scoring.
Critical weaknesses
  1. The choice of 46 dimensions is not sufficiently justified. Several semantic and perceptual dimensions follow existing benchmarks, and some dimensions appear closely related. Although the main contribution is the numerical evaluation approach, the taxonomy itself feels more like a combination of existing evaluation categories. An analysis of redundancy and the additional information provided by each dimension would help explain why this many dimensions are needed.
  2. The evaluation details are not equally clear across dimensions, and the choice of specialized models is not sufficiently supported by comparisons with alternatives. For example, R03 and R11 share vision models but apply different calculations to their outputs: R03 measures shape consistency using normalized Chamfer distance after point-cloud alignment, while R11 compares the relative 3D positions of objects with the relations described in the prompt. However, unlike R03, R11 does not have clearly specified relation-checking rules, tolerances, or thresholds for mapping the results to 1–5 scores. It is unclear, for example, how containment is checked using the extracted geometry. These details should be clarified to make the evaluation easier to understand and reproduce.
  3. Table 8 shows that human alignment is sensitive to evaluator thresholds. For example, changing the SAM 3 confidence threshold from 0.10 to 0.50 reduces Spearman correlation from 0.601 to 0.132. This raises concerns about how well the selected thresholds transfer to other videos and generation models.
  4. Human alignment varies substantially across dimensions. In my view, the overall Kendall correlation of 0.643 is not sufficient on its own to support the paper's broad claim of high consistency with human judgments. Table A4 reports even lower values for object permanence (0.108), motion continuity (0.214), and light and shadow spatial consistency (0.246). These are important parts of physical plausibility, so the overall score does not fully reflect the weaknesses of individual evaluators. The claims about human alignment should therefore be more qualified.
Questions for authors

Iterating over my comments on the weaknesses,

  1. What criteria were used to select the 46 evaluation dimensions? Have you analyzed redundancy or overlap between dimensions?
  2. Could you clarify the rules, tolerances, and score thresholds for per-dimension scoring principles? Please add formal definitions.
  3. Given the large changes in human alignment under different thresholds (Table 8), how well do the selected thresholds generalize to unseen videos and generation models?
  4. How do the authors interpret the low correlations for object permanence, motion continuity, and light–shadow spatial consistency in Table A4? Do these results limit the claim of high consistency with human judgments?
3 comments

Q1: The 46 dimensions provide complementary diagnostic information

Thank you for this question.

  1. We retain criteria for specific, observable failures with executable scoring rules. Missing an action, rendering it with unstable imagery, and rendering it clearly but violating physics require distinct semantic, perceptual and physical criteria.
  2. The 25 physical dimensions cover objects, spatial relations and temporal changes (7/7/11). The 14 semantic and 7 perceptual dimensions assess instruction following and visual/temporal quality. Established criteria contextualize physical scores; our focus is measuring objects, trajectories and interactions.
  3. Comparing scores on the same videos yields 3,890 jointly scored outputs covering 230 dimension pairs. Spearman correlation is defined for 190; 156 have ∣ρ∣<0.5|ρ| < 0.5. Together with the ordering comparisons below, this supports complementarity in the 46-dimensional evaluation as a whole, beyond repeating one score. The appendix will include the matrix and shared-sample counts.
  4. With prompt and modality fixed, 149/230 pairs reverse the ordering of the same two models at least once; every dimension participates in a reversal pair. Requiring both score gaps to reach 2 points leaves 58 pairs involving 38 dimensions. Different criteria thus reveal different model strengths on tested scenes, supporting dimension-level diagnosis.

Q2: Formal scoring definitions

Thank you for requesting clarification.

  1. Appendix D.2–D.3 will specify formulas, tolerances, score bins and missing-measurement handling; D.4 will give semantic/perceptual rubrics. Reference and reference-free rules will be distinguished to match the main experiments.
  2. R03 assesses individual shape; R11 assesses inter-object overlap using masks and relative depth. Its thresholds are 10% of the larger mask area and 15% of the narrower normalized depth interval. Depth flags and persistent overlap give P=2D+TP=2D+T, mapped to five grades. We will specify definitions and bin boundaries.
  3. For containment, R11 measures mask/depth overlap. Where IF-4 is also assigned, it checks requested containment or support, providing instruction adherence alongside measured geometry.
  4. Models supply the required inputs: SAM3 gives masks, VGGT relative geometry, and VLMs assess complex-shape integrity. Three VLM backends give pairwise ρ=0.705–0.836, supporting this component.

Q3: Evidence for threshold generalization

Thank you for asking about transfer.

  1. One protocol handles six generators in T2V/I2V: 6,044/6,291 physical records are scored (96.1% coverage) across 12 configurations, demonstrating applicability across generators and input modes.
  2. With one tracking-area setting on the same 30 prompts across six motion dimensions, human-score correlations are 0.892/0.853/0.768 for Seedance/MiniMax H3/Wan2.1. The MiniMax-selected setting retains agreement on two other generators, supporting transfer.
  3. In a retrospective five-fold check, we hold out one prompt per dimension and select using the other 24 videos. All folds choose the main setting; pooled held-out ρ=0.853. Full-30-video ρ is 0.853 versus 0.721/0.654 for alternatives. The selection is stable across prompts.
  4. On 20 R01/R09 videos measurable in both existing SAM3 result sets, identical human ratings give ρ=0.914/0.911; 19 integer grades match. The sets score 75/89 and 71/89 records overall. Similar shared-video scores and differing coverage show why threshold selection should also consider measurement coverage.

Q4: Interpreting correlations with human ratings by dimension

Thank you for raising these points.

  1. On 375 video–dimension pairs from three T2V models, ρ=0.772 and τ-b=0.643; 20/25 physical dimensions have ρ≥0.5, and mean dimension-level τ-b=0.556. Multiple properties support overall agreement. We will present pooled and per-dimension results together.
  2. R13/R20/R21 are weakly positive in this sample: ρ=0.269/0.240/0.125 and τ-b=0.246/0.214/0.108. Lighting-direction recovery, stable motion estimates and identity across occlusion demand capable perception models. Their accuracy affects rankings; saved intermediate outputs help trace and address these effects.
  3. Grading explains part of the difference: on the same 15 R20 videos, ρ falls from 0.356 before rounding to 0.240 after five-grade conversion. People can also miss lighting inconsistencies in scenes without regular geometry, so numerical direction changes and overall impressions may differ in sensitivity (Ostrovsky et al., 2005).
  4. Of 158 same-prompt comparisons distinguishable by both methods, 142 agree (89.9%). All 375 comparisons include 208 automated ties and 9 human-only ties. Removing R13/R20/R21 preserves all six model ranks within each modality; remaining pairs give ρ=0.789, τ-b=0.660. Matched comparisons and dimension exclusion support our results beyond pooled correlation.

Thank you for the clarifications and the changes made to the manuscript. But I still think my question regarding how the selected evaluator thresholds (Q3) generalize still needs some extra validation. It would be nice to test a new video generation model (not included in the current study) and show that the scores are still aligned with human judgments. Moreover, I see that the authors have not uploaded the revised version of the paper yet.

Dear Reviewer 2YYj,

Thank you for your follow-up and helpful suggestion. We have added an experiment on a new video generator to directly test the transfer of the selected thresholds.

Q1: Generalization of evaluator thresholds

1. Validation on a new generator

We evaluated the T2V outputs of LTX-Video 0.9.8 Distilled, which was not included in the original study, on R01 depth and R11 spatial relations. The videos follow the original specification of 720p, 10 seconds, and 24 fps. We distributed a new evaluation questionnaire to the same 10 annotators from the original study and collected their independent ratings. Their mean score for each video was used as the human rating. We kept the original scoring rules unchanged and compared the three SAM3 confidence thresholds from the paper, without retuning them using the new human ratings.

ScopeSAM3 thresholdCoverageSpearman ρKendall τ-b
R010.05100%0.8450.736
R010.10100%0.8450.736
R010.5086.4%0.6100.538
R110.05100%0.4230.371
R110.10100%0.8030.708
R110.5083.3%0.1460.116
Overall0.05100%0.7530.583
Overall0.10100%0.8670.729
Overall0.5085.0%0.5790.481

The previously selected 0.10 threshold achieves the highest overall human agreement on the new generator while maintaining 100% coverage. The per-dimension results support this choice: it ties with 0.05 on R01 and outperforms both alternatives on R11. These results directly support transfer of the selected threshold to the new generator and the two tested dimensions.

2. The role of threshold calibration

The SAM3 confidence threshold filters candidate masks. Subsequent scores are determined by dimension-specific physical measurements and constraint checks. For example, R01 compares object depth ordering, while R11 checks spatial anomalies using multi-frame segmentation and geometry. This parameter affects metrics using the corresponding detection path, with effects that vary across dimensions. It is not shared by all 46 dimensions. The identical R01 results at 0.05 and 0.10 also show that changing the candidate-filtering threshold does not necessarily change the final physical scores.

3. PVBench's core contributions

With the available foundation models, we have taken multiple steps to support evaluation validity and transfer: fixed scoring rules, unchanged settings across generators, separate recording of unavailable measurements, and independent human validation, now extended to a new generator. Threshold calibration is a supporting component. Its effects on a subset of metrics do not change PVBench's core methodological contributions: measuring physical plausibility through object-level quantities and explicit constraints, providing interpretable numerical evidence beyond holistic MLLM judgments, and enabling detailed diagnosis across 46 dimensions with paired T2V/I2V prompts. The new experiment further supports the practical use of this approach without generator-specific recalibration.

4. Foundation-model capabilities and future work

We acknowledge that further improvements in measurement depend closely on the detection, segmentation, and geometry-recovery capabilities of the foundation models. Different confidence thresholds can retain or remove different candidates, affecting coverage and scores for some metrics. Nevertheless, the new results show that the previously selected setting maintains strong agreement with independent human judgments without recalibration for the new generator.

As segmentation and geometry models improve, we expect measurement coverage and stability to improve further. We will include stronger perception models and continued validation across additional generators and scenes in our future work.

Q2: Availability of the revised manuscript

  1. The revised manuscript was already uploaded and is available through the paper link.
  2. Thank you for reminding us to make the changes easier to locate. We have now marked the modified passages in red for convenient reference.

We hope that the additional validation in our rebuttal and the clarifications in the revised manuscript further strengthen your overall assessment of our work.

Thank you again for your time and constructive feedback.

Best regards,
Authors of PVBench

Reviewer_qFtjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PVBench, a benchmark for evaluating physical plausibility in text-to-video and image-to-video generation. It covers 46 dimensions across semantic, perceptual, and physical aspects using 600 paired prompts. Its main contribution is to complement VLM-based judgments with measurements from specialized vision models, including geometry, object motion, and conservation-law consistency. The authors evaluate several recent video generation models and validate the automatic metrics through human annotation. The results suggest that the proposed measurements generally agree with human judgments and that current models remain considerably weaker in physical plausibility than in semantic alignment or visual quality, especially for material behavior, mechanics, and conservation laws.

Critical strengths
  1. The most important contribution is the attempt to operationalize physical plausibility through measurable states and residuals rather than relying exclusively on binary human or MLLM judgments.

  2. The broad taxonomy, paired T2V/I2V prompts, and inclusion of embodied and difficult scenes make PVBench potentially useful for diagnosing model failures.

  3. The paper evaluates 12 model configurations and provides domain- and dimension-level results. The reported human agreement, evaluator-backend comparison, and numerical-versus-MLLM analysis support the central claim that quantitative measurements can be more discriminative than semantic scoring.

Critical weaknesses
  1. I am not truely convinced that all of the reported “physical quantities” are reliable. For example, absolute scale is difficult to recover from an unconstrained monocular video, but it is later used to evaluate acceleration and energy. The displacement metric also appears to assign a better score to smaller motion, which may favor nearly static videos. These points need clarification since quantitative measurement is the paper’s main contribution.

  2. The human study is encouraging but still limited. It uses only three T2V models and a small number of examples per dimension. It is also unclear whether the same human ratings were used both to tune the thresholds and to report the final correlations. Experimental results show that everal individual metrics have rather low correlations even though the overall result is strong.

Questions for authors
  1. Could the authors clarify the displacement metric R15? Why does Table A15 treat smaller displacement as better? Is this supposed to be an error relative to the expected motion rather than the displacement itself?

  2. How is reliable metric scale obtained from a generated monocular video? How sensitive are the acceleration and energy metrics to errors in this scale estimate?

  3. Were the human annotations used to tune any thresholds? If so, is there a separate held-out set for reporting the final correlations?

  4. Some dimensions, such as motion continuity and object permanence, show low correlation with human ratings. Do the authors have an explanation for these cases, and how much do they affect the overall score?

3 comments

Q1: R15 does not reward smaller total displacement

Thank you for asking us to clarify R15.

  1. Table A15's R15_worldline_l2 is position error against a reference: long motion can have low error. Given T2V/I2V model limitations, our 216 samples lack reference tracks and use continuity scoring. We will distinguish both cases in the appendix and Table A15.

  2. Reference-free scoring checks position jumps, then deducts one point each for abnormal object-count changes and speed increases after reversal. Total path length is only recorded. All 216 records have zero jump rate; the other two checks reproduce their grades, identifying each deduction.

  3. In 24/36 prompt–modality groups, a longer path scores above a shorter one on the same scene. R15 therefore does not grade total distance as “smaller is better.”

  4. Continuous tracking of a stationary object can also score highly. Where IF-3 is assigned, it separately checks action completion: LingBot I2V on pendulum prompt U46V2-0317 scores 5 on R15 and 2 on IF-3. R15 has ρ = 0.765 on 15 human-rated examples. Removing R15 and averaging the other 24 physical dimensions preserves all six model ranks within each modality.


Q2: Effects of monocular scale errors on physical scores

Thank you for your question about metric scale.

  1. Depth Anything 3 supplies approximate metric depth. We divide median valid depth in the middle frame by the median norm of tracked positions to convert the 3D tracks to meters.

  2. R16 measures speed stability or acceleration consistency using the standard deviation of speed divided by mean speed, or standard deviation of successive speed differences divided by their mean absolute value. With tracks and sampling fixed, a common length factor cancels. The reported R16 score thus does not require metric conversion.

  3. R24 checks mechanical energy and successive bounce-height ratios. Length scaling by ss multiplies kinetic/potential energy by s2s^2/ss, but preserves peak-height ratios relative to the trajectory minimum. Of 252 records, 220 lose points only through bounce checks; in 142, these alone reach the minimum grade of 1. With tracks and peaks fixed, those 142 final grades are scale-invariant. These results arise from scale-independent bounce checks and are thus little affected by scale errors.


Q3: Correlation with human ratings after excluding calibration samples

Thank you for asking about calibration.

  1. Tracking-area calibration used 30 MiniMax human-rated pairs. The reported ρ = 0.772 uses all 375 pairs. Retrospectively excluding those 30 leaves 345 pairs across all 25 dimensions, with ρ = 0.765: only 0.007 lower.

  2. A stricter exclusion removes all three generators' outputs for each prompt used in known area calibration or additional SAM3 ratings. The remaining 282 pairs across 19 dimensions give ρ = 0.766, retaining similar agreement without those scenes.

  3. Five prompts per dimension, three T2V models and ten raters per video yield 125 prompts, 375 video–dimension pairs and 3,750 judgments. Within-model ρ is 0.811/0.840/0.840 for Seedance/MiniMax/Wan2.1. Prompt-group resampling gives a descriptive 95% interval of [0.722, 0.819] for overall ρ.

SamplePairsDimsSpearman ρKendall τ-b
Full375250.7720.643
Exclude reused pairs345250.7650.634
Exclude overlapping prompts282190.7660.635

Q4: Explaining lower correlations with human ratings

Thank you for your comments on these correlations.

  1. R20 needs interpolation and 3D tracks; R21 must recover object identity after occlusion. Segmentation, tracking or identity errors enter these scores. R21 logs include targets detected in every reappearance frame whose original ID never returns. Stronger foundation models can improve these measurements.

  2. Human judgments and numerical tests can differ in sensitivity to lighting changes. Ostrovsky et al. found illumination inconsistencies harder to detect without regular geometry, including in real-scene images. This offers a plausible explanation for R13 differing from holistic ratings (2005).

  3. Five grades compress differences. The 13 R20 outputs graded 4 span continuous scores 4.04–4.91. On the same 15 videos and ratings, continuous-score ρ = 0.356 versus 0.240 after rounding. Some ordering consistent with people becomes tied.

  4. R13/R20/R21 retain weak positive sample trends: ρ = 0.269/0.240/0.125. Removing them and equally averaging the other 22 dimensions preserves all six model ranks within both T2V and I2V; scores fall by 0.26–2.46 on a 0–100 scale. The remaining 330 human-rated pairs give ρ = 0.789 and τ-b = 0.660. With 20/25 physical dimensions at ρ ≥ 0.5, multiple properties support the overall result.

Thank authors for their detailed rebuttal. As my major concerns have been sovled, and I still think this paper has good contributions, I decide to keep my original positive score.

Dear Reviewer qFtj,

Thank you for your careful review and positive assessment of our work. We are glad that our rebuttal has addressed your major concerns. Your constructive feedback has helped us clarify the evaluation protocol and improve the manuscript.

Best regards,
Authors of PVBench

Comments on the paper

1 comment

We have uploaded a revised manuscript in response to the reviewers' comments. Appendix B clarifies how the capabilities of the underlying vision models affect agreement with human ratings. Sections D.2 and D.3, together with Table A15, clarify the scoring definitions, the evaluation rules with and without reference data, and how event occurrence is handled in scoring. Appendix H and Figure A1 provide the dimension correlation matrix and shared video counts. All changes are confined to the appendices. The main text and original experimental results remain unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.