- Rating
- 2
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the reliability of physics evaluation for text-to-video generation models, focusing on the sensitivity of evaluation results to semantically equivalent prompts. The authors propose Prompt Set Evaluation (PSE), which constructs multiple equivalent descriptions of the same physical scenario using an LLM and evaluates video generation models by aggregating scores across these prompts. Experiments on VideoPhy-2 and PhyGenBench with five video generation models show that prompt rewriting changes physical scores in approximately 35% of comparisons and affects model rankings. The proposed approach improves score stability and ranking consistency. The authors further analyze the sources of evaluation variability through repeated scoring, deterministic evaluators, human judgments, and generation-seed comparisons.
Critical strengths
- The paper identifies an important limitation of existing physics benchmarks for video generation. The finding that semantically equivalent prompts can substantially affect evaluation results provides useful insights into the reliability of current evaluation protocols.
- The paper presents relatively comprehensive experimental analyses across multiple benchmarks and video generation models. The additional experiments involving evaluator repeatability, human judgments, and generation seeds strengthen the evidence for prompt sensitivity.
Critical weaknesses
- The proposed PSE primarily combines equivalent prompt generation with score averaging and variance estimation. Multi-prompt evaluation has already been investigated in previous studies, such as Mizrahi et al. (2024). Therefore, the main contribution appears to be an empirical extension of existing evaluation strategies to video physics benchmarks rather than a substantially new methodology.
- The main advantage of PSE is demonstrated through reduced score variance and improved ranking consistency. However, such improvements are largely expected when averaging multiple samples. More importantly, the human evaluation results suggest that automatic evaluators may respond to visual differences unrelated to physical plausibility. PSE does not directly address these systematic biases, and the paper provides limited evidence that the aggregated scores better reflect the actual physical understanding of video generation models.
- The experiments use only 15 scenarios from each benchmark, which may not adequately represent the diversity of physical phenomena. The independent prompt-set validation and prompt-versus-seed experiments are also restricted to a small number of models and scenarios. These limitations weaken the generalizability of the conclusions.
Questions for authors
- Can the authors provide more direct evidence that PSE improves the validity of physics evaluation rather than merely reducing score variance? For example, would PSE achieve better agreement with human judgments or physically grounded reference evaluations than the original single-prompt protocol?
- Under a fixed computational budget, how does increasing the number of prompts compare with evaluating more physical scenarios or using more generation seeds? Would the reported improvements remain significant with a larger and more diverse set of scenarios?
Response to Reviewer wFkY
Dear Reviewer wFkY,
As the discussion period is drawing to a close, we would like to follow up on our response. We would be grateful to know whether it addresses your concerns. If any point remains open, we would be glad to address it, and we hope you can share it soon, so that we can respond in time. Thank you again for your time and your thoughtful review. We look forward to your feedback.