acceptodds
Under review as a conference paper at ICLR 2027

Does One Prompt Represent a Physical Scenario? Physics Evaluation through Prompt Sets

Abstract

Evaluating whether video generation models understand the physical world is essential, since physical understanding is a prerequisite for their use as world models. Existing physics benchmarks generate a video from a text prompt describing a physical scenario and score its physical plausibility. However, each scenario is represented by a single prompt, although it can be described in many semantically and physically equivalent ways. We show that this single-prompt design makes the evaluation sensitive to the choice of prompt. Using a large language model, we rewrite benchmark prompts while preserving their semantics and physics, and evaluate five video generation models on VideoPhy-2 and PhyGenBench with fixed seeds. Rewriting alone changes the physical score in 35.5% and 35.7% of cases, flips the official pass decision in 13.3%, and changes the point ranking of 4 of the 5 models on each benchmark. To address this prompt sensitivity, we propose Prompt Set Evaluation, which scores each scenario over a set of validated equivalent prompts and reports the mean score with its variation; five prompts per scenario substantially improve the stability of scores and rankings. The sensitivity is not random noise, since it persists under fully repeatable evaluators. In a blinded human study of video pairs whose scores differ, raters seldom share the evaluator's preference and judge about half of the pairs equally plausible, which suggests that the vision-language models serving as evaluators also respond to differences between videos unrelated to physics. Physical understanding should therefore be evaluated over a distribution of descriptions rather than a single prompt.

Primary area:
datasets and benchmarks
Keywords:
Video generation, Physical understanding evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WRUBOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of physics evaluation for text-to-video generation models, focusing on the sensitivity of evaluation results to semantically equivalent prompts. The authors propose Prompt Set Evaluation (PSE), which constructs multiple equivalent descriptions of the same physical scenario using an LLM and evaluates video generation models by aggregating scores across these prompts. Experiments on VideoPhy-2 and PhyGenBench with five video generation models show that prompt rewriting changes physical scores in approximately 35% of comparisons and affects model rankings. The proposed approach improves score stability and ranking consistency. The authors further analyze the sources of evaluation variability through repeated scoring, deterministic evaluators, human judgments, and generation-seed comparisons.

Critical strengths
  1. The paper identifies an important limitation of existing physics benchmarks for video generation. The finding that semantically equivalent prompts can substantially affect evaluation results provides useful insights into the reliability of current evaluation protocols.
  2. The paper presents relatively comprehensive experimental analyses across multiple benchmarks and video generation models. The additional experiments involving evaluator repeatability, human judgments, and generation seeds strengthen the evidence for prompt sensitivity.
Critical weaknesses
  1. The proposed PSE primarily combines equivalent prompt generation with score averaging and variance estimation. Multi-prompt evaluation has already been investigated in previous studies, such as Mizrahi et al. (2024). Therefore, the main contribution appears to be an empirical extension of existing evaluation strategies to video physics benchmarks rather than a substantially new methodology.
  2. The main advantage of PSE is demonstrated through reduced score variance and improved ranking consistency. However, such improvements are largely expected when averaging multiple samples. More importantly, the human evaluation results suggest that automatic evaluators may respond to visual differences unrelated to physical plausibility. PSE does not directly address these systematic biases, and the paper provides limited evidence that the aggregated scores better reflect the actual physical understanding of video generation models.
  3. The experiments use only 15 scenarios from each benchmark, which may not adequately represent the diversity of physical phenomena. The independent prompt-set validation and prompt-versus-seed experiments are also restricted to a small number of models and scenarios. These limitations weaken the generalizability of the conclusions.
Questions for authors
  1. Can the authors provide more direct evidence that PSE improves the validity of physics evaluation rather than merely reducing score variance? For example, would PSE achieve better agreement with human judgments or physically grounded reference evaluations than the original single-prompt protocol?
  2. Under a fixed computational budget, how does increasing the number of prompts compare with evaluating more physical scenarios or using more generation seeds? Would the reported improvements remain significant with a larger and more diverse set of scenarios?
Reviewer_wFkYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors identify a very insightful but relatively overlooked phenomenon: physics benchmarks for video generation are sensitive to the choice of prompts. To address this issue, they propose Prompt Set Evaluation, which uses an LLM to generate semantically and physically equivalent prompt rewrites and calculates the mean evaluation score across the prompt set to mitigate prompt sensitivity.

The experiments show that prompt rewriting alone changes the physical score in 35.5% and 35.7% of cases on two benchmarks, flips the official pass decision in 13.3% of cases on VideoPhy-2, and changes the rankings of 4 out of 5 models on each benchmark.

Critical strengths

1.The paper provides an insightful observation that physics benchmarks for video generation are sensitive to prompt variations. Even when the rewritten prompts preserve the same physical meaning, they can change evaluation scores, pass/fail decisions, and model rankings across two benchmarks, despite fixed generation seeds and physical judgments that do not directly use the rewritten prompts.

2.This is an important and relatively underexplored direction that deserves more attention and further investigation from the research community.

Critical weaknesses

1.Limited analysis of the causes of prompt sensitivity. Although the paper identifies this phenomenon, the authors should provide a more detailed analysis of how prompt changes affect model performance and which factors, such as model architecture, module design, training data, or training pipelines, contribute to this sensitivity. Furthermore, identifying which specific prompt changes trigger significant performance differences would provide more useful insights for the research community.

2.Insufficient evidence of the effectiveness of Prompt Set Evaluation beyond score stability. The experiments demonstrate that prompt changes can lead to different evaluation scores. However, it remains unclear whether these score changes reflect genuine improvements or degradations in physical plausibility. Although the paper includes qualitative examples and human evaluations, more targeted qualitative results and experiments would help demonstrate whether Prompt Set Evaluation improves the validity of physics evaluation, rather than merely reducing score variation.

Questions for authors

1.Could the authors provide a more detailed analysis of how different prompt variations affect video generation performance? Are there specific model architectures, module designs, training data characteristics, or training pipelines that contribute to prompt sensitivity?

2.Although the paper describes how equivalent prompts are generated and validated, could the authors provide more practical guidance on how to design and select prompt sets? In particular, which types of prompt changes are most likely to trigger substantial differences in physical evaluation scores?

3.How can we determine whether a score change caused by prompt rewriting reflects a genuine difference in physical plausibility or an evaluation artifact? Could the authors provide additional qualitative comparisons or experiments to better demonstrate the effectiveness of Prompt Set Evaluation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wFkY

Dear Reviewer wFkY,

As the discussion period is drawing to a close, we would like to follow up on our response. We would be grateful to know whether it addresses your concerns. If any point remains open, we would be glad to address it, and we hope you can share it soon, so that we can respond in time. Thank you again for your time and your thoughtful review. We look forward to your feedback.

Reviewer_wFkYOpenReview ↗

Thank You for the Rebuttal and Discussion on Interpretability

I appreciate the authors' rebuttal and additional experiments, which provide stronger empirical evidence of prompt sensitivity in video generation.

However, my main remaining concern is the lack of deeper insights into why semantically equivalent prompt variations lead to substantial performance differences. The current analysis mainly characterizes the phenomenon without sufficiently explaining its underlying mechanisms, making the paper feel more like an empirical report than an explanatory study.

I recognize the value of these empirical findings, but I am still weighing them against my remaining concerns before deciding whether to revise my initial rating.

AuthorsOpenReview ↗

Response to Reviewer wFkY

We thank the reviewer for the response and for pressing us on the causes of the sensitivity.

The contribution of the paper. We agree that the paper is primarily an empirical study, and that an analysis of the causes would be valuable for the community. We believe, however, that the first step is to establish the phenomenon and to understand its extent, and we are glad that the reviewer recognizes the value of these empirical findings. The main contribution of this paper is to identify a problem: physics benchmarks for video generation score each scenario with a single prompt, yet an equivalent prompt changes the score in about a third of the comparisons, flips pass decisions, and changes the ranking of the models. We believe this finding matters for how such benchmarks are designed and reported, whatever its cause, because a conclusion drawn from one prompt may not hold for another.

What we know about the cause. Although the analysis of the cause is not the main contribution, it narrows the cause down:

  • The sensitivity is not evaluator noise: the VideoPhy-2 evaluator returns identical scores when it scores a video again.
  • A rewrite changes the generated video about as much as a new seed does.
  • Many score changes are not physical differences that people see: raters judged 41 of 80 such video pairs equally plausible.
  • The sensitivity is not specific to one architecture or model size: it appears in all five models.

What remains open. We regret that we cannot provide rigorous experiments on the factors the reviewer asks about, such as architecture, module design, and training data, within the discussion period. Such experiments need extensive ablation studies on models that differ in one factor only, which is compute-intensive. We consider them important future work. We hope that this paper helps future benchmarks evaluate the physical understanding of video models more comprehensively and fairly.

Reviewer_dWhBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether physics benchmarks depend excessively on how a scenario is described. The authors generate conservative paraphrases, validate their physical and semantic equivalence, and evaluate generated videos while keeping the physics evaluator’s questions unchanged. They propose Prompt Set Evaluation: average performance over equivalent descriptions and report its variation. The central contribution is evidence that description choice is a consequential source of uncertainty in physical-video evaluation.

Critical strengths
  1. Validated paraphrases, fixed generation seeds, and deterministic evaluator controls convincingly demonstrate wording sensitivity in physics evaluation. This identifies a consequential source of benchmark uncertainty.

  2. Independent prompt-set replication and an equal-cost prompt–seed comparison provide practical support for averaging across descriptions.

Critical weaknesses
  1. Methodological novelty is limited: averaging over paraphrases is established in Mizrahi et al. (2024). The blanket single-prompt characterization also overlooks PhyWorldBench’s multiple prompt types, although those variants differ from strictly equivalent rewrites.

  2. The evidence supports a narrower conclusion than general benchmark reliability: only 15 scenarios per benchmark are studied, no ranking reversal is statistically resolved, and low human agreement prevents attributing evaluator differences specifically to nonphysical bias.

Questions for authors
  1. What explains the differences between the main and rescored VideoPhy-2 results in Table S6, given the reported identical checkpoint, frame sampling, decoding, and library versions?

  2. Given the different filtering and selection rules in Appendix F, should the second prompt set be interpreted as replication under the same distribution or robustness across different construction procedures?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dWhB

Dear Reviewer dWhB,

As the discussion period is drawing to a close, we would like to follow up on our response. We would be grateful to know whether it addresses your concerns. If any point remains open, we would be glad to address it, and we hope you can share it soon, so that we can respond in time. Thank you again for your time and your careful and constructive review. We look forward to your feedback.

Comments on the paper

1 comment

Dear reviewer, the authors have submitted a rebuttal. Can you please engage with it before the end of the discussion period? Thanks, your AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.