acceptodds
Under review as a conference paper at ICLR 2027

PhyGround: Benchmarking Physical Reasoning in Generative World Models

Abstract

Generative world models are increasingly used for video generation, where learned simulators are expected to capture the physical rules that govern real-world dynamics. However, evaluating whether generated videos actually follow these rules remains challenging. Existing physics-focused video benchmarks have made important progress, but they still face three key challenges, including the coarse evaluation frameworks that hide law-specific failures, response biases and fatigue that undermine the validity of annotation judgments, and automated evaluators that are insufficiently physics-aware or difficult to audit. To address those challenges, we introduce , a criteria-grounded benchmark for evaluating physical reasoning in video generation. The benchmark contains 250 curated prompts, each augmented with an expected physical outcome, and a taxonomy of 13 physical laws across solid-body mechanics, fluid dynamics, and optics. Each law is operationalized through observable sub-questions to enable per-law diagnostics. We evaluate eight modern video generation models through a large-scale, quality-controlled human study, grounded on social science lab experiment design. A total of 459 annotators provided 5,796 complete annotations and over 37.4K fine-grained labels; after quality control, the retained annotations exhibited high split-half model-ranking correlations (Spearman's ). To support reproducible automated evaluation, we release , an open physics-specialized VLM judge. PhyJudge-9B achieves substantially lower aggregate relative bias than Gemini-3.1-Pro (3.3% vs. 16.6%). We release prompts, human annotations, model checkpoints, and evaluation code; links are available at:https://huggingface.co/anonymouscla.

Primary area:
generative models
Keywords:
Video Generation, World Models, Physical Reasoning, Physics-based Evaluation, Vision-Language Models (VLMs), Benchmarking, Reproducibility

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_t55VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhyGround, a criteria-grounded benchmark for evaluating physical reasoning in video generation, with a large-scale human study and an open physics-specialized VLM judge, PhyJudge-9B.

Critical strengths
  1. The proposed PhyGround benchmark provides a more fine-grained evaluation framework by decomposing physical reasoning into multiple physical laws and observable criteria, enabling more detailed diagnosis of model capabilities.

  2. The large-scale human study, quality control process, and release of PhyJudge-9B improve the reproducibility and scalability of physics-aware video generation evaluation.

Critical weaknesses

1、The effectiveness of PhyJudge-9B would be more convincingly demonstrated with a more comprehensive scaling analysis. Currently, the authors compare PhyJudge-9B with several general-purpose VLMs of different scales, but do not investigate the impact of model size under the same physics-specific training setting. It remains unclear whether the performance improvement mainly comes from the proposed physics-aware supervision or simply from the model capacity. Experiments with different backbone sizes (e.g., smaller and larger variants of the same VLM family) after the same fine-tuning procedure would help better understand the contribution of the proposed training strategy.

2、The authors compare PhyJudge-9B mainly with general-purpose VLMs. Since existing open-source physics-oriented evaluators, such as VideoPhy-2-AutoEval and the WorldModelBench judge, are available, comparisons with these models would provide a more comprehensive evaluation and better position the contribution of PhyJudge-9B.

3、The generality of the proposed physics supervision is not sufficiently evaluated. Currently, PhyJudge-9B is built on a single VLM backbone, making it unclear whether the improvement comes from the proposed physics-specific data or from backbone-specific adaptation. The authors are encouraged to fine-tune multiple open-source VLMs using the proposed annotations and compare their improvements, which would better demonstrate the scalability and transferability of the proposed physics supervision.

4、The comparison with closed-source VLM judges is somewhat limited. Since the paper evaluates Claude and Gemini as general-purpose VLM baselines, including additional strong VLM baselines such as GPT-based models would provide a more comprehensive comparison and better position the performance of PhyJudge-9B.

Questions for authors

see Weaknesses

5 comments

Thank you very much for your constructive comments! We are grateful for your four concrete suggestions, and we have run all four as new experiments. Below, we answer each point in turn. Unless stated otherwise, every number below is on the paper's 250 held-out Veo-3.1 videos. We report aggregate relative bias as defined in Table 5, whose Overall column averages the General and Physics columns (lower is better; "bias" for short below). Our judges and GPT use the paper's judge prompt, and the open evaluators use their official prompts.

Conventions. Significant means that the 95% bootstrap interval of a difference, resampled over prompts, excludes zero.

W1

[...] do not investigate the impact of model size under the same physics-specific training setting. It remains unclear whether the performance improvement mainly comes from the proposed physics-aware supervision or simply from the model capacity. Experiments with different backbone sizes (e.g., smaller and larger variants of the same VLM family) after the same fine-tuning procedure would help [...]

A1: Thank you for this well-posed question. To separate the physics supervision from model capacity, we ran the size ablation you suggest as a new experiment. We took four sizes of one VLM family, Qwen3.5-2B, -4B, -9B (the backbone of PhyJudge-9B) and -27B, and scored the 250 held-out Veo-3.1 videos with each size in two settings. Zero-shot is the released model with no fine-tuning, given the same judge prompt and frames as PhyJudge-9B. PhyJudge recipe is the training procedure of PhyJudge-9B, applied to each size: LoRA supervised fine-tuning (rank 32) for one epoch on the human scores (general dimensions and physical laws) of the seven non-Veo generators, with the hyperparameters of Table 24. The improvement comes from the physics supervision: the PhyJudge recipe brings aggregate bias below 10% at all four Qwen3.5 sizes, far below the zero-shot bias of every size.

Overall aggregate bias (%)2B4B9B (PhyJudge-9B)27B
Zero-shot36.519.931.631.1
PhyJudge recipe8.64.73.37.2

Capacity is not what decides calibration: zero-shot bias does not fall with size (the 27B starts where the 9B does, and the 4B starts lowest), while the recipe calibrates every size from 2B upward, below every closed judge of Table 5 in Overall bias (see A4). We reused the 9B hyperparameters for every size without retuning, so we do not read a size trend into the differences among the fine-tuned sizes. Both 2B entries use the 218 videos for which its zero-shot version returns a parseable answer. We have added this table to the revised PDF (Appendix P.1).

W2

Since existing open-source physics-oriented evaluators, such as VideoPhy-2-AutoEval and the WorldModelBench judge, are available, comparisons with these models would provide a more comprehensive evaluation and better position the contribution of PhyJudge-9B.

A2: Thank you for this constructive suggestion. We evaluated the released VideoPhy-2-AutoEval and WorldModelBench judge checkpoints on all 2,000 PhyGround videos. We distinguish their native outputs when comparing them with our human ratings.

On PhyGround's own scale, judges trained with the PhyJudge recipe are better calibrated than VideoPhy-2-AutoEval. VideoPhy-2-AutoEval returns holistic 1 to 5 semantic-alignment and physics scores per video plus per-rule labels (followed, violated or undeterminable), and the WorldModelBench judge returns five fixed yes/no violation flags. Neither grades a per-law violation on PhyGround's 1 to 5 severity scale.

The WorldModelBench judge's output is the number of its five checks a video passes (0 to 5), with no counterpart on the 1 to 5 scale, so the calibration comparison is with VideoPhy-2-AutoEval. As PhyGround's Physics scorer, every generator is scored by one of two family-held-out judges (the PhyJudge recipe on Qwen3.5-9B, each trained without any video from the scored generator's model family). VideoPhy-2-AutoEval's holistic physics score stands in for every law of its video.

Relative bias vs. human mean (%)PhyJudge recipeVideoPhy-2-AutoEval
Physics, mean absolute over the 8 generators (family-held-out judges)8.118.6

In short, used as PhyGround's Physics scorer, the PhyJudge recipe tracks human ratings significantly more closely than VideoPhy-2-AutoEval on generator families its judge never saw in training, and PhyJudge-9B is the only one of the three evaluators that returns PhyGround's per-law 1 to 5 scores.

W3

PhyJudge-9B is built on a single VLM backbone, making it unclear whether the improvement comes from the proposed physics-specific data or from backbone-specific adaptation. The authors are encouraged to fine-tune multiple open-source VLMs using the proposed annotations and compare their improvements [...]

A3: Thank you for asking whether the supervision transfers across backbones. The physics supervision also transfers to a backbone from outside the Qwen family. We applied the PhyJudge recipe of A1 to MiMo-VL-7B (Xiaomi LLM-Core Team, 2025), whose language model (Xiaomi's MiMo-7B) comes from outside the Qwen family, and its aggregate bias falls from 24.4% zero-shot to 15.3%, with every answer in the required score format.

Counting the four Qwen3.5 sizes in A1, the PhyJudge recipe lowers the bias of all five backbones reported here. We have added the MiMo-VL-7B row to the same table (Appendix P.1).

W4

Since the paper evaluates Claude and Gemini as general-purpose VLM baselines, including additional strong VLM baselines such as GPT-based models would provide a more comprehensive comparison [...]

A4: Thank you for suggesting a GPT baseline; PhyJudge-9B is far better calibrated than both GPT versions we tested, gpt-6-astra and gpt-5.5. Following the suggestion, we ran both as closed judges on the 250 held-out Veo-3.1 videos, with the identical prompt and frames that PhyJudge-9B receives. The two closed judges of Table 5 are shown for reference.

Relative bias (%), held-out Veo-3.1GeneralPhysicsOverall
PhyJudge-9B1.05.53.3
gpt-6-astra23.434.729.0
gpt-5.523.234.729.0
Claude Opus 4.7 (Table 5)16.210.013.1
Gemini-3.1-Pro (Table 5)17.315.916.6

PhyJudge-9B has the lowest bias in every column, significantly below both GPT versions.

We hope these four experiments address the reviewer's concerns.

References

Xiaomi LLM-Core Team. (2025). MiMo-VL technical report. arXiv:2506.03569.

AuthorsOpenReview ↗

Thank You for Your Insightful Suggestions; We Hope to Address All of Them Before the Discussion Closes in 2 Days. Could You Kindly Let Us Know If Any Concerns Remain?

Dear Reviewer t55V,

Thank you so much for your insightful suggestions! In our responses, we have addressed each of your concerns:

  1. Physics supervision versus model capacity (W1). We applied the PhyJudge recipe to four sizes of one VLM family (Qwen3.5-2B to -27B). It brings aggregate bias below 10% at every size, far below the zero-shot bias of every size, so the improvement comes from the physics supervision rather than from capacity (Appendix P.1).
  2. Comparison with open physics evaluators (W2). Used as PhyGround's Physics scorer on generator families its judge never saw in training, the PhyJudge recipe tracks human ratings significantly more closely than VideoPhy-2-AutoEval. PhyJudge-9B is also the only one of the three evaluators that returns PhyGround's per-law 1 to 5 scores.
  3. Transfer across backbones (W3). The same recipe also lowers the bias of MiMo-VL-7B, whose language model comes from outside the Qwen family, so the supervision is not tied to one backbone (Appendix P.1).
  4. GPT-based judges (W4). We added gpt-6-astra and gpt-5.5 as closed judges with the identical prompt and frames. PhyJudge-9B has the lowest bias in every column, significantly below both GPT versions.

Could you kindly let us know whether our responses have addressed your concerns? As the discussion period closes on November 18, we would be grateful to hear from you before then, so that we still have time to address any remaining concern.

Thank you again for your time and valuable feedback!

Reviewer_JkVwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PhyGround to evaluate whether video generation models can produce visually plausible physical processes given a specified outcome. The benchmark contains 250 tasks with text prompts and first-frame images, covering 13 physical criteria across solid-body mechanics, fluids, and optics. Each video is rated on a five-point scale for semantic alignment, physical temporal validity, object persistence, and the applicable physical criteria. The authors conduct a human evaluation of 2,000 videos generated by eight models, retaining 4,576 complete annotations from 352 annotators after quality control. The authors further incorporate criterion-specific sub-questions into the judge prompts as observation checklists and fine-tune PhyJudge-9B on the human ratings. In the main experiment, the judge is trained on data from seven generators and evaluated on held-out Veo-3.1 videos. The results show that generators with similar overall scores can have different strengths across domains, while fine-tuning improves both aggregate score calibration and per-video error relative to the base model.

Critical strengths
  1. The evaluation breakdown is useful, and the human annotation study is well designed. In Table 2, models with similar overall scores perform differently on solid-body and fluid criteria, demonstrating the additional information provided by the breakdown. Random assignment, concealed model identities, quality control, and stable overall rankings from the 352 retained annotators strengthen confidence in the model comparisons.

  2. The judge's fine-tuning gains are supported by substantial experiments. With the same backbone and prompts, aggregate relative bias decreases from 31.6% to 3.3%, and per-video MAE decreases from 1.29 to 0.79 (Tables 5 and 28). Eight-fold leave-one-generator-out evaluation, transfer to VideoPhy-2, holdouts of related groups of physical criteria, and sensitivity analyses of the human reference further support its generalization value.

Critical weaknesses
  1. The criterion scores may conflate event completion with physical plausibility. Appendix L explicitly defines the evaluation as conditioned on a specified outcome, but low scores remain difficult to interpret when the event does not occur: failure to complete the task does not by itself establish that the corresponding physical process is implausible. The prevalence of such cases and their impact on criterion-level rankings remain unclear, limiting the attribution of scores to specific physical capabilities.

  2. Independent diagnosis remains insufficiently robust for some criteria. In Table 21, the split-half model-ranking correlations for fluid continuity, displacement, and boundary interaction are 0.38, 0.49, and 0.50, respectively; Table 19 also merges several criteria with limited sample support. These results limit some of the fine-grained conclusions. The paper should clarify which criterion-level differences between models are sufficiently supported.

Questions for authors
  1. Please clarify the scoring policy when the specified event does not occur, report the proportion of such cases, and examine whether the main criterion-level conclusions hold when results are stratified by event completion.
  2. For the criteria with lower ranking stability in Table 21, which differences between models remain statistically supported, and which should be treated as exploratory findings?
  3. As an optional extension, could the benchmark be evaluated on additional recently released video generation models that support text and first-frame image inputs, where feasible, to further assess its applicability (such as MiniMax-H3)?
5 comments

We thank the reviewer for the careful reading and for recognizing both the per-law breakdown and the design of the human study.

W1 + Q1

The criterion scores may conflate event completion with physical plausibility. [...] failure to complete the task does not by itself establish that the corresponding physical process is implausible. The prevalence of such cases and their impact on criterion-level rankings remain unclear [...]

Please clarify the scoring policy when the specified event does not occur, report the proportion of such cases, and examine whether the main criterion-level conclusions hold when results are stratified by event completion.

A1: Thank you for separating these two failure modes so clearly. The paper's conclusions hold once every video with per-video Semantic Alignment (SA, defined below) under 3 is removed, which removes every video with SA ≤ 2: on the remaining videos, the Physics ranking of the eight generators matches the full set at Spearman 0.95, and all four per-domain contrasts of Table 23 (Wan2.2-27B-A14B against Veo-3.1 on Solid-Body and Fluid, Cosmos-2B against LTX-2-19B on Optical and Fluid) keep their sign.

Scoring policy. A law whose event does not happen receives 1. The training module shown to annotators before rating (Figure 5) sets this rule by example. In its baseball example, where the bat never touches the ball, Collision and Momentum both receive 1, with the reason "No real collision occurs — the bat never makes clear contact with the ball, so there is no physically plausible impact response." In its tire example, where the tire never crushes the bottle, Collision also receives 1. The rating page had no not-applicable option; every law was rated on the same 1 to 5 scale.

Prevalence. We measured how often the stated event is missing with a new analysis of the paper's human ratings. Semantic Alignment (SA) is rated against the full prompt, which states the outcome, so per-video SA, the mean over annotators, grades event completion: at 2 or below the event is not realized, at 4 or above it is clearly realized, and in between it is partly realized. By this measure, the event is not realized in 699 of the 2,000 videos (35.0%), from 17.2% (Wan2.2-27B-A14B) to 49.6% (Cosmos-2B).

Stratified results. To test whether the per-domain and per-law conclusions rest on videos in which the event is missing, we recomputed the leaderboard and the generator-by-law profile on the SA ≥ 3 videos only. Agreement with the full set is Spearman for the rankings of the eight generators and Pearson for the 8 × 13 generator-by-law profile:

SA ≥ 3 vs. full setOverallPhysicsSolid-BodyFluidOpticalGenerator-by-law profile
Agreement0.900.950.930.810.950.84

The four contrasts keep their sign at SA ≥ 3 and also at the stricter SA ≥ 4 (356 videos). The per-domain and per-law conclusions therefore do not rest on videos in which the event is missing. We have added the scoring policy, the prevalence and the stratified results to the revised PDF (Appendix O.2).

W2 + Q2

In Table 21, the split-half model-ranking correlations for fluid continuity, displacement, and boundary interaction are 0.38, 0.49, and 0.50, respectively [...] The paper should clarify which criterion-level differences between models are sufficiently supported.

For the criteria with lower ranking stability in Table 21, which differences between models remain statistically supported, and which should be treated as exploratory findings?

A2: Thank you for your insightful comment! Per your comment, we have now clarified those arguments.

First, why these laws are worth measuring separately. A fluid scene can show plausible flow while liquid vanishes or appears from nowhere, which a single fluid score would hide. PhyGround therefore scores fluid continuity, displacement and boundary interaction as separate laws, which no prior benchmark we know of does (VideoPhy-2's rule laws include Conservation of Mass and Buoyancy, WorldModelBench has one fluid check, PhyGenBench's mechanics laws include none of the three, VideoPhy groups prompts by solid–fluid interaction without per-law scores, and Physics-IQ has one fluid-dynamics category). To our knowledge, it is also the first physics benchmark for video generation to report how reliably humans rank generators on each law (Table 21); this per-law measurement is what lets us answer the reviewer's question with numbers. Table 19 compares generators on these laws through two merged columns, Conserv. and Liq-Solid, whose human ranking reliability reaches 0.72 and 0.82 (table below).

We projected split-half ranking reliability (Table 21 for individual laws; a new analysis of the same ratings for merged columns) to the full annotator pool with the Spearman–Brown formula; this is the human standard. Against it we set the Spearman agreement with the full-pool human ranking of VideoPhy-2-AutoEval, the WorldModelBench judge, and PhyJudge-9B trained without the scored generator's model family. Liq-Solid merges boundary interaction with buoyancy, and Conserv. merges displacement with fluid continuity. In each column, bold marks the evaluator closest to humans and underline the second closest.

Ranking agreementFluid continuityDisplacementBoundary interactionConserv.Liq-Solid
Human standard (full pool)0.550.660.670.720.82
VideoPhy-2-AutoEval vs. human0.17‾\underline{0.17}0.51−0.480.25‾\underline{0.25}0.19‾\underline{0.19}
WorldModelBench judge vs. human−0.48−0.08−0.26‾\underline{-0.26}−0.33−0.20
PhyJudge-9B vs. human0.510.30‾\underline{0.30}0.460.570.38

No automatic evaluator reaches the human standard on any of the five, so these laws remain an open problem for automatic evaluation, and PhyGround's human ratings are the reference for them. PhyJudge-9B is closer to humans than the WorldModelBench judge on all five and than VideoPhy-2-AutoEval on four.

A2 (continued): Second, which differences are supported. In each column of Table 19 we tested all pairs of the eight generators with a bootstrap over each generator's (video, law) scores averaged across annotators (2,000,000 resamples) and Holm correction within the column (α = 0.05). For the three laws in question, the table gives what the paper reads from the corresponding column and the differences that are statistically supported (1 to 5 scale).

Laws in questionColumn of Table 19Paper's readingStatistically supported differences
Boundary interactionLiq-SolidVeo-3.1 leads decisivelyVeo-3.1 above all seven other generators (by 0.55 to 1.08); on boundary interaction alone, above six of them (0.68 to 1.11)
Displacement, fluid continuityConserv.Veo-3.1 and OmniWeaving near-tied at the topVeo-3.1 above Wan2.2-TI2V-5B (0.74) and Cosmos-2B (0.65); OmniWeaving above Wan2.2-TI2V-5B (0.61); Veo-3.1 vs. OmniWeaving 0.13 [−0.21, 0.47], no detectable difference, consistent with the stated near-tie

Both readings agree with these tests. Displacement and fluid continuity are compared through the Conserv. column, which is the only level at which the paper compares generators on them. We have added the counts of supported pairs for every column of Table 19 to the revised PDF (Appendix O.3).

Q3

As an optional extension, could the benchmark be evaluated on additional recently released video generation models that support text and first-frame image inputs [...] (such as MiniMax-H3)?

A3: Thank you for suggesting this extension! Inspired by your comment, we have now evaluated MiniMax-H3 and Cosmos3-Nano on the complete set of 250 prompts and corresponding first-frame images, producing 500 new videos under the original generation protocol.

To support a large-scale assessment, we recruited 190 new annotators and collected ratings for every video, with at least three annotators per video. We followed the original study’s human-evaluation design, grounded in social science laboratory experiment principles, including the same rubric, training examples, randomized assignment and presentation order, model blinding, and quality-control pipeline.

Additionally, we tested the unchanged PhyJudge-9B against these independent human ratings. Neither generator appeared in its training data. Its per-video Overall MAE is 0.72 on MiniMax-H3 and 0.79 on Cosmos3-Nano, compared with 0.79 on held-out Veo-3.1. These results extend the benchmark’s evaluation to two recent generators and provide human validation of judge transfer. Full results are reported in Appendix P.2.

We hope these analyses answer the reviewer's questions and are glad to provide further detail.

Reviewer_oS8rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to evaluate physical adherence in generated videos through explicit physical criteria, human annotations, and a specialized VLM judge. I find the topic interesting, the benchmark design solid, and the experimental evaluation detailed and thorough. While the contribution on theoretical understanding is relatively limited, I see the contribution of this work comes from well-executed engineering and I don't have major concerns on this aspect.

Critical strengths
  1. The paper addresses a timely and important problem evaluation of physical plausibility in generated videos.

  2. The evaluation of individual physical rules provides useful diagnostic information beyond an overall score. The concrete sub-questions make the intended evaluation criteria easier to inspect.

  3. The substantial human annotation effort, including annotator training, quality control, and reliability analyses, is a strength of the benchmark.

Critical weaknesses

I don't have experiment wise significcant critical weakness. Below is some minor issue that I think worth flagging:

  1. Section 3.5 states that only the LoRA parameters are trainable, while Table 24 lists the vision–language aligner as trainable with a separate learning rate. Please reconcile these descriptions and specify which parameters are updated.

  2. the labels in Figure 2 are too small to read comfortably at normal viewing size. Enlarging the figure or using a simpler visualization would improve readability.

Questions for authors

Could you clarify whether the vision–language aligner is updated alongside the LoRA parameters, given the different descriptions in Section 3.5 and Table 24?

4 comments

Thank you very much for your recognition! We sincerely appreciate your positive assessment of our work and the efforts you spent to help improve it. We have taken your comments to heart and addressed them accordingly. We look forward to your further thoughts!

W1 + Q1

Section 3.5 states that only the LoRA parameters are trainable, while Table 24 lists the vision–language aligner as trainable with a separate learning rate. Please reconcile these descriptions and specify which parameters are updated.

Could you clarify whether the vision–language aligner is updated alongside the LoRA parameters [...]?

A1: Thank you for your keen observation. We appreciate your careful reading. To clarify, the aligner's own weights stay frozen; LoRA adapters on its linear layers are trained together with those on the language model. LoRA adapters on the language model use a learning rate of 1e-4, and LoRA adapters on the aligner use the 2e-6 listed in Table 24. The vision encoder and all base weights stay frozen, so the released checkpoint contains only the LoRA adapters.

Per your comment, we have reconciled the two descriptions. In Section 3.5, we clarify that “only the LoRA parameters θ are trainable: LoRA adapters are attached to all linear layers of the language model and of the vision-language aligner, while the vision encoder and all base weights stay frozen (Table 24).” Besides, Table 24 now describes the aligner row as LoRA adapters with their learning rate, instead of “trainable”, so the two places use the same wording.

In addition, we have checked the rest of the manuscript, including the training details in Appendix H, to make sure the trainable parameters are described the same way.

W2

the labels in Figure 2 are too small to read comfortably at normal viewing size. Enlarging the figure or using a simpler visualization would improve readability.

A2: Thank you for your careful observation! Inspired by your comment, we have made Figure 2 easy to read and understand. We have redrawn Figure 2 to show only what Section 3.1 uses it for: the range of physical events across the 250 prompts. The inner ring groups the prompts into five classes of physical events (rigid-body motion, gravity and contact, fluids, optics, and tool use), each with the number of prompts that use one of its verbs. The outer ring shows the 20 most frequent root verbs, one slice per verb, with each label inside its own slice; “…” marks the less frequent verbs of a class. We removed the ring of direct nouns, which held most of the small labels, and list the nouns in a table instead (Table 38, Appendix O.1), so no information is lost. In print, the smallest label is now 7.5 pt. The caption and the sentence in Section 3.1 that cites the figure have been updated to match. We hope you enjoy reading the new figure.

Thank you once again for your constructive suggestions and for helping us improve the quality of our paper. We are grateful for your efforts and look forward to hearing from you!

AuthorsOpenReview ↗

Thank You for Supporting the Acceptance of Our Paper

Thank you so much for your positive rating in support of accepting our paper, and for your insightful suggestions! We are especially grateful that you found the evaluation of physical plausibility in generated videos a timely and important problem, appreciated the useful diagnostic information that our evaluation of individual physical rules provides beyond an overall score, with concrete sub-questions that make the criteria easier to inspect, and considered our substantial human annotation effort, including annotator training, quality control, and reliability analyses, a strength of the benchmark.

We are fully committed to resolving all of your concerns. Could you kindly let us know whether you have any remaining concerns? We would be glad to address them.

Thank you again for your time and valuable feedback!

Thank you for the response and clarifications. My concerns have been addressed, and I remain supportive of acceptance.

AuthorsOpenReview ↗

Thank You for Confirming Acceptance and for Your Professional and Insightful Suggestions

Dear Reviewer,

Thank you very much for your prompt response, for confirming that our responses have addressed your concerns, and for your continued support for acceptance! We are especially grateful that you found the benchmark design solid and the experimental evaluation detailed and thorough.

We sincerely appreciate your accurate and professional judgment, as well as your careful and insightful suggestions throughout the review and discussion process. Your feedback on the description of the trainable parameters and on the readability of Figure 2 has made our paper clearer and more precise. Thank you once again for your time and dedicated service!

Reviewer_JpsnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new benchmark Phyground, intended to diagnose physical failures in generative video models at finer granularity than a single physics score and previous work. Phyground contains 250 curated text-plus-first-frame prompts and assigns each prompt two to four applicable laws drawn from a 13-law taxonomy spanning solid-body mechanics, fluid dynamics to optics. The benchmark scores three general dimensions: semantic alignment, physical temporal validity, and object persistence. Each relevant law is evaluated on a five-point likert scale.

Critical strengths

S1) The general idea of the paper with finer grained failures of gen video world models is extremely relevant for the community. S2) The amount of human annotations is impressive and provides a good indication of the measurements' human alignment, that often lacks in other work.

Critical weaknesses

W1) Limited evidence for extensibility without human annotations. The main model comparison in Table 2 relies on human-annotated videos. While the authors release PhyJudge-9B to enable automatic evaluation, it remains unclear how reliably the benchmark can be extended to new video generation models or new scenarios without collecting additional human annotations.

W2) PhyJudge-9B is fine-tuned using human annotations of videos generated by seven of the eight evaluated models, with Veo-3.1 held out for evaluation. Given that many of the ranked models are not current SoTA (from a fall 2026 lense), this depicts a clear gap since newer models in 2026 (e.g. Cosmos 3 or MiniMax-H3 and its variants) do make different types of mistakes than older models in modeling physics leading to a potential systematic bias.

W3) The main goal was to measure physical reasoning and is even included in the paper title. However the scenario prompts are enriched with the expected outcome (see A.1.4). I respectfully disagree here: "Many original prompts describe only the initial setup without stating the expected physical result. For instance, “The grabber tools let go of the ball and block” does not mention that the objects should fall. Such prompts let the model avoid the physical event entirely, making evaluation vacuous. A model should be able to physically reason about the world and determine that an object falls, if nothing holds the object in its place. With the foreshadowing of the physical outcome, this is a prompt following benchmark.

W4) No evaluation of PhyJudge-9B on unseen scenarios. The authors evaluate PhyJudge-9B on videos from the held-out generator Veo-3.1, which is an appropriate test of cross-generator generalization. However, the underlying 250 scenarios and their augmented prompts are also represented in training through videos generated by the other seven models. Consequently, the reported evaluation does not establish generalization to unseen physical scenarios.

W5) Uncorrected errors in included and prompt extended scenarios. Since the initial arXiv submission of PhyGround, Physics-IQ Verified has been released as an updated version of Physics-IQ that corrects numerous errors in the original dataset. However, my inspection of the released PhyGround dataset on Hugging Face revealed that at least some of these errors remain, despite the manual filtering and verification procedure described in Appendix A.1.3. For example, PhyGround scenario ID 153 (0179_perspective-center_trimmed-two-balls-pass, “Two balls pass”) describes blue and yellow balls colliding and bouncing apart, reproducing the description from the original Physics-IQ dataset. Physics-IQ Verified explicitly flags this description as “Factually Incorrect” prominently in their paper and corrects the ball colors to grey and brown, as shown in the conditioning image. This example raises concerns about the effectiveness of the dataset verification pipeline.

Questions for authors

Q1) To address W2, the current evaluation should be extended with at least 2 other more recent models to assess how much PhyJudge-9B scorings agree with human ratings.

Q2) Table 1 highlights the autors' method (phyground) and not the highest value per column in bold font. This should be adressed by the authors.

12 comments

Thank you very much for your careful review! Inspired by your comments, we have run three new experiments and several new analyses, and we believe the paper is now stronger. Below, we answer each point, giving the answer first and then the evidence. We have added the new results to the revised manuscript (Appendices P.2 and Q).

Conventions. Brackets give 95% intervals: bootstrap intervals resampled over prompts, except for the odds ratio, whose interval comes from the mixed-effects model.

W1

Limited evidence for extensibility without human annotations. The main model comparison in Table 2 relies on human-annotated videos. While the authors release PhyJudge-9B to enable automatic evaluation, it remains unclear how reliably the benchmark can be extended to new video generation models or new scenarios without collecting additional human annotations.

A1: Thank you for raising this question. To clarify, yes for ranking: PhyJudge-9B extends the benchmark to new generators and new scenarios without collecting additional human annotations. In Appendix J, we have already tested the two cases you raise and their combination: a generator the judge never saw, prompts it never saw, and both at once (below).

Two 2026 generators, with new human ratings. MiniMax-H3 and Cosmos3-Nano each generated videos for all 250 PhyGround prompts under the same generation protocol as the paper’s generators. We collected human ratings for all 250 videos per model, from 190 newly recruited annotators with at least three per video, under the paper’s annotation protocol (same rubric, training examples and quality control). PhyJudge-9B never saw either model. Its per-video error on both stays at the level of the held-out Veo-3.1 in the paper (Appendix P.2). We report the full results in A2.

A generator the judge has never seen (J.4, Table 30). The paper runs the same test on the original eight generators. We retrained the judge eight times, each time leaving one generator out, and then scored that generator’s 250 videos. Mean absolute bias was 0.37, compared with 0.50 for a judge trained on all eight, and calibration was equal or better in 7 of 8 folds.

Prompts never seen in training (J.6, Table 33). You also asked about new scenarios. Here we retrain the judge with the PhyJudge-9B recipe, exclude the prompts of 175 videos from every training fold, and score those videos. The retrained judge’s bias is +0.07, while the untuned backbone’s is +2.21; when we also remove the physical laws being scored from training, it is +0.11.

Everything new at once (J.5, Table 31). On the VideoPhy-2 test set, with every test video whose caption matches one of our prompts removed, the generators, prompts, videos and labeling criteria are all external. Without any change, PhyJudge-9B ranks VideoPhy-2’s seven generators at Spearman (×100) 71.4, while its untuned backbone is at −10.7.

We hope our response has addressed your concern.

W2 + Q1

W2) PhyJudge-9B is fine-tuned using human annotations of videos generated by seven of the eight evaluated models, with Veo-3.1 held out for evaluation. Given that many of the ranked models are not current SoTA (from a fall 2026 lense), this depicts a clear gap since newer models in 2026 (e.g. Cosmos 3 or MiniMax-H3 and its variants) do make different types of mistakes than older models in modeling physics leading to a potential systematic bias.

Q1) To address W2, the current evaluation should be extended with at least 2 other more recent models to assess how much PhyJudge-9B scorings agree with human ratings.

A2: Thank you for this timely point! Per your comment, we have now included two major recent models, namely Cosmos3-Nano as Cosmos 3’s variant [1] and MiniMax-H3 [2]. We first used each model to produce 250 videos under our original prompt set that includes 250 prompts, under the same generation protocol as our previous models. Then, we leveraged PhyJudge-9B to score each video generated by those two new models. We added both models to PhyGround's evaluation (Appendix P.2); PhyJudge-9B never saw either model in training.

Human ratings (new). For both new generators, 190 newly recruited annotators rated all 250 videos per model, at least three per video, on the paper's platform with its rubric, training examples and quality-control pipeline. At 4 fps, the paper’s validation setting, PhyJudge-9B’s Overall bias against the new annotators is −0.32 [−0.42, −0.22] on MiniMax-H3 and +0.58 [+0.48, +0.68] on Cosmos3-Nano. Per-video agreement, next to held-out Veo-3.1 in the paper:

Per videoMiniMax-H3Cosmos3-NanoHeld-out Veo-3.1
MAE0.720.790.79
Within ±1 (%)74.870.468.8

Whole generator families the judge never saw (new experiment). To test whether an unseen family is scored with a systematic offset, we split the eight generators by family and trained two judges with the PhyJudge-9B recipe, each on the videos of four generators, then scored the other four generators' 1,000 videos: one judge never saw Veo, OmniWeaving or LTX, the other never saw Wan or Cosmos. For each held-out generator we compare its Physics score from the judge with the one from the paper's human ratings, as a relative bias, (judge − human) / human.

Held-out familiesHeld-out generatorsMean absolute relative Physics bias
Veo, OmniWeaving, LTX46.7%
Wan, Cosmos49.5%
All eight88.1%

We have added both models to the revised manuscript: Appendix P.2 compares PhyJudge-9B with the new human ratings on them.

References

[1] NVIDIA. Cosmos 3: Omnimodal World Models for Physical AI. https://arxiv.org/abs/2606.02800, 2026.

[2] MiniMax. MiniMax H3: An Open Model Breaking the Boundaries Between Tasks and Modalities. https://www.minimax.io/blog/minimax-h3, 2026.

W3

The main goal was to measure physical reasoning and is even included in the paper title. However the scenario prompts are enriched with the expected outcome (see A.1.4). I respectfully disagree here: "Many original prompts describe only the initial setup without stating the expected physical result. For instance, “The grabber tools let go of the ball and block” does not mention that the objects should fall. Such prompts let the model avoid the physical event entirely, making evaluation vacuous. A model should be able to physically reason about the world and determine that an object falls, if nothing holds the object in its place. With the foreshadowing of the physical outcome, this is a prompt following benchmark.

A3: Thank you for your important comment! To clarify, realizing the requested event still requires physically consistent dynamics, which our 13 law-specific criteria assess. A video can depict a glass falling and eventually shattering while also showing it hovering or breaking before contact. The requested outcome alone does not establish physical consistency (Appendix L).

To test whether law scores capture failures beyond semantic alignment, we analyzed the paper’s human ratings. Among 356 videos with high semantic alignment (mean Semantic Alignment (SA) rating ≥ 4), 26.4% still score below 3 on at least one physical law. Thus, strong alignment with the requested event can coexist with physical failures identified by PhyGround.

Besides, stating the outcome also establishes a common target for evaluation. Without it, a glass that lands intact can be read as a rigid-body violation or as a sturdy glass (Appendix L); stating "the glass shatters" removes this ambiguity about unobserved materials while still requiring coherent dynamics. Therefore, PhyGround measures physical reasoning conditioned on a specified event, which prompt following alone does not satisfy.

We also ran this comparison directly (new experiment, Appendix Q). For 50 prompts we wrote a version with the same length and first frame that does not state the outcome. We first checked on the text alone that the two versions differ in the stated outcome and not in wording quality: in an IRB-approved between-subjects study, 35 participants judged whether each prompt directly states a physical result and said yes for 82.6% of the original and 29.4% of the new versions (p < .001), while rated naturalness and clarity barely differed (p = 0.66 and 0.68). We then generated both versions with nine generators (the paper's seven open generators plus MiniMax-H3 and Cosmos3-Nano from Q1) and collected new human ratings for all 900 videos against the original prompt and laws. In this matched study where the outcome is not stated, annotator law scores drop by 0.19 [0.07, 0.31], but the ranking of the nine generators is nearly unchanged (Spearman 0.93 on law scores). Physics with the stated outcome also predicts outcome inference without it: in a mixed-effects logistic regression with a random intercept per scenario, each additional point of a generator's mean law score with the stated outcome multiplies the odds that it realizes the outcome when it is not stated (at least half of the raters say it occurs) by 2.3 [1.8, 2.8].

W4

No evaluation of PhyJudge-9B on unseen scenarios. The authors evaluate PhyJudge-9B on videos from the held-out generator Veo-3.1, which is an appropriate test of cross-generator generalization. However, the underlying 250 scenarios and their augmented prompts are also represented in training through videos generated by the other seven models. Consequently, the reported evaluation does not establish generalization to unseen physical scenarios.

A4: Thank you for your careful thought! We agree that the Veo-3.1 split tests cross-generator generalization. However, to clarify, our paper has already included separate evaluations on unseen prompts and external scenarios.

Unseen prompts (Appendix J.6, Table 33). We evaluate judges trained using the PhyJudge-9B recipe on 175 videos (601 video–law samples) whose prompts appear in no training fold. On these, the recipe achieves MAE 0.85 and 82.9% within-±1 agreement, compared with 2.35 and 22.0% for the zero-shot backbone. Excluding the evaluated law group from training also yields MAE 0.85.

External scenarios (Appendix J.5, Table 31). The unchanged PhyJudge-9B is additionally evaluated on VideoPhy-2. After removing videos whose captions match PhyGround prompts, its per-video MAE is 1.05, compared with 1.33 for the backbone. This tests transfer to external prompts, videos, generators, and human labeling criteria.

On both tests, the judge stays closer to the human scores than the zero-shot backbone on scenarios it never saw in training.

Reviewer_Tj1AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark to evaluate physics correctness in generated videos and provide a finetuned vlm as judge for evaluation. The benchmark itself does not seem to be very novel since there are multiple existing benchmarks for evaluating physics, and some of them also provide a VLM judge. But the size and coverage of this benchmark are larger, and the measurement and evaluation criteria are well-designed.

Critical strengths
  1. The proposed benchmark and VLM judge provide a useful evaluation protocol design and open evaluation infrastructure. The prompt annotation effort is substantial and can be useful for future research in related directions.
  2. The dataset, evaluation scripts and VLM evaluator are publicly available, which could be useful for the community
Critical weaknesses
  1. The existing benchmark capabilities are understated. Section 2 says existing open evaluators are trained on holistic single scores and cannot provide per-law diagnostics, but VideoPhy-2 already collects 1-5 semantic and physical-commonsense scores, annotations of individual physical rules, and trains its evaluator on both scoring and rule classification
  2. The physical outcomes are specified in the text and cannot test physical reasoning where the model needs to infer the outcomes. It would make the benchmark more informative if it evaluated both explicit and physical reasoning and studied their relationship
Questions for authors

Could the trained VLM judge help improve physics correctness as a learned reward?

6 comments

Thank you very much for your constructive comments! We are grateful for your efforts in helping us improve our work. Inspired by your comments, we have made our comparison with existing benchmarks more specific. More importantly, we ran a new controlled experiment with both explicit and implicit versions of 50 prompts: nine generators produced 900 videos, and 229 participants took part across an IRB-approved construct validation of the prompts and a blinded human rating study, following experimental design standards in the social sciences. We sincerely hope our responses address your concerns and look forward to your further thoughts.

W1

The existing benchmark capabilities are understated. Section 2 says existing open evaluators are trained on holistic single scores and cannot provide per-law diagnostics, but VideoPhy-2 already collects 1-5 semantic and physical-commonsense scores, annotations of individual physical rules, and trains its evaluator on both scoring and rule classification.

A1: Thank you for raising this important point. Per your comment, the revised Section 2 now describes the existing benchmarks’ capabilities in more detail, in particular VideoPhy-2’s rule annotations and its evaluator’s rule classification. It now reads: "VideoCon-Physics and the WorldModelBench judge output holistic scores or yes/no answers to fixed violation questions. VideoPhy-2-AutoEval additionally classifies individual physical rules as followed, violated, or undeterminable. Its rules are generated by Gemini from a caption of each video, so videos from different generators are checked against different rules, and a violation carries no severity grade."

The difference that remains is how rules are assigned and scored, and with it comparability across generators. VideoPhy-2 writes rules from each generated video (its Section 2, Stage 3; rule prompt in Appendix F), so in its released test set, videos of the same prompt from different generators carry different rule lists, often of different length. Its law-level analysis (Figure 5) accordingly reports one violation rate per law, pooled over all generators. In contrast, PhyGround fixes two to four laws per prompt from a 13-law taxonomy, applies them to every generator's video of that prompt, and scores each law, from 1 to 5 with observable sub-questions. All generators are rated on the same laws for the same prompt. This shared basis is what lets Tables 2 and 19 compare generators law by law and show that models with similar overall scores differ on solid-body and fluid laws. Accordingly, we have extended this sentence in the manuscript to read “First, PhyGround supports more diagnostic evaluation by combining explicit expected-outcome prompts with per-law decomposition, and applies the same laws to every generator’s video of a prompt, so generators can be compared law by law.”

Conventions. Brackets give 95% intervals: bootstrap intervals resampled over prompts, except for the odds ratio, whose interval comes from the mixed-effects model.

W2

The physical outcomes are specified in the text and cannot test physical reasoning where the model needs to infer the outcomes. It would make the benchmark more informative if it evaluated both explicit and physical reasoning and studied their relationship.

A2: Thank you for this important direction! Inspired by your comment, we ran the comparison you propose as a new controlled experiment (Appendix Q), approved by the IRB of one author’s institution. It involves 50 matched prompt pairs, nine generators, 900 videos and 229 human participants. In short, we found that stating the outcome makes the task easier but does not change how generators compare. The ranking is nearly the same in both settings, and a generator’s physics with the stated outcome predicts how often it infers the outcome on its own. PhyGround’s stated-outcome design measures an ability that carries over to outcome inference.

We briefly discuss our experiment as follows. It follows standard experimental design in the social sciences in three steps.

Step 1: Construct validation of the prompts. For a representative set of 50 prompts, we wrote a version that withholds the outcome (implicit) next to the original that states it (explicit), with the same first frame and word count. Before generating any video, we validated this manipulation on the text alone (Chester & Lasko, 2021; Hauser et al., 2018). In a between-subjects study, 35 participants (after a pre-specified straight-lining rule; 30 prompts each) judged whether each prompt directly states a physical result. They indicated yes for 82.6% of explicit and 29.4% of implicit prompts (Welch t(26.1) = 7.03, p < .001), while rated naturalness (4.17 vs. 4.08, p = 0.66) and clarity (4.40 vs. 4.34, p = 0.68) barely differed. Therefore, the explicit and implicit prompt groups differ in the stated outcome, while wording quality is nearly the same.

Step 2: Matched generation. We used nine generators (the paper’s seven open generators plus MiniMax-H3 (MiniMax, 2026) and Cosmos3-Nano (NVIDIA, 2026)) to generate videos in both versions of every prompt with the same first frame, default settings and seed, resulting in 50 × 2 × 9 = 900 videos.

Step 3: Blinded human rating. 194 raters who passed the paper’s quality control rated all 900 videos (up to 15 videos each; 868 videos rated by three or more raters) with the paper’s rubric (Semantic Alignment (SA) and a 1 to 5 score per law), against the explicit prompt in both conditions. As a manipulation check, they also answered whether the described result occurs. In a between-subjects design (Charness et al., 2012), each rater saw only one condition and was not told which condition a video came from. Each page shows one scenario with three of the nine generators, grouped so that every pair of generators shares a page equally often, with the left-to-right order rotated. All comparisons are paired within the scenario and generator.

Without − with the stated outcome (1 to 5)SAMean law score
Human ratings, all 50 prompts−0.32 [−0.46, −0.16]−0.19 [−0.31, −0.07]
Human ratings, 37 prompts with survey yes-rate ≤ 0.5−0.26 [−0.44, −0.08]−0.17 [−0.32, −0.04]
PhyJudge-9B, all 50 prompts−0.19 [−0.30, −0.09]−0.07 [−0.14, −0.01]

Findings. Our analyses reveal that withholding the outcome makes the specified result appear less often and lowers law scores. On the relationship you ask about, the two settings are distinct but closely linked. At the generator level, the nine generators rank nearly the same with and without the stated outcome (Spearman 0.95 for SA and 0.93 for the mean law score). At the video level, a mixed-effects logistic regression with a random intercept per scenario shows that a generator’s physics with the stated outcome predicts whether it infers the outcome when it is not stated: each additional point of explicit Physics score (mean law score) multiplies the odds of implicit success (at least half of the raters say the outcome occurs) by 2.3 [1.8, 2.8]. PhyJudge-9B shows the same direction on both scores.

The stated outcome also does not reduce PhyGround to prompt following: in a new analysis of the paper’s ratings, among the 356 videos that clearly deliver the stated outcome (mean SA ≥ 4), 26.4% still have at least one law rated below 3. PhyGround thus measures physical adherence given a specified event, and the new experiment shows that this complements, and predicts, performance when the outcome must be inferred.

Q1

Could the trained VLM judge help improve physics correctness as a learned reward?

A3: Thank you for your insightful question. Our answer is yes for selecting among candidate videos, which we tested directly; training a generator with it is future work.

A reward model must prefer the physically better of several candidate videos; a new best-of-4 selection experiment tests exactly this without training a generator. We split the paper's eight generators into two pools of four: pool 1 (Veo-3.1, OmniWeaving, LTX-2.3-22B, LTX-2-19B) and pool 2 (Wan2.2-27B-A14B, Wan2.2-TI2V-5B, Cosmos-14B, Cosmos-2B). For each pool we trained a held-out judge with PhyJudge-9B's recipe (LoRA supervised fine-tuning on PhyGround's per-law human scores) but without any video from that pool, so it has never seen the candidates' generators. For each of the 250 prompts, the held-out judge scores the pool's four videos and keeps the one with the highest mean law score, which we then compare on the video's 1 to 5 human Physics score (the mean of its human law scores). The released PhyJudge-9B, whose training data include human scores for every generator here except Veo-3.1, is shown for reference.

Human Physics score, judge's pick − random pickPool 1Pool 2Both pools
Held-out judges+0.16+0.08+0.12 [0.06, 0.18]
Released PhyJudge-9B+0.21+0.20+0.21 [0.15, 0.26]

The held-out judges pick videos that humans rate higher than a random pick in each pool, showing that the recipe's scores carry a preference for physical correctness that transfers to generators the judge never trained on, which is the property a reward model needs.

This makes three uses of PhyJudge-9B as a reward natural next steps: best-of-N or rejection sampling at inference time, which the experiment above emulates; preference pairs ranked by PhyJudge-9B for Direct Preference Optimization (DPO) of a video generator; and reinforcement-learning fine-tuning (e.g., GRPO-style policy optimization) with PhyJudge-9B's score as the reward. Because PhyJudge-9B scores each law separately, it can also give per-law rewards that target a specific failure such as collision or fluid continuity.

Thank you again for all your comments!

References

Charness, G., Gneezy, U., & Kuhn, M. A. (2012). Experimental methods: Between-subject and within-subject design. Journal of Economic Behavior & Organization, 81(1), 1–8.

Chester, D. S., & Lasko, E. N. (2021). Construct validation of experimental manipulations in social psychology: Current practices and recommendations for the future. Perspectives on Psychological Science, 16(2), 377–395.

Hauser, D. J., Ellsworth, P. C., & Gonzalez, R. (2018). Are manipulation checks necessary? Frontiers in Psychology, 9, 998.

MiniMax. (2026). MiniMax H3: An open model breaking the boundaries between tasks and modalities. https://www.minimax.io/blog/minimax-h3

NVIDIA. (2026). Cosmos 3: Omnimodal world models for physical AI. arXiv:2606.02800.

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your thoughtful reviews and constructive suggestions, which have helped us improve our paper!

We have tried our best to address each of your concerns in our point-by-point responses. If you have any remaining concerns or further questions, please let us know. We are fully committed to resolving all of your concerns.

Thank you again for your time and valuable feedback!

Best regards, The Authors

AuthorsOpenReview ↗

Thank You to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

As the discussion period comes to a close, we would like to express our sincere gratitude to you all for your valuable time, excellent guidance, and constructive feedback!

The discussion has significantly helped us improve the clarity and rigor of our work. We have addressed all the concerns with our best efforts. In summary:

  • Physics supervision versus capacity and backbone. The PhyJudge recipe brings Overall aggregate bias below 10% at every size of one VLM family (2B to 27B), and it also lowers the bias of MiMo-VL-7B, a backbone from outside that family (Appendix P.1).
  • Comparison with stronger judges. With the identical prompt and frames, PhyJudge-9B has the lowest bias in every column against gpt-6-astra and gpt-5.5. On generator families its judge never saw in training, the PhyJudge recipe has significantly lower mean absolute relative Physics bias than VideoPhy-2-AutoEval.
  • Stated versus inferred outcomes. In a new controlled experiment with 900 videos and 229 participants (Appendix Q), the nine generators rank nearly the same with and without the stated outcome (Spearman 0.93 on the mean law score).
  • Event completion versus physical plausibility. Rankings adjusted for whether the stated result occurs correlate with the original rankings at Spearman 0.97 for explicit prompts and 0.82 for implicit prompts.

We are pleased to have developed PhyGround, a fine-grained benchmark that decomposes physical plausibility into individual physical laws and observable criteria, grounded in a large-scale, quality-controlled human study, together with PhyJudge-9B, an open judge that makes this evaluation reproducible and scalable.

We hope our responses have clarified the remaining questions. Thank you again for your hard work, and we are devoted to further contributing to the video generation community!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.