- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PhyGround, a criteria-grounded benchmark for evaluating physical reasoning in video generation, with a large-scale human study and an open physics-specialized VLM judge, PhyJudge-9B.
Critical strengths
-
The proposed PhyGround benchmark provides a more fine-grained evaluation framework by decomposing physical reasoning into multiple physical laws and observable criteria, enabling more detailed diagnosis of model capabilities.
-
The large-scale human study, quality control process, and release of PhyJudge-9B improve the reproducibility and scalability of physics-aware video generation evaluation.
Critical weaknesses
1、The effectiveness of PhyJudge-9B would be more convincingly demonstrated with a more comprehensive scaling analysis. Currently, the authors compare PhyJudge-9B with several general-purpose VLMs of different scales, but do not investigate the impact of model size under the same physics-specific training setting. It remains unclear whether the performance improvement mainly comes from the proposed physics-aware supervision or simply from the model capacity. Experiments with different backbone sizes (e.g., smaller and larger variants of the same VLM family) after the same fine-tuning procedure would help better understand the contribution of the proposed training strategy.
2、The authors compare PhyJudge-9B mainly with general-purpose VLMs. Since existing open-source physics-oriented evaluators, such as VideoPhy-2-AutoEval and the WorldModelBench judge, are available, comparisons with these models would provide a more comprehensive evaluation and better position the contribution of PhyJudge-9B.
3、The generality of the proposed physics supervision is not sufficiently evaluated. Currently, PhyJudge-9B is built on a single VLM backbone, making it unclear whether the improvement comes from the proposed physics-specific data or from backbone-specific adaptation. The authors are encouraged to fine-tune multiple open-source VLMs using the proposed annotations and compare their improvements, which would better demonstrate the scalability and transferability of the proposed physics supervision.
4、The comparison with closed-source VLM judges is somewhat limited. Since the paper evaluates Claude and Gemini as general-purpose VLM baselines, including additional strong VLM baselines such as GPT-based models would provide a more comprehensive comparison and better position the performance of PhyJudge-9B.
Questions for authors
see Weaknesses
5 comments
W3
PhyJudge-9B is built on a single VLM backbone, making it unclear whether the improvement comes from the proposed physics-specific data or from backbone-specific adaptation. The authors are encouraged to fine-tune multiple open-source VLMs using the proposed annotations and compare their improvements [...]
A3: Thank you for asking whether the supervision transfers across backbones. The physics supervision also transfers to a backbone from outside the Qwen family. We applied the PhyJudge recipe of A1 to MiMo-VL-7B (Xiaomi LLM-Core Team, 2025), whose language model (Xiaomi's MiMo-7B) comes from outside the Qwen family, and its aggregate bias falls from 24.4% zero-shot to 15.3%, with every answer in the required score format.
Counting the four Qwen3.5 sizes in A1, the PhyJudge recipe lowers the bias of all five backbones reported here. We have added the MiMo-VL-7B row to the same table (Appendix P.1).
W4
Since the paper evaluates Claude and Gemini as general-purpose VLM baselines, including additional strong VLM baselines such as GPT-based models would provide a more comprehensive comparison [...]
A4: Thank you for suggesting a GPT baseline; PhyJudge-9B is far better calibrated than both GPT versions we tested, gpt-6-astra and gpt-5.5. Following the suggestion, we ran both as closed judges on the 250 held-out Veo-3.1 videos, with the identical prompt and frames that PhyJudge-9B receives. The two closed judges of Table 5 are shown for reference.
| Relative bias (%), held-out Veo-3.1 | General | Physics | Overall |
|---|---|---|---|
| PhyJudge-9B | 1.0 | 5.5 | 3.3 |
| gpt-6-astra | 23.4 | 34.7 | 29.0 |
| gpt-5.5 | 23.2 | 34.7 | 29.0 |
| Claude Opus 4.7 (Table 5) | 16.2 | 10.0 | 13.1 |
| Gemini-3.1-Pro (Table 5) | 17.3 | 15.9 | 16.6 |
PhyJudge-9B has the lowest bias in every column, significantly below both GPT versions.
We hope these four experiments address the reviewer's concerns.
References
Xiaomi LLM-Core Team. (2025). MiMo-VL technical report. arXiv:2506.03569.
Thank You for Your Insightful Suggestions; We Hope to Address All of Them Before the Discussion Closes in 2 Days. Could You Kindly Let Us Know If Any Concerns Remain?
Dear Reviewer t55V,
Thank you so much for your insightful suggestions! In our responses, we have addressed each of your concerns:
- Physics supervision versus model capacity (W1). We applied the PhyJudge recipe to four sizes of one VLM family (Qwen3.5-2B to -27B). It brings aggregate bias below 10% at every size, far below the zero-shot bias of every size, so the improvement comes from the physics supervision rather than from capacity (Appendix P.1).
- Comparison with open physics evaluators (W2). Used as PhyGround's Physics scorer on generator families its judge never saw in training, the PhyJudge recipe tracks human ratings significantly more closely than VideoPhy-2-AutoEval. PhyJudge-9B is also the only one of the three evaluators that returns PhyGround's per-law 1 to 5 scores.
- Transfer across backbones (W3). The same recipe also lowers the bias of MiMo-VL-7B, whose language model comes from outside the Qwen family, so the supervision is not tied to one backbone (Appendix P.1).
- GPT-based judges (W4). We added gpt-6-astra and gpt-5.5 as closed judges with the identical prompt and frames. PhyJudge-9B has the lowest bias in every column, significantly below both GPT versions.
Could you kindly let us know whether our responses have addressed your concerns? As the discussion period closes on November 18, we would be grateful to hear from you before then, so that we still have time to address any remaining concern.
Thank you again for your time and valuable feedback!
Thank you very much for your constructive comments! We are grateful for your four concrete suggestions, and we have run all four as new experiments. Below, we answer each point in turn. Unless stated otherwise, every number below is on the paper's 250 held-out Veo-3.1 videos. We report aggregate relative bias as defined in Table 5, whose Overall column averages the General and Physics columns (lower is better; "bias" for short below). Our judges and GPT use the paper's judge prompt, and the open evaluators use their official prompts.
Conventions. Significant means that the 95% bootstrap interval of a difference, resampled over prompts, excludes zero.
W1
A1: Thank you for this well-posed question. To separate the physics supervision from model capacity, we ran the size ablation you suggest as a new experiment. We took four sizes of one VLM family, Qwen3.5-2B, -4B, -9B (the backbone of PhyJudge-9B) and -27B, and scored the 250 held-out Veo-3.1 videos with each size in two settings. Zero-shot is the released model with no fine-tuning, given the same judge prompt and frames as PhyJudge-9B. PhyJudge recipe is the training procedure of PhyJudge-9B, applied to each size: LoRA supervised fine-tuning (rank 32) for one epoch on the human scores (general dimensions and physical laws) of the seven non-Veo generators, with the hyperparameters of Table 24. The improvement comes from the physics supervision: the PhyJudge recipe brings aggregate bias below 10% at all four Qwen3.5 sizes, far below the zero-shot bias of every size.
Capacity is not what decides calibration: zero-shot bias does not fall with size (the 27B starts where the 9B does, and the 4B starts lowest), while the recipe calibrates every size from 2B upward, below every closed judge of Table 5 in Overall bias (see A4). We reused the 9B hyperparameters for every size without retuning, so we do not read a size trend into the differences among the fine-tuned sizes. Both 2B entries use the 218 videos for which its zero-shot version returns a parseable answer. We have added this table to the revised PDF (Appendix P.1).
W2
A2: Thank you for this constructive suggestion. We evaluated the released VideoPhy-2-AutoEval and WorldModelBench judge checkpoints on all 2,000 PhyGround videos. We distinguish their native outputs when comparing them with our human ratings.
On PhyGround's own scale, judges trained with the PhyJudge recipe are better calibrated than VideoPhy-2-AutoEval. VideoPhy-2-AutoEval returns holistic 1 to 5 semantic-alignment and physics scores per video plus per-rule labels (followed, violated or undeterminable), and the WorldModelBench judge returns five fixed yes/no violation flags. Neither grades a per-law violation on PhyGround's 1 to 5 severity scale.
The WorldModelBench judge's output is the number of its five checks a video passes (0 to 5), with no counterpart on the 1 to 5 scale, so the calibration comparison is with VideoPhy-2-AutoEval. As PhyGround's Physics scorer, every generator is scored by one of two family-held-out judges (the PhyJudge recipe on Qwen3.5-9B, each trained without any video from the scored generator's model family). VideoPhy-2-AutoEval's holistic physics score stands in for every law of its video.
In short, used as PhyGround's Physics scorer, the PhyJudge recipe tracks human ratings significantly more closely than VideoPhy-2-AutoEval on generator families its judge never saw in training, and PhyJudge-9B is the only one of the three evaluators that returns PhyGround's per-law 1 to 5 scores.