- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PhySimCode, a benchmark for physics video-to-code generation. The dataset contains 160.6K procedurally generated samples spanning 162 physics phenomena, generated using SciPy and PyBullet. It measures three capabilities: video understanding, physics understanding, and programming abilities of MLLMs. It evaluates ten closed- and open-source MLLMs, exposing a substantial video-to-simulation gap and positioning PhySimCode as a testbed for multimodal systems that translate perception into precise, executable, and scientifically verifiable physics.
Critical strengths
- The task is new and is a useful task: generating executable code from physics videos. This could support downstream tasks, such as physics-aware symbolic simulations.
- Using SciPy and PyBullet could help to improve physical plausibility, potentially improving it compared with approaches that rely solely on VLMs or LLMs to generate videos directly.
- The analysis of existing benchmarks and the comparison in Table 1 is clear and well organized. Table 1 provides a useful comparison of their capabilities with other benchmarks.
Critical weaknesses
-
The validity of the randomly sampled physical parameters is unclear. How do the authors ensure that the sampled parameters and their combinations are physically reasonable? For example, on an inclined plane, a heavy object may slide or roll down, while a lighter object may remain stationary due to friction. Furthermore, it is unclear whether different parameter settings can produce similar motions, which could make the underlying physical parameters difficult to identify from videos.
-
The reliability of the LLM-based evaluation is unclear. The reported average Likert scores rely on LLM judges, but the paper does not clearly explain whether their judgments were validated by human experts. How do the authors ensure that the LLM judges can reliably assess the correctness of the underlying physical laws?
Questions for authors
-
Figure 2 is difficult to read. In addition, the paper mentions 192 prompts, but it is unclear how these prompts are defined or how they relate to the 162 experiment types.
-
The expertise of the human evaluators is unclear. The paper does not provide sufficient information about their backgrounds or whether they have expertise in physics, making it difficult to assess the reliability of their judgments.
-
The evaluation protocol for failed code executions is unclear. How are samples handled when the generated code fails to execute or render a video? Are these failures excluded from the evaluation or assigned a score? This could affect the reported results.
-
The use of the full dataset is unclear. Although the paper introduces 160,655 samples, only 2,430 are used for evaluation. What is the purpose of the remaining samples, and how do they contribute to the benchmark? The authors should clarify how the evaluation subset was selected and how the larger dataset supports the benchmark.
-
References for the evaluated models appear to be missing. Models such as Claude Sonnet 4.6, GPT-5-mini, Grok-4-fast-reasoning, Gemini 2.5 Pro, and Amazon Nova 2 Lite are listed, but the corresponding references are unclear. The authors should provide appropriate citations or links to the official model documentation.
-
The relationship between human and automatic evaluation remains unclear. Human evaluation provides a useful reference, but it does not by itself establish the validity of the automatic metrics. The authors should investigate how well the automatic metrics correlate with human judgments, particularly for physical plausibility and physics equivalence.
-
The scalability of the benchmark is questionable if human evaluation is required. If evaluating new models requires substantial human verification, the benchmark may be difficult to use in practice. The authors should clarify whether the automatic metrics can provide a reliable and scalable alternative to human evaluation.
23 comments
Weakness 2: reliability of LLM judges through human validation
We thank the reviewer for raising this. To address it, we have run a check on human-alignment of the LLM-judge Likert scores.
We sampled 200 inferred physical-law blocks, 20 from each of the 10 benchmarked MLLMs (10 SciPy and 10 Kubric per model). They span 117 of the 162 experiments, with no experiment repeated within a model. Human experts rated every item on the same rubric as the LLM judges, seeing exactly the inputs the judges see. Correctness and equivalence were rated in separate blind passes with independently shuffled item IDs, so the reference law could not anchor the correctness judgment. Model identity was hidden throughout.
We see that 95.5% of correctness and 86.5% of equivalence ratings from the judge panel fall within $\pm$1 of the human rating. The match fraction with human scores look like
| Judge | Correctness: exact | Correctness: ±1 | Equivalence: exact | Equivalence: ±1 |
|---|---|---|---|---|
| GPT-5-Nano | 0.675 | 0.935 | 0.370 | 0.820 |
| Gemini-2.5-Flash | 0.755 | 0.910 | 0.535 | 0.840 |
| Grok-4.1-Fast | 0.735 | 0.940 | 0.480 | 0.830 |
| Judge panel (mean) | 0.725 | 0.955 | 0.455 | 0.865 |
So, the paper's central claim is reproduced with human ratings. Humans rate physical-law correctness at 4.72 and equivalence at 1.84. The judge panel gives 4.57 and 1.99. All 10 models show a positive gap under both. Humans separate "states valid physics" from "states the right physics" even more strictly than our judges. Humans and judges also rank the 10 models similarly on equivalence (Spearman = 0.86).
We treat the human ratings as a fourth rater and report quadratic-weighted Cohen's for all rater pairs, as in Figure 4 (n = 200):
| Correctness | Human | GPT-5-Nano | Gemini-2.5-Flash |
|---|---|---|---|
| GPT-5-Nano | 0.544 | ||
| Gemini-2.5-Flash | 0.596 | 0.645 | |
| Grok-4.1-Fast | 0.553 | 0.634 | 0.708 |
| Equivalence | Human | GPT-5-Nano | Gemini-2.5-Flash |
|---|---|---|---|
| GPT-5-Nano | 0.419 | ||
| Gemini-2.5-Flash | 0.579 | 0.620 | |
| Grok-4.1-Fast | 0.593 | 0.729 | 0.760 |
Mean human-judge is 0.56 (correctness) and 0.53 (equivalence); mean inter-judge is 0.66 and 0.70.
One caveat when reading : at n = 200 it is imprecise, more so when ratings are skewed to one end. Correctness is near-saturated (82% of items rated 5 by humans), so its is depressed by high chance agreement. Equivalence ratings span the full scale, so chance agreement is much lower. There, the of the judge panel's mean against humans, 0.59, shows that judges and humans agree on which laws match the reference well beyond what their score distributions alone would produce.
Question 1: figure 2 reading difficulty
Thank you for catching this. "192 Prompts" in the Stage 1 box of Fig. 2 is a typo for 162 and we will correct the number in the figure. To make Fig. 2 easier to read, we will redraw it. Stage 1 (ideation) produces the 162 experiment prompts from the 69 expert and 93 selected LLM ideas. We will merge Stage 2 (coding) and Stage 3B (CoT generation) into a single generation stage, since the simulation code and CoT are all produced in the same inference call. Rendering the videos by executing that code with randomized parameters then becomes Stage 3. But then we do an iterative correction, which is Stage 4 (multi-round review). It will be shown as its two sub-stages: 2-10 rounds of human-only physical-plausibility checks on the rendered videos (L214-215), and the three-round LLM-assisted debiasing followed by human verification (L230-241). We will also enlarge the font and thus we hope that the idea written in the main text will be easy to visualize from the updated Fig. 2.
Weakness 1: physical plausibility of params and identifiability
Thank you so much for this valuable feedback.
The sampling ranges were first proposed by Claude Opus 4.7, prompted to keep the resulting dynamics physically plausible. They then went through the paper's debiasing pipeline (L230–238), which ends with a human verification step. As described in Section 3.1, human experts iterated over each experiment for 2-10 rounds until the behavior rendered from its range was physically plausible. The motion itself is produced by the governing equations (SciPy) or a physics engine (PyBullet), not by us. What we ensure is that each experiment's equations are correct and that the rendered dynamics violate no physical law. Different parameter combinations therefore produce different physical regimes as a consequence of the physics. On the reviewer's example: under Coulomb friction, whether an object slides depends on whether tan$\theta$ exceeds μ, not on its mass, and both outcomes are valid physics that the simulator resolves correctly. This is intentional. One experiment covers many sub-cases depending on its inputs, which is what lets the dataset scale, and it means that more samples from the same experiment add real information.
We understand the reviewer's identifiability concern: different parameters producing similar motion. One way to identify such cases could be joint perturbation study following our sensitivity analysis expt in A.6, but is intractable (discussed in Reviewer vVXZ W3). Identical dynamics from different parameters arise from coupling among parameters, including the unknown overall scale (see our reply to Reviewer 6URQ W1). Evaluating such cases perfectly would require writing down every coupling relation of every experiment and a separate evaluation for each, which is infeasible and, for now, unnecessary. Many of these cases can be resolved under reasonable assumptions (e.g. g = 9.81 m/s2, acting downward). Also, models are nowhere near finding such alternative solutions. Restricted to parameters that are recoverable from the video, the best model still recovers only 9.7% within $\pm$20%. If models produced equivalent alternative combinations, their re-rendered videos would match the input. Yet human physics-equivalence ratings stay at or below 2.86/5.