acceptodds
Under review as a conference paper at ICLR 2027

PhySimCode: A Benchmark and Evaluation Method for Physics Video to Code Generation

Abstract

Multimodal large language models (MLLMs) are good at video understanding and code generation, but it remains unclear whether they can recover the physical laws underlying videos, a capability essential for converting videos into executable simulations and for providing structured physical supervision in robotics. Captioning motion video does not imply that a model has inferred its governing dynamics, estimated quantitative parameters, or can regenerate the observed scene from code. Existing video–physics benchmarks evaluate plausibility, property estimation, or future-state prediction, but offer limited insight into whether models can translate visual dynamics into runnable simulation code. We introduce **PhySimCode**, a benchmark for physics video-to-code generation: given only a raw video, a model must infer the relevant physics, recover parameter values, and generate runnable Python that reproduces the input video. The dataset contains **160.6K procedurally generated samples** spanning **162 physics phenomena** across analytic SciPy simulations and PyBullet-rendered rigid-body, articulated, fluid, and contact-rich scenes; each sample bundles the source simulation code, sampled physical parameters, rendered video, and a structured chain-of-thought covering observations, laws, ODEs, numerical methods, and validation checks. We evaluate ten state-of-the-art closed- and open-source MLLMs along physical-law correctness, simulation equivalence, code executability, and parameter recovery. Models reliably state physics laws (4.27–4.93/5) yet score only 1.51–2.47/5 on inferring them from video, and the best system (Claude Sonnet 4.6) recovers just 7.7% of ground-truth parameters within , exposing a substantial video-to-simulation gap and positioning PhySimCode as a testbed for multimodal systems that translate perception into precise, executable, and scientifically verifiable physics. [Dataset](https://kaggle.com/datasets/f38d602eae34cadbac5ed3585739c8e92c8897a040d50d18c0992e96659d53d2) and [Code](https://github.com/PhysicsBenchmark/physics-video-to-code.git) for generation and evaluation are released.

Primary area:
datasets and benchmarks
Keywords:
Physics understanding, code generation, multimodal models, chain-of-thought

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dfDyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PhySimCode, a benchmark for physics video-to-code generation. The dataset contains 160.6K procedurally generated samples spanning 162 physics phenomena, generated using SciPy and PyBullet. It measures three capabilities: video understanding, physics understanding, and programming abilities of MLLMs. It evaluates ten closed- and open-source MLLMs, exposing a substantial video-to-simulation gap and positioning PhySimCode as a testbed for multimodal systems that translate perception into precise, executable, and scientifically verifiable physics.

Critical strengths
  1. The task is new and is a useful task: generating executable code from physics videos. This could support downstream tasks, such as physics-aware symbolic simulations.
  2. Using SciPy and PyBullet could help to improve physical plausibility, potentially improving it compared with approaches that rely solely on VLMs or LLMs to generate videos directly.
  3. The analysis of existing benchmarks and the comparison in Table 1 is clear and well organized. Table 1 provides a useful comparison of their capabilities with other benchmarks.
Critical weaknesses
  1. The validity of the randomly sampled physical parameters is unclear. How do the authors ensure that the sampled parameters and their combinations are physically reasonable? For example, on an inclined plane, a heavy object may slide or roll down, while a lighter object may remain stationary due to friction. Furthermore, it is unclear whether different parameter settings can produce similar motions, which could make the underlying physical parameters difficult to identify from videos.

  2. The reliability of the LLM-based evaluation is unclear. The reported average Likert scores rely on LLM judges, but the paper does not clearly explain whether their judgments were validated by human experts. How do the authors ensure that the LLM judges can reliably assess the correctness of the underlying physical laws?

Questions for authors
  1. Figure 2 is difficult to read. In addition, the paper mentions 192 prompts, but it is unclear how these prompts are defined or how they relate to the 162 experiment types.

  2. The expertise of the human evaluators is unclear. The paper does not provide sufficient information about their backgrounds or whether they have expertise in physics, making it difficult to assess the reliability of their judgments.

  3. The evaluation protocol for failed code executions is unclear. How are samples handled when the generated code fails to execute or render a video? Are these failures excluded from the evaluation or assigned a score? This could affect the reported results.

  4. The use of the full dataset is unclear. Although the paper introduces 160,655 samples, only 2,430 are used for evaluation. What is the purpose of the remaining samples, and how do they contribute to the benchmark? The authors should clarify how the evaluation subset was selected and how the larger dataset supports the benchmark.

  5. References for the evaluated models appear to be missing. Models such as Claude Sonnet 4.6, GPT-5-mini, Grok-4-fast-reasoning, Gemini 2.5 Pro, and Amazon Nova 2 Lite are listed, but the corresponding references are unclear. The authors should provide appropriate citations or links to the official model documentation.

  6. The relationship between human and automatic evaluation remains unclear. Human evaluation provides a useful reference, but it does not by itself establish the validity of the automatic metrics. The authors should investigate how well the automatic metrics correlate with human judgments, particularly for physical plausibility and physics equivalence.

  7. The scalability of the benchmark is questionable if human evaluation is required. If evaluating new models requires substantial human verification, the benchmark may be difficult to use in practice. The authors should clarify whether the automatic metrics can provide a reliable and scalable alternative to human evaluation.

23 comments
AuthorsOpenReview ↗

Weakness 1: physical plausibility of params and identifiability

Thank you so much for this valuable feedback.

The sampling ranges were first proposed by Claude Opus 4.7, prompted to keep the resulting dynamics physically plausible. They then went through the paper's debiasing pipeline (L230–238), which ends with a human verification step. As described in Section 3.1, human experts iterated over each experiment for 2-10 rounds until the behavior rendered from its range was physically plausible. The motion itself is produced by the governing equations (SciPy) or a physics engine (PyBullet), not by us. What we ensure is that each experiment's equations are correct and that the rendered dynamics violate no physical law. Different parameter combinations therefore produce different physical regimes as a consequence of the physics. On the reviewer's example: under Coulomb friction, whether an object slides depends on whether tan$\theta$ exceeds μ\mu, not on its mass, and both outcomes are valid physics that the simulator resolves correctly. This is intentional. One experiment covers many sub-cases depending on its inputs, which is what lets the dataset scale, and it means that more samples from the same experiment add real information.

We understand the reviewer's identifiability concern: different parameters producing similar motion. One way to identify such cases could be joint perturbation study following our sensitivity analysis expt in A.6, but is intractable (discussed in Reviewer vVXZ W3). Identical dynamics from different parameters arise from coupling among parameters, including the unknown overall scale (see our reply to Reviewer 6URQ W1). Evaluating such cases perfectly would require writing down every coupling relation of every experiment and a separate evaluation for each, which is infeasible and, for now, unnecessary. Many of these cases can be resolved under reasonable assumptions (e.g. g = 9.81 m/s2s^2, acting downward). Also, models are nowhere near finding such alternative solutions. Restricted to parameters that are recoverable from the video, the best model still recovers only 9.7% within $\pm$20%. If models produced equivalent alternative combinations, their re-rendered videos would match the input. Yet human physics-equivalence ratings stay at or below 2.86/5.

AuthorsOpenReview ↗

Weakness 2: reliability of LLM judges through human validation

We thank the reviewer for raising this. To address it, we have run a check on human-alignment of the LLM-judge Likert scores.

We sampled 200 inferred physical-law blocks, 20 from each of the 10 benchmarked MLLMs (10 SciPy and 10 Kubric per model). They span 117 of the 162 experiments, with no experiment repeated within a model. Human experts rated every item on the same rubric as the LLM judges, seeing exactly the inputs the judges see. Correctness and equivalence were rated in separate blind passes with independently shuffled item IDs, so the reference law could not anchor the correctness judgment. Model identity was hidden throughout.

We see that 95.5% of correctness and 86.5% of equivalence ratings from the judge panel fall within $\pm$1 of the human rating. The match fraction with human scores look like

JudgeCorrectness: exactCorrectness: ±1Equivalence: exactEquivalence: ±1
GPT-5-Nano0.6750.9350.3700.820
Gemini-2.5-Flash0.7550.9100.5350.840
Grok-4.1-Fast0.7350.9400.4800.830
Judge panel (mean)0.7250.9550.4550.865

So, the paper's central claim is reproduced with human ratings. Humans rate physical-law correctness at 4.72 and equivalence at 1.84. The judge panel gives 4.57 and 1.99. All 10 models show a positive gap under both. Humans separate "states valid physics" from "states the right physics" even more strictly than our judges. Humans and judges also rank the 10 models similarly on equivalence (Spearman ρ\rho = 0.86).

We treat the human ratings as a fourth rater and report quadratic-weighted Cohen's κ\kappa for all rater pairs, as in Figure 4 (n = 200):

CorrectnessHumanGPT-5-NanoGemini-2.5-Flash
GPT-5-Nano0.544
Gemini-2.5-Flash0.5960.645
Grok-4.1-Fast0.5530.6340.708
EquivalenceHumanGPT-5-NanoGemini-2.5-Flash
GPT-5-Nano0.419
Gemini-2.5-Flash0.5790.620
Grok-4.1-Fast0.5930.7290.760

Mean human-judge κ\kappa is 0.56 (correctness) and 0.53 (equivalence); mean inter-judge κ\kappa is 0.66 and 0.70.

One caveat when reading κ\kappa: at n = 200 it is imprecise, more so when ratings are skewed to one end. Correctness is near-saturated (82% of items rated 5 by humans), so its κ\kappa is depressed by high chance agreement. Equivalence ratings span the full scale, so chance agreement is much lower. There, the κ\kappa of the judge panel's mean against humans, 0.59, shows that judges and humans agree on which laws match the reference well beyond what their score distributions alone would produce.

AuthorsOpenReview ↗

Question 1: figure 2 reading difficulty

Thank you for catching this. "192 Prompts" in the Stage 1 box of Fig. 2 is a typo for 162 and we will correct the number in the figure. To make Fig. 2 easier to read, we will redraw it. Stage 1 (ideation) produces the 162 experiment prompts from the 69 expert and 93 selected LLM ideas. We will merge Stage 2 (coding) and Stage 3B (CoT generation) into a single generation stage, since the simulation code and CoT are all produced in the same inference call. Rendering the videos by executing that code with randomized parameters then becomes Stage 3. But then we do an iterative correction, which is Stage 4 (multi-round review). It will be shown as its two sub-stages: 2-10 rounds of human-only physical-plausibility checks on the rendered videos (L214-215), and the three-round LLM-assisted debiasing followed by human verification (L230-241). We will also enlarge the font and thus we hope that the idea written in the main text will be easy to visualize from the updated Fig. 2.

Reviewer_vVXZOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PhySimCode evaluates if multimodal models can translate a physics video via simulation code, including identifying the laws and physical parameters. The benchmark contains ~160K procedurally generated samples across 162 phenomena, combining videos, source code, numerical parameters, and structured physics explanations. Its evaluation separates physical-law correctness, correspondence with the observed phenomenon, parameter recovery, code execution, and regenerated-video equivalence.

Critical strengths
  • The benchmark try to probe a meaningful distinction: stating physical laws and identifying them from motion, estimating parameters, and generating a working simulation are different capabilities. The combined evaluation exposes failures that captioning or code executability alone would miss.

  • The resource and validation are substantial: ~160K samples across 162 phenomena, human-assisted curation, a stage-to-metric mapping, frame-budget experiments, and a matched authorship analysis.

  • The equation evaluation distinguishes whether a model can state a physically correct law from whether it identifies the law governing the observed scene. This is informative because recalling a valid equation does not establish grounded physical understanding.

  • Having Astra and Fable add useful evidence that stronger models still face quantitative recovery challenges and also remove any concerns that these frontier models could solve these problems at any moment.

Critical weaknesses
  • Although, I highlighted that frontier models were used but only 10 handpicked samples were used, but that's not critical.

  • Although, the equations could be used more directly to check whether the generated code implements the physics stated by the model. Reporting cases where the equation is correct but the simulated dynamics are inconsistent would help probe these models better.

  • The sensitivity analysis measures the effect of changing individual parameters, but does not establish their joint distinctiveness. Hence, it would be valuable to see how evaluation handle alternative parameter combinations that reproduce equivalent dynamics?

Questions for authors

None

5 comments
AuthorsOpenReview ↗

Weakness 1: frontier model eval on more samples

Thanks for the encouraging feedback!

We agree. The 10 samples (5 SciPy, 5 Kubric, one per experiment) were chosen as a stress test from cases where all 10 benchmarked models performed poorly, and evaluating frontier models on the full benchmark was beyond our budget. Even on this subset, the frontier models execute 100% of samples, yet the gap in stating correct physical laws and inferring physical laws from the video persists. For the camera-ready version, we will evaluate the frontier models on a larger randomly sampled subset.

AuthorsOpenReview ↗

Weakness 2: consistency between equation and simulation

We thank the reviewer for this suggestion. Our existing human video scoring on whether each regenerated video exhibits the same physical phenomenon as the ground-truth video already allows a first look (Section 4.1). Conditioning on cases where the model has correctly identified the governing law (LLM-judged law equivalence ≥\ge 4/5, this happened for only 44 of the 980 rated videos that had parsable law block), only 25% of the resulting simulations are rated physically equivalent (≥\ge 4/5), while 59% are rated non-equivalent (≤\le 2/5). For example, models state the correct coupled equations for a double-pendulum impact or two-body gravity, yet the simulation shows different dynamics. Identifying the right law is thus only the first step. Turning it into a simulation that reproduces the phenomenon requires substantial further work such as setting up the scene geometry, initial and boundary conditions, contacts and constraints, and a correct numerical implementation. We agree that directly checking whether the generated code implements the model's own stated equations will be a valuable metric (but it is hard, since not all simulation codes use the equations directly, rather some solutions work by computing the change after every time step and do not rely on computing closed form equation of any dynamics). We will include this breakdown with representative cases in the camera-ready version.

AuthorsOpenReview ↗

Weakness 3: joint sensitivity analysis for alternative parameters

Thank you so much for raising this concern. We want to clarify that exhaustive joint perturbation is intractable. With mm parameters taking nn values each, there are nmn^m combinations to re-render. Our experiments have about 10 parameters each (1,576 over 162), so even 10 values per parameter means 101010^{10} videos per experiment. Rarely a few of those parameter combinations could have led to identical dynamics due to coupling (eg., two masses of 1kg and 1 kg colliding with each other should be pixel identical to two masses of 4 kg each). We analyze this parameter coupling phenomenon in our reply to Reviewer 6URQ (W1).

Our evaluation already credits alternative combinations. Parameter recovery is only one of four axes. We execute the generated code and compare the re-rendered video with the GT video, using human physics-equivalence ratings and DINOv2/VideoCLIP. The inferred law is judged separately. A model that found a different combination with the same dynamics would lose points only on some specific parameter recovery and score well on the other axes. This does not happen. Human physics equivalence of the regenerated videos is only 1.13–2.86/5 across all ten models. So low parameter recovery reflects wrong dynamics identification, not valid alternative solutions.

Reviewer_6URQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a benchmark with 160K procedurally generated samples across 162 experiments in 2D and 3D. Each sample includes the video, the simulation code, the parameters, and a CoT annotation. Ten MLLMs are tested on 2,430 of these samples. The evaluation uses LLM judges for physical laws (good) and ±20% parameter recovery (of the best system). The main finding is that models can state physical laws but fail to infer them from video.

Critical strengths
  1. The task is well motivated as an inverse graphic task. Requiring an executable simulation is a stronger test of physical understanding than QA or plausibility judgments.
  2. The generators can produce data across a broad range of experiments, and the pipeline is released.
  3. The authors debias with three-round LLM-assisted pipeline is sophisticated.
Critical weaknesses
  1. The parameter-recovery metric may be unclear. Several invariance are not mentioned, especially (translation and rotational) symmetric variance (for camera) and the depth (or scale) invariance. Since the model gets no spatial scale reference, sizes and speeds are only determined up to the camera scale. This can also largely invalidate the Table 5's Accuracy column and sensitivity stratification experiments, though the latter is well-motivated,

  2. The paper claims video understanding and programming are "largely solved." It seems to be only for physics system classification, not for system identification, as can be seen from the human score or from the results shown in Figure 7 or 8. In generally, this also means that the physical-law metric are weak: correctness is a simple classification, the failure mode shown in A.9 is concerning (the llm-as-judge seems to be ill-defined here; correlation being as low as 0.70 in Figure 4c). It seems that the metric is ill-defined, or the prompt ill-designed, or the evaluation metric needs more work to make it more reliable.

  3. The I/O of the models are too weak Frame of 360180 and 240240 seems arbitrary; token usage allowed are range from 8-32K is unfair, and perhaps far too short for all the models. It is unsure if the model is at their best condition for the evaluation. Also, the code generation seems to be one-round and Reflexion or multi-round are not allowed, limiting the full capability of the VLM/LLM's ability.

  4. The real experiments are conducted on a 2430-sample size dataset, which is much smaller than the claimed 160K.

Questions for authors

Besides the weakness proposed above, I also wonder

  1. Did the authors manually verify the CoT and the ground truth as stated in "For example, we checked LLM-generated dataset for any error in Physics concepts or equations in the CoT through human verification" (Line 494-495)
  2. How is the accuracies and generation quality being evaluated given 47% - 99% various video save rates. (Table 5)
  3. Does allowing more round of code iterations/edits make the performance better? Does allowing more tokens make the models more capable of solving the problems?
10 comments
AuthorsOpenReview ↗

Weakness 1: Param recovery metric clarification

We really liked this valid concern, many thanks to the reviewer. Exact parameter estimation from a single video is confounded by several factors, and invariance is one of them. We still score parameters because many of them are absolutely recoverable from the video under reasonable assumptions. Others are recoverable only up to a reference the model decides on its own, so there can be different parameter combinations inferred. A few of them can also lead to pixel-identical videos due to parameter coupling (below). Handling it perfectly requires deriving all the (often non-linear) relations among parameters, for every experiment, that causes identical dynamics and checking predictions against them. That is infeasible by hand for 162 diverse phenomena (see our reply to Reviewer vVXZ, W3).

Scale: We do not define a scale in the prompt (e.g. "1 pixel = 1 m"). Even that would not fully solve the problem. One solution could be give some parameter values as reference and ask for the rest. But that needs hand-written prompts assigning values to specific objects for every sample and can confuse the model. For Kubric, the model defines its own camera (eye, target, resolution), and for SciPy it writes values in SI units. But the scale is not arbitrary. The model is told 10 FPS, duration-preserving (App. A.4), so time is known in absolute units. With a reasonable physical assumption (g = 9.81 m/s², acting downward), many parameters follow from timing alone. A pendulum's period T=2π(L/g)T = 2\pi\sqrt(L/g) gives L, and a free-fall time gives the drop height (h=12gt2h=\frac{1}{2}gt^2). Inferring metric scale from the dynamics is what PhySimCode is meant to test.

Coupling. Some parameters are not absolutely recoverable even with perfect physics understanding. In an elastic collision only the mass ratio is observable, and in an oscillation k and m appear only as km\frac{k}{m}. App. A.6 already discusses such weakly observable parameters. A model that predicted values consistent with these couplings but at a different reference would score low on parameter recovery. But are models actually that close? Not really! If they produced such equivalent combinations, the re-rendered videos would match the input. Yet human physics-equivalence ratings of the regenerated videos are at most 2.86/5.

Rotation and mirroring. With gravity pointing down, angles measured from the vertical or the floor are fixed in 2D SciPy scenes; in Kubric, perspective makes this harder but still possible in many scenes. If a model reproduces a mirrored but otherwise correct simulation, only position- or orientation-dependent parameters are penalized. Law correctness and equivalence, and the human plausibility, physics-equivalence and appearance ratings, are unaffected. Camera translation and rotation affect only coordinates. The metric already excludes scene geometry (Sec. 4.1), and coordinates are only 44 of the 1,284 scored parameters.

Redoing param accuracy check on recoverable parameters only. We manually labelled all 1,576 GT parameters experiment by experiment, as recoverable or non-recoverable. We assumed lengths, heights and speeds are recoverable only where gravity and time set scale; masses, stiffnesses and forces are never absolute; dimensionless quantities (angles to the vertical or floor, restitution, object counts, and friction where it is not confounded with g) are recoverable.

451 parameters (28.6%) are recoverable. Recomputing Table 9 on these moved the best model (Claude Sonnet 4.6) only from 7.7% to 9.7% within ±20\pm20%. The model ranking is nearly unchanged (Spearman ρ = 0.95). The five strongest models do better on recoverable than on non-recoverable parameters (Claude 9.6% vs 6.7%), while the five weaker ones do worse, so only the strongest models use the available cues, and only a little. Many correct values look like guesses: 16% come from a value the model repeats unchanged for all 15 samples of an experiment, i.e. regardless of the video.

So monocular ambiguity accounts for about two points at most, and the conclusion that current models cannot recover physical parameters from video stands. In the revision we will add the recoverable-only accuracy next to Table 5's accuracy, list every parameter's label in the appendix, and discuss these invariances in Sec. 4.1.

AuthorsOpenReview ↗

Weakness 2: validity of law correctness metric

We made the "largely solved" claim about the general capabilities of current MLLMs. The claim is relative: perception and code synthesis are not the main bottleneck, quantitative physical inference is, and PhySimCode isolates it. We will rephrase the text to say this. We agree that frontier models can qualitatively identify the phenomenon reasonably well, but not quantitatively tell its equations and parameters.

Law correctness is not classification, it is open-vocabulary inference, there is no list of experiments or laws. From law equivalence alone, we cannot tell whether a model fails because it does not know the physics, or because it knows the physics but cannot infer the right law from the video. Law correctness separates these two cases.

Fig. 4c reports quadratic-weighted Cohen's κ\kappa (0.70–0.82), not correlation, which is substantial to almost perfect agreement. The judges are diverse, but diversity does not hurt reliability. They agree within one point on 86.1% of 23,585 items and differ by three or more points on only 1.9%. These rare disagreements (A.9) are cases on which human raters can also differ. We saw the same ambiguity when our human expert rated these items. Each judge has its own biases, which is why we use a panel of judges from three model families and average their scores. Ranking the 10 models by any single judge reproduces the panel's ranking (Spearman ρ≥0.92\rho \ge 0.92).

In the study done for Reviewer dfDy, where a human expert rates physical law correctness and equivalence, following the same rubric as LLM judges, LLMs agree with the human within one point on 95.5% of items for correctness and 86.5% for equivalence, and ranks the models the same way on equivalence (ρ\rho = 0.86). The human also independently confirms the paper's central finding: laws are stated correctly but rarely match the physics in the video, which means LLM-as-judge and both law metrics are well-defined.

The rubric for judging already treats rephrasings and algebraic rearrangements as equivalent. We can add an explicit rule in the prompt that formulations yielding the same equations of motion are equivalent. But model ranking does not depend on the judge, so this change cannot affect any conclusion.

AuthorsOpenReview ↗

Weakness 3: eval fairness from models' I/O budget

Thanking the reviewer for raising this. Each open-source model receives the largest number of frames its context window allows (see our framecount ablation expt in A.14). The SciPy and Kubric videos are rendered at different native sizes, so they were resized to a similar per-frame pixel count (360×180360\times180 and 240×240240\times240) to bound the visual-token cost and leave room for more frames. The output token limit was rarely hit. In the original runs on all 2,430 samples, outputs reached ≥98\ge98% of the budget for only 0.0-0.7% of samples for the closed models, 0.0-3.0% for four open models, and 7.6% for GLM-4.1V-Thinking. Re-running the five open models with 2×2\times and 4×4\times the budget improved no metric beyond run-to-run variation (details in our answer to your Q3).

We ran a full-resolution input expt with Qwen3-VL-30B and GPT-5 mini, the strongest open and closed models on law equivalence, on a 162-sample subset (one random sample per experiment) under the most generous setting we could provide: native resolution and aspect ratio; the full clip; high-detail image input for GPT-5 mini; a 32K output budget; Qwen's context raised to its 262K maximum. All metrics below are compared with the original runs on the same 162 samples.

MetricQwen3-VL: original → fullGPT-5 mini: original → full
Law equivalence (1–5)1.98 → 1.882.42 → 2.45
Law correctness (1–5)4.76 → 4.714.69 → 4.45
Parameter recovery (±20%)6.00% → 6.62%6.70% → 6.39%
Runs without error74.1% → 72.2%90.7% → 88.9%
DINOv2 similarity0.547 → 0.5520.663 → 0.672
VideoCLIP similarity0.715 → 0.7300.785 → 0.785

No metric improves meaningfully for either model. More visual detail and a larger budget do not close the gap. The bottleneck is the models' ability to infer the physics from the video, not the input we give them.

We could not extend this to more models within the rebuttal period. For closed models, full-resolution input multiplies input token cost a lot. For the other open models, it is impossible or ineffective: InternVL3.5's 40K context fits only about 40 native frames; Pixtral and GLM-4.1V cannot hold about half of the full clips; Gemma-3 internally resizes every image to 896×896896\times896.

We also ran a Reflexion-style multi-round experiment with 7 models (details in our answer to your Q3). Iteration fixes programming errors, but leaves parameter recovery and DINOv2 video similarity essentially unchanged. We will add the full-resolution expt to the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.