acceptodds
Under review as a conference paper at ICLR 2027

WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?

Abstract

LLM-based agents are increasingly advancing scientific and engineering problem solving, with physics simulation emerging as a challenging yet practical testbed for reproducing complex physical phenomena with application in embodied AI, games and films. As the workhorse of such simulation, a solver computes how the state of a dynamic system evolves over time. Building such solvers requires physical understanding to identify appropriate models, mathematical reasoning to formulate the underlying dynamics, and software engineering to implement them as executable code, yet this capability of LLM agents remains underexplored. To this end, we introduce WorldSolver, a benchmark of 168 simulation tasks derived from physical phenomena in 61 classic computer graphics papers, spanning 7 physical domains. Each task contains a code scaffold that provides a fixed simulation environment for the scene, with the solver implementation left for the agent to complete. Specifically, we evaluate them along three dimensions: Execution Checks for successful execution, Visual Fidelity for reproducing the intended dynamic behavior in the rendered simulation, and Physical Plausibility for physics-grounded verification of the generated dynamics. Experiments on frontier agents reveal that producing executable solvers is difficult itself, and satisfying visual and physical correctness is even harder. GPT-5.6-Sol and Claude-Opus-5 perform comparatively better than the other evaluated agents, yet achieve overall scores of only 48.7% and 46.7%, respectively. WorldSolver is an early step toward agentic solver generation, and we hope it helps drive progress toward agents that can faithfully simulate the dynamic physical world.

Primary area:
datasets and benchmarks
Keywords:
Physical Dynamics Simulation, Code Generation, Benchmark Dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tDdAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes WorldSolver, a benchmark for evaluating whether LLM agents can generate physics simulation solvers from natural language descriptions. WorldSolver includes 168 tasks from 61 computer graphics papers, covering seven physics domains. Each task provides a fixed simulation environment, and agent is responsible for implementing the physics solver.

The evaluation for generated solvers has three criteria: 1). execution success; 2). visual fidelity; 3). and physical plausibility. Visual fidelity is evaluated using a VLM. Physical plausibility on the other hand is measured through numerical checks on the generated simulation trajectories.

Seven LLM agents are evaluated in this paper. GPT 5.6 Sol and Claude Opus 5 achieve the highest overall scores (48.7 and 46.7). The results show that physically accurate simulation solvers remains challenging for current LLM agents.

Critical strengths

1). This paper proposes a useful benchmark for evaluating physics solver generation. The paper introduces an interesting benchmark that requires LLM agents to implement physics solvers rather than rely on existing simulation engines. By keeping the simulation environment fixed, the benchmark focuses on the agent's ability to understand physical problems, then whether they can write working solvers. The 168 tasks across seven physics domains also provide a good coverage for different type of physical phenomena in our daily life.

Critical weaknesses

1). The physical evaluation does not always reflect actual physics correctness. The paper's own examples show that incorrect simulations can still receive high physical scores. For example, a nearly stationary wind turbine receives a physical score of 100, even though it fails to produce the expected motion (figure 15,). Without simple baseline solvers (or some reference solutions) for comparison, it is difficult to know whether the physical score truly measures simulation correctness.

2). The model comparison may not be reliable. The evaluated models use different agent frameworks. For example, GPT 5.6 Sol uses Codex CLI, but Claude Opus 5 uses Claude Code. Each model is also only evaluated once for each task, without confidence intervals. GPT 5.6 Sol is used as the visual evaluator while it is one of the agents evaluated in the paper. Although the paper includes human evaluation, it does not report agreement for each model.

Questions for authors

1). Could the authors clarify how the evaluation handles cases where the numerical constraints are satisfied but the expected physical behavior is not reproduced? Authors can also comparing against simple baselines, such as a static solver, and clarify which physical quantities are independently verified.

2). Can authors provide results from repeated runs (a confidence interval would also be fine)?

3). Since GPT 5.6 Sol is used for both evaluated agent and visual judge, can authors provide results using an independent VLM or report the agreement with human ratings for each evaluated model?

4 comments

Thank you for your careful review and thoughtful suggestions. Below, we provide additional experiments to address your concerns.

Q1: Without simple baseline solvers (or some reference solutions) for comparison, it is difficult to know whether the physical score truly measures simulation correctness.

A1: We agree that comparing against reference solutions is important to verify whether the physical score truly measures simulation correctness. Following your suggestion, we used mature physics simulators to build reference solutions across all seven domains. As shown below, they achieve a 97.5 average physical score, demonstrating that our physical score reflects simulation correctness:

MetricFluidRigidPlasticElasticRodsClothMultiphysicsOverall
Physical Plausibility91.196.699.899.098.998.099.397.5

We further computed Trajectory Similarity against these reference trajectories as an external quantitative check, which closely aligns with our Physical Plausibility scores:

MetricGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
Physical Plausibility57.552.836.235.233.834.219.8
Trajectory similarity66.656.037.337.335.632.314.8

Q2: Could the authors clarify how the evaluation handles cases where the numerical constraints are satisfied but the expected physical behavior is not reproduced?

A2: Our protocol handles this by combining complementary metrics. The "wind turbine" case can serve as strong evidence for our multi-dimensional design. Specifically, even when the Physical Score is misleadingly high (100.0), the overall evaluation can be corrected by the Visual Score (6.1) and the Trajectory Similarity metric (13.5), ensuring a reliable final ranking.

SolverPhysical PlausibilityVisual FidelityTrajectory SimilarityAvg
Reference100.082.3100.094.1
Claude-Opus-560.681.666.169.4
GPT-5.6-Sol100.06.113.539.9

Q3: The evaluated models use different agent frameworks.

A3: To bring out the full potential of each model, we adopt a framework-native pairing strategy, aligning each model with its officially recommended agent framework (e.g., Codex CLI for GPT-5.6-Sol, Kimi Code for Kimi-K2.7-Code, and Claude Code for Claude-Opus-5). This approach follows established practices in recent benchmarks [1,2,3].

Q4: Can authors provide results from repeated runs (a confidence interval would also be fine)?

A4: Following your suggestion, we computed the 95% bootstrap confidence intervals for the overall scores across all evaluated models:

StatisticGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
Score48.746.729.324.623.622.717.1
CI[44.2, 53.0][42.3, 51.0][24.6, 34.1][20.1, 29.2][19.4, 27.9][18.7, 26.9][13.4, 20.9]

Q5: Since GPT 5.6 Sol is used for both evaluated agent and visual judge, can authors provide results using an independent VLM or report the agreement with human ratings for each evaluated model?

A5: Following your suggestion, we added an independent VLM (Gemini-3.8-Flash, which is not among the evaluated agents) as an additional visual judge. As shown below, the visual scores and overall trends remain consistent.

Visual JudgeGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
GPT45.844.627.322.521.821.215.6
GPT + Gemini45.343.927.422.422.122.516.5

We have updated the manuscript with the above content. Thank you again for your helpful suggestions, and please let us know if you have any further questions.

References

[1] Luo et al., "GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?", arXiv:2606.17861, 2026.

[2] Dong et al., "From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents", arXiv:2610.26098, 2026.

[3] Gao et al., "3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code", NeurIPS, 2026.

First, thanks for the response and the additional experiments. The added reference solutions, confidence intervals and second visual judge help address several points from my previous review.

However, I am still not fully convinced by the validation of the physical score. The reference solutions score highly, but the nearly static turbine also receives a physical score of 100. This leaves my concern about detecting incorrect dynamics unresolved. Could you clarify which quantities (such as external forces and signed distances) are independently verified against the fixed scene, and which are taken directly from the solver’s outputs?

Also, Eq. (9) averages only the visual and physical scores, while the turbine example in the response averages all three metrics. Is trajectory similarity used only for validation on the 35-task subset or is it included in the official benchmark score? The mapping from RMS error to the reported similarity score also needs to be specified as well.

I will keep my score at 2 for now, but I am open to reconsidering it after these points are clarified by authors.

We sincerely appreciate your constructive feedback and provide the following clarifications regarding your concerns.

Q1: The reference solutions score highly, but the nearly static turbine also receives a physical score of 100. This leaves my concern about detecting incorrect dynamics unresolved. Could you clarify which quantities (such as external forces and signed distances) are independently verified against the fixed scene, and which are taken directly from the solver’s outputs?

A1: We acknowledge that Physical Plausibility has limitations when considered in isolation, which is precisely why we adopt three complementary metrics to provide a more comprehensive evaluation. While the turbine example receives a misleadingly high Physical Plausibility score, our combined evaluation effectively captures the incorrect dynamics and ensures a fair assessment.

  • The fixed scaffold provides the scene geometry, physical conditions, execution protocol and rendering. It defines the simulation problem, while the solver computes the resulting state evolution and exports the quantities required for evaluation, such as velocities and contact forces. The evaluator then computes the physical scores from the solver outputs. In other words, the physical quantities used for evaluation come directly from the solver outputs.

  • For comprehensiveness and faithfulness, we included the turbine example as an edge case to illustrate the limitations of Physical Plausibility when considered in isolation. In most cases, Physical Plausibility does not produce misleadingly high scores, and its scores remain consistent with the external Trajectory Similarity check.

MetricGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
Physical Plausibility57.552.836.235.233.834.219.8
Trajectory similarity66.656.037.337.335.632.314.8

Specifically, as shown in the turbine example below, although GPT-5.6-Sol receives a high Physical Plausibility score, its low Visual Fidelity and Trajectory Similarity scores reveal the incorrect dynamics, ensuring a fair overall assessment.

SolverPhysical PlausibilityVisual FidelityTrajectory SimilarityAvg
Reference100.082.3100.094.1
Claude-Opus-560.681.666.169.4
GPT-5.6-Sol100.06.113.539.9

Q2: Also, Eq. (9) averages only the visual and physical scores, while the turbine example in the response averages all three metrics. Is trajectory similarity used only for validation on the 35-task subset or is it included in the official benchmark score?

A2: Sorry for the confusion. Trajectory Similarity is included in our official benchmark, but we currently report its results on a 35-task subset in Section 3.4 due to time constraints. Eq. (9) currently averages only Visual Fidelity and Physical Plausibility, as Trajectory Similarity is not yet included in the reported main results. We will update Eq. (9) to incorporate all three metrics in the final version, along with the full results on the complete benchmark.

Q3: The mapping from RMS error to the reported similarity score also needs to be specified as well.

A3: We apologize for the lack of clarity, and have revised the manuscript to explicitly specify the mapping from RMS error to the trajectory similarity score. Specifically, we use the standard Gaussian form, k(d)=exp⁡(−d2/(2σ2))k(d)=\exp(-d^2/(2\sigma^2)), and parameterize trajectory similarity as S=100Cexp⁡[−ln⁡(2)(E/τ)2]S=100C\exp\left[-\ln(2)(E/\tau)^2\right], where EE is the RMS error, CC is temporal coverage, and τ\tau is the half-score error scale. This mapping is bounded, smooth, and monotonically decreasing with error. We use τ=0.5\tau=0.5 as default and report sensitivity analysis below, showing that the performance ranking remains highly stable.

τ\tauGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
0.354.746.230.628.126.226.511.1
0.566.656.037.337.335.632.314.8
0.772.761.641.241.941.235.917.1

We hope these clarifications address your concerns, and we would be happy to address any further questions.

First of all, thanks for the new additional clarifications. The explanation for the trajectory similarity metric and sensitivity analysis address my question related to the RMS error mapping and help me to clarify the confusion.

But for the reliability of the physical plausibility evaluation, I still have concern for it. In your A1, the physical quantities used for evaluation is come from the solver outputs ("In other words, the physical quantities used for evaluation come directly from the solver outputs."). I am not fully convinced that these checks can reliably identify incorrect dynamics, as turbine example shows me that an incorrect case can receive a physical score of 100.

Additional trajectory similarity metric is helpful but it is not included in eq.9 yet or reported main benchmark results. Therefore, my concerns still remain.

I appreciate the authors' efforts and time, but I will keep my score at 2.

Reviewer_YuoqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldSolver evaluates whether coding agents can translate descriptions of physical phenomena into working numerical solvers. The benchmark contains 168 tasks drawn from 61 computer graphics papers across seven physical domains. Each task supplies the scene, physical setup, code interfaces, and rendering pipeline. The agent chooses a physical model and numerical method, then implements the solver using basic scientific libraries.

The evaluation checks whether the generated code executes, uses a VLM to assess the rendered motion, and calculates physical scores from task-specific constraints on the simulated trajectories. The authors also compare visual scores with human ratings on 231 rendered cases and repeat the VLM judgments to examine their consistency. The leading models obtain overall scores of 48.7 and 46.7 out of 100, indicating substantial room for improvement on the benchmark.

Critical strengths
  1. I like the motivation for testing solver implementation. The fixed scenes and interfaces let the evaluation focus on the agent's physical modeling and numerical choices. The 168 tasks across seven domains provide a broad collection for studying this capability. The task setup and evaluation pipeline are clearly explained.

  2. The evaluation combines rendered behavior with numerical checks on the simulated trajectories. These provide complementary information about the generated solvers, illustrated by concrete examples in the appendix. Human ratings on 231 rendered cases and five repeated VLM evaluations of each fixed output provide useful validation of the visual scoring.

Critical weaknesses
  1. The paper explains how physical scores are calculated, but gives limited evidence that the checks and tolerances reliably distinguish correct and incorrect solvers. Validation with reliable implementations and representative physical errors would help establish the quality of this central benchmark component. The scaffold checks in Appendix A.2 address a different question.

  2. The stability tests repeat judgments of the same output using the same VLM. They do not assess variation across newly generated solvers or different judges. Without those comparisons, close results such as 48.7 versus 46.7 do not establish a stable model ordering.

Questions for authors
  1. How did you validate the physical scoring rules and tolerances? Please describe tests using reliable solvers and representative incorrect implementations, and explain which domains these tests cover. I am interested in whether the checks detect meaningful physical errors, beyond whether the scene scaffolds work correctly.

  2. Was each model–task pair generated only once? Do you have repeated generation results on a representative subset, or results from another VLM judging the existing videos? These would help determine which comparisons are stable. If they are unavailable, please make clear which conclusions concern broad performance differences and which depend on close rankings.

2 comments

We sincerely appreciate your thoughtful feedback and your positive comments on our motivation and pipeline.

Q1: How did you validate the physical scoring rules and tolerances? Please describe tests using reliable solvers and representative incorrect implementations, and explain which domains these tests cover.

A1: We validated our physical scoring rules and tolerances through reliable references, representative incorrect implementations, and external trajectory checks across all seven domains:

  • Validation with reliable solvers. We implemented reference solutions using mature physics engines (Genesis and MuJoCo) across a subset of 35 tasks, including 5 cases per domain across all 7 domains. They achieve an average Physical Plausibility score of 97.5, demonstrating that our rules and tolerances accurately identify physically correct implementations.
MetricFluidRigidPlasticElasticRodsClothMultiphysicsOverall
Physical Plausibility91.196.699.899.098.998.099.397.5
  • Detection of representative failures. As shown in Appendix C.3, we provide a granular comparison of valid and flawed solvers on the same tasks, showing that our physical checks reliably distinguish simulation quality. Specifically, in the Elastic Rod Meandering task, a valid solver achieves a physical score of 86.79, whereas a failed implementation with missing dynamics drops sharply to 1.15.
  • Trajectory Similarity check. Using the reference implementations across all seven domains, we introduce a Trajectory Similarity metric to compare agent-generated trajectories against the references. This metric serves as an external check for our evaluation, and its strong alignment with Physical Plausibility further confirms that our metric reliably distinguishes solver quality.
MetricGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
Physical Plausibility57.552.836.235.233.834.219.8
Trajectory similarity66.656.037.337.335.632.314.8

Q2: Was each model–task pair generated only once? Do you have repeated generation results on a representative subset, or results from another VLM judging the existing videos? These would help determine which comparisons are stable. If they are unavailable, please make clear which conclusions concern broad performance differences and which depend on close rankings.

A2:

  • Model-task pair run. Following standard practice in recent agentic coding benchmarks [1, 2, 3], each model–task pair is evaluated with a single generation pass. To quantify ranking stability across the 168 tasks, we computed 95% bootstrap confidence intervals for the overall scores:
StatisticGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
Score48.746.729.324.623.622.717.1
CI[44.2, 53.0][42.3, 51.0][24.6, 34.1][20.1, 29.2][19.4, 27.9][18.7, 26.9][13.4, 20.9]
  • Independent VLM judge. We use Gemini-3.8-Flash as an additional judge on the existing rendered videos, which is not among the evaluated agents. As shown below, the visual scores and overall rankings remain highly consistent.
Visual JudgeGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
GPT45.844.627.322.521.821.215.6
GPT + Gemini45.343.927.422.422.122.516.5
  • Consistent advantage. Although the previous overall scores of GPT-5.6-Sol (48.7) and Claude-Opus-5 (46.7) are close, GPT-5.6-Sol consistently leads across every dimension, including Visual Fidelity (45.8 vs. 44.6), Physical Plausibility (57.5 vs. 52.8), and the external Trajectory Similarity in subset (66.6 vs. 56.0).
  • Close results of GPT and Claude. In this paper, we treat GPT-5.6-Sol and Claude-Opus-5 together as top-tier agents. Our primary conclusion is that existing models still struggle on this task, even the best-performing agents score under 50%, demonstrating solver generation as a challenging testbed for future progress.

All these additional results and clarifications have been included in the revised paper. We sincerely appreciate your constructive comments and welcome any further discussion.

References

[1] Luo et al., "GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?", arXiv:2606.17861, 2026.

[2] Chi et al., "GameDevBench: Evaluating Agentic Capabilities Through Game Development", ICLR, 2026.

[3] Lu et al., "WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis", NeurIPS, 2026.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. We sincerely appreciate your valuable suggestions and have provided detailed responses to your comments. We hope our responses have adequately addressed your concerns. Please feel free to let us know if you have any further questions.

Reviewer_pBL6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes WorldSolver that evaluates whether LLM coding agents can implement physics solvers from descriptions of target phenomena and fixed simulation scaffolds. It contains 168 tasks across seven physical domains. Each scaffold fixes the geometry, physical settings, execution interface, and rendering configuration, while the agent implements the solver. Generated programs are evaluated through execution checks, VLM-based visual assessment, and task-specific numerical checks of physical consistency. The paper compares seven coding agents and analyzes their performance across domains.

Critical strengths
  1. Broad task coverage and substantial benchmark construction. The benchmark covers 168 scenarios across seven physical domains, including contact, deformation, fluids, and coupled phenomena. Organizing these scenarios into executable tasks provides a useful resource for studying scientific coding agents beyond conventional programming problems.
  2. Controlled comparison through fixed scaffolds. Fixing scene geometry, physical parameters, execution interfaces, and rendering settings reduces variation unrelated to solver implementation. This makes comparisons between agents more interpretable and helps focus the evaluation on their modeling, numerical-method selection, and implementation choices.
Critical weaknesses

The evaluation metrics do not adequately assess physical correctness, and the composite score lacks sufficient justification. The final score is the equal average of Visual Fidelity and Physical Plausibility. Both components have clear limitations: Visual scores use only 20 frames judged by a VLM. They detect missing events but cannot reliably measure fine-grained quantities (velocity, collision timing, deformation) or catch brief inter-frame artifacts, so a high score does not imply physical correctness. Physical scores check only 3–5 predefined constraints. Appendix C.2 shows a nearly static wind turbine receiving a perfect score solely because residuals are near zero, while failing to produce the target phenomenon. The checks verify consistency of exported quantities rather than correctness of the process. Since either score can be misleadingly high, their average is unvalidated as a measure of physical correctness.

Questions for authors

Since each task already fixes the geometry, physical parameters, and initial conditions via a code scaffold, why not use existing mature physics simulators to generate reference trajectories under the same conditions and evaluate the generated solvers against them with quantitative error metrics on key physical quantities? This would more directly measure physical correctness and provide a necessary external check for the current VLM- and partial-constraint-based protocol.

2 comments

Thank you for your thorough review. We appreciate your insights regarding the metrics, and conducted additional experiments to address your concerns below.

Q1: The composite score lacks sufficient justification.

A1: We adopt the equal average of Visual Fidelity and Physical Plausibility because reproducing visible phenomena and obeying physical laws are equally important for a faithful simulation. To show that our conclusions do not rely on a specific weight, we tested alternative combinations below and the performance rankings remain unchanged:

V : PGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
25% : 75%50.147.830.325.624.523.517.9
50% : 50%48.746.729.324.623.622.717.1
75% : 25%47.245.728.323.522.721.916.4

Q2: Why not use existing mature physics simulators to generate reference trajectories and evaluate the generated solvers against them?

A2: Thank you for this insightful suggestion. To provide a rigorous external reference, we built reliable implementations using mature physics engines, such as Genesis and MuJoCo, on a subset of 35 representative tasks across all 7 physical domains. As shown below, these reference trajectories achieve an average Physical Plausibility score of 97.5, showing that our physical scores are calibrated with mature simulators:

MetricFluidRigidPlasticElasticRodsClothMultiphysicsOverall
Physical Plausibility91.196.699.899.098.998.099.397.5

We further evaluated the agent-generated trajectories against the reference implementations using Trajectory Similarity. The results show that this external check aligns closely with our internal physical scores, providing solid validation for our evaluation. Implementation details and Trajectory Similarity definitions are included in the revised manuscript, and we will release the complete reference set and use Trajectory Similarity as an official metric in the final version.

MetricGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
Physical Plausibility57.552.836.235.233.834.219.8
Trajectory similarity66.656.037.337.335.632.314.8

Q3: Both components have clear limitations. Visual scores use only 20 frames judged by a VLM, and physical scores check only 3–5 predefined constraints.

A3: We agree that neither metric alone is sufficient, which is why our protocol combines them to work together. Visual Fidelity detects key events and missing phenomena, while Physical Plausibility checks fine-grained quantities across continuous time steps. Together, these complementary metrics mitigate each other's limitations, ensuring a robust and comprehensive evaluation.

  • Frame number for Visual score. To check whether 20 frames miss visual details or brief artifacts, we re-evaluate the rendered videos using 40 frames. As shown below, the visual scores and model rankings remain highly consistent, confirming that 20 frames are sufficient for reliable visual evaluation:
FramesGPT-5.6-SolClaude-Opus-5Gemini-3.7-FlashDeepSeek-V4.1-FlashGLM-5.3Qwen-3.8-MaxKimi-K2.7-Code
2045.844.627.322.521.821.215.6
4047.344.127.321.820.621.514.9
  • Physical scores check only 3–5 predefined constraints. We agree that the "wind turbine" example shows a limitation in the Physical Plausibility. However, this case can also demonstrate the comprehensiveness of our multi-dimensional evaluation: even when one component is misleadingly high, the overall assessment is balanced and corrected by our other metrics. In this case, the Visual Fidelity of 6.1 and the Trajectory Similarity of 13.5 clearly penalize the static solution, ensuring a reliable final ranking:
SolverPhysical PlausibilityVisual FidelityTrajectory SimilarityAvg
Reference100.082.3100.094.1
Claude-Opus-560.681.666.169.4
GPT-5.6-Sol100.06.113.539.9

Thank you again for your valuable feedback, and we are happy to discuss any further questions.

Dear Reviewer,

I hope this message finds you well. We have carefully revised our manuscript and sincerely hope that our revisions have adequately addressed your concerns. Your feedback is very important to us, and we would be more than happy to address any remaining questions.

Thank you again for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.