- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This manuscript proposes WorldSolver, a benchmark for evaluating whether LLM agents can generate physics simulation solvers from natural language descriptions. WorldSolver includes 168 tasks from 61 computer graphics papers, covering seven physics domains. Each task provides a fixed simulation environment, and agent is responsible for implementing the physics solver.
The evaluation for generated solvers has three criteria: 1). execution success; 2). visual fidelity; 3). and physical plausibility. Visual fidelity is evaluated using a VLM. Physical plausibility on the other hand is measured through numerical checks on the generated simulation trajectories.
Seven LLM agents are evaluated in this paper. GPT 5.6 Sol and Claude Opus 5 achieve the highest overall scores (48.7 and 46.7). The results show that physically accurate simulation solvers remains challenging for current LLM agents.
Critical strengths
1). This paper proposes a useful benchmark for evaluating physics solver generation. The paper introduces an interesting benchmark that requires LLM agents to implement physics solvers rather than rely on existing simulation engines. By keeping the simulation environment fixed, the benchmark focuses on the agent's ability to understand physical problems, then whether they can write working solvers. The 168 tasks across seven physics domains also provide a good coverage for different type of physical phenomena in our daily life.
Critical weaknesses
1). The physical evaluation does not always reflect actual physics correctness. The paper's own examples show that incorrect simulations can still receive high physical scores. For example, a nearly stationary wind turbine receives a physical score of 100, even though it fails to produce the expected motion (figure 15,). Without simple baseline solvers (or some reference solutions) for comparison, it is difficult to know whether the physical score truly measures simulation correctness.
2). The model comparison may not be reliable. The evaluated models use different agent frameworks. For example, GPT 5.6 Sol uses Codex CLI, but Claude Opus 5 uses Claude Code. Each model is also only evaluated once for each task, without confidence intervals. GPT 5.6 Sol is used as the visual evaluator while it is one of the agents evaluated in the paper. Although the paper includes human evaluation, it does not report agreement for each model.
Questions for authors
1). Could the authors clarify how the evaluation handles cases where the numerical constraints are satisfied but the expected physical behavior is not reproduced? Authors can also comparing against simple baselines, such as a static solver, and clarify which physical quantities are independently verified.
2). Can authors provide results from repeated runs (a confidence interval would also be fine)?
3). Since GPT 5.6 Sol is used for both evaluated agent and visual judge, can authors provide results using an independent VLM or report the agreement with human ratings for each evaluated model?
4 comments
First, thanks for the response and the additional experiments. The added reference solutions, confidence intervals and second visual judge help address several points from my previous review.
However, I am still not fully convinced by the validation of the physical score. The reference solutions score highly, but the nearly static turbine also receives a physical score of 100. This leaves my concern about detecting incorrect dynamics unresolved. Could you clarify which quantities (such as external forces and signed distances) are independently verified against the fixed scene, and which are taken directly from the solver’s outputs?
Also, Eq. (9) averages only the visual and physical scores, while the turbine example in the response averages all three metrics. Is trajectory similarity used only for validation on the 35-task subset or is it included in the official benchmark score? The mapping from RMS error to the reported similarity score also needs to be specified as well.
I will keep my score at 2 for now, but I am open to reconsidering it after these points are clarified by authors.
We sincerely appreciate your constructive feedback and provide the following clarifications regarding your concerns.
Q1: The reference solutions score highly, but the nearly static turbine also receives a physical score of 100. This leaves my concern about detecting incorrect dynamics unresolved. Could you clarify which quantities (such as external forces and signed distances) are independently verified against the fixed scene, and which are taken directly from the solver’s outputs?
A1: We acknowledge that Physical Plausibility has limitations when considered in isolation, which is precisely why we adopt three complementary metrics to provide a more comprehensive evaluation. While the turbine example receives a misleadingly high Physical Plausibility score, our combined evaluation effectively captures the incorrect dynamics and ensures a fair assessment.
-
The fixed scaffold provides the scene geometry, physical conditions, execution protocol and rendering. It defines the simulation problem, while the solver computes the resulting state evolution and exports the quantities required for evaluation, such as velocities and contact forces. The evaluator then computes the physical scores from the solver outputs. In other words, the physical quantities used for evaluation come directly from the solver outputs.
-
For comprehensiveness and faithfulness, we included the turbine example as an edge case to illustrate the limitations of Physical Plausibility when considered in isolation. In most cases, Physical Plausibility does not produce misleadingly high scores, and its scores remain consistent with the external Trajectory Similarity check.
| Metric | GPT-5.6-Sol | Claude-Opus-5 | Gemini-3.7-Flash | DeepSeek-V4.1-Flash | GLM-5.3 | Qwen-3.8-Max | Kimi-K2.7-Code |
|---|---|---|---|---|---|---|---|
| Physical Plausibility | 57.5 | 52.8 | 36.2 | 35.2 | 33.8 | 34.2 | 19.8 |
| Trajectory similarity | 66.6 | 56.0 | 37.3 | 37.3 | 35.6 | 32.3 | 14.8 |
Specifically, as shown in the turbine example below, although GPT-5.6-Sol receives a high Physical Plausibility score, its low Visual Fidelity and Trajectory Similarity scores reveal the incorrect dynamics, ensuring a fair overall assessment.
| Solver | Physical Plausibility | Visual Fidelity | Trajectory Similarity | Avg |
|---|---|---|---|---|
| Reference | 100.0 | 82.3 | 100.0 | 94.1 |
| Claude-Opus-5 | 60.6 | 81.6 | 66.1 | 69.4 |
| GPT-5.6-Sol | 100.0 | 6.1 | 13.5 | 39.9 |
Q2: Also, Eq. (9) averages only the visual and physical scores, while the turbine example in the response averages all three metrics. Is trajectory similarity used only for validation on the 35-task subset or is it included in the official benchmark score?
A2: Sorry for the confusion. Trajectory Similarity is included in our official benchmark, but we currently report its results on a 35-task subset in Section 3.4 due to time constraints. Eq. (9) currently averages only Visual Fidelity and Physical Plausibility, as Trajectory Similarity is not yet included in the reported main results. We will update Eq. (9) to incorporate all three metrics in the final version, along with the full results on the complete benchmark.
Q3: The mapping from RMS error to the reported similarity score also needs to be specified as well.
A3: We apologize for the lack of clarity, and have revised the manuscript to explicitly specify the mapping from RMS error to the trajectory similarity score. Specifically, we use the standard Gaussian form, , and parameterize trajectory similarity as , where is the RMS error, is temporal coverage, and is the half-score error scale. This mapping is bounded, smooth, and monotonically decreasing with error. We use as default and report sensitivity analysis below, showing that the performance ranking remains highly stable.
| GPT-5.6-Sol | Claude-Opus-5 | Gemini-3.7-Flash | DeepSeek-V4.1-Flash | GLM-5.3 | Qwen-3.8-Max | Kimi-K2.7-Code | |
|---|---|---|---|---|---|---|---|
| 0.3 | 54.7 | 46.2 | 30.6 | 28.1 | 26.2 | 26.5 | 11.1 |
| 0.5 | 66.6 | 56.0 | 37.3 | 37.3 | 35.6 | 32.3 | 14.8 |
| 0.7 | 72.7 | 61.6 | 41.2 | 41.9 | 41.2 | 35.9 | 17.1 |
We hope these clarifications address your concerns, and we would be happy to address any further questions.
First of all, thanks for the new additional clarifications. The explanation for the trajectory similarity metric and sensitivity analysis address my question related to the RMS error mapping and help me to clarify the confusion.
But for the reliability of the physical plausibility evaluation, I still have concern for it. In your A1, the physical quantities used for evaluation is come from the solver outputs ("In other words, the physical quantities used for evaluation come directly from the solver outputs."). I am not fully convinced that these checks can reliably identify incorrect dynamics, as turbine example shows me that an incorrect case can receive a physical score of 100.
Additional trajectory similarity metric is helpful but it is not included in eq.9 yet or reported main benchmark results. Therefore, my concerns still remain.
I appreciate the authors' efforts and time, but I will keep my score at 2.
Thank you for your careful review and thoughtful suggestions. Below, we provide additional experiments to address your concerns.
A1: We agree that comparing against reference solutions is important to verify whether the physical score truly measures simulation correctness. Following your suggestion, we used mature physics simulators to build reference solutions across all seven domains. As shown below, they achieve a 97.5 average physical score, demonstrating that our physical score reflects simulation correctness:
We further computed Trajectory Similarity against these reference trajectories as an external quantitative check, which closely aligns with our Physical Plausibility scores:
A2: Our protocol handles this by combining complementary metrics. The "wind turbine" case can serve as strong evidence for our multi-dimensional design. Specifically, even when the Physical Score is misleadingly high (100.0), the overall evaluation can be corrected by the Visual Score (6.1) and the Trajectory Similarity metric (13.5), ensuring a reliable final ranking.
A3: To bring out the full potential of each model, we adopt a framework-native pairing strategy, aligning each model with its officially recommended agent framework (e.g., Codex CLI for GPT-5.6-Sol, Kimi Code for Kimi-K2.7-Code, and Claude Code for Claude-Opus-5). This approach follows established practices in recent benchmarks [1,2,3].
A4: Following your suggestion, we computed the 95% bootstrap confidence intervals for the overall scores across all evaluated models:
A5: Following your suggestion, we added an independent VLM (Gemini-3.8-Flash, which is not among the evaluated agents) as an additional visual judge. As shown below, the visual scores and overall trends remain consistent.
We have updated the manuscript with the above content. Thank you again for your helpful suggestions, and please let us know if you have any further questions.
References
[1] Luo et al., "GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?", arXiv:2606.17861, 2026.
[2] Dong et al., "From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents", arXiv:2610.26098, 2026.
[3] Gao et al., "3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code", NeurIPS, 2026.