acceptodds
Withdrawn from ICLR 2027

PhysWeaver: Multi-Agent Programming for Executable Physical Worlds

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video-based world models generate visually plausible rollouts, but since they infer dynamics in latent states, they enforce no explicit physical constraints: contacts drift, shapes distort, and motion loses consistency. We present PhysWeaver, a multi-agent framework that instead constructs the world as executable simulation code. The plan agent converts the natural-language prompt into a structured scene plan that the user can inspect and approve. The code agent implements the plan as an executable simulation script, grounded in a curated skill library, a generative 3D asset pipeline, and retrieval over the simulator source. After execution, the visual-analysis agent describes the rendered rollout, while deterministic physics checks scan the simulated trajectories for anomalies. The review agent evaluates this execution evidence, and the code agent iteratively repairs the program until it satisfies the plan objectives and physical constraints. On a suite of 80 demos selected from the PhyWorldBench benchmark, PhysWeaver satisfies the benchmark's full correctness criterion—semantic adherence and physical correctness judged jointly—on 82.5% of demos, against 52.5% for the strongest of ten text-to-video models, scored under the same criterion on their officially released benchmark videos. The same construction loop extends to interactive use, including a live ROS driving environment in a generated city.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multi-Agent System, Code-Based World Simulation, Inference-Time Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BNbVOpenReview ↗
Rating
1
Confidence
1(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PhysWeaver, a multi-agent framework that instead constructs the world as executable simulation code. Technically, the work uses plan agent to convert a prompt to a structured scene plan, code agent to process the plan and implement the executable simulation script, followed by visual-analysis agent and review agent for judgement, evaluation and refinement. In terms of results, on a suite of 80 demos selected from the PhyWorldBench benchmark, PhysWeaver satisfies the benchmark’s full correctness criterion with semantic adherence and physical correctness judged jointly.

Critical strengths
  1. The work is well motivated to generate scenarios that is sim-ready, and the output is an executable and inspectable environment, supporting the insight and claim. PhysWeaver produces PyChrono programs with explicit geometry, constraints, controllers, and sensors. This is useful for applications requiring interventions, state access, and repeatable execution. The practical contribution is automating simulator construction, which remains labor-intensive even when a mature physics engine is available.
  2. The reported improvement is substantial within the evaluated setting. On the 80 scenarios, PhysWeaver achieves 82.5% joint semantic and physical correctness, versus 52.5% for Pika, equivalent to 66 versus 42 successful scenarios. It satisfies 135/146 individual physical criteria. These results support the narrower conclusion that simulator-based construction can outperform the tested video generators on this selected set of visible physical behaviors.
  3. The framework combines approximately 30 classes of static checks, mandatory simulator retrieval, and three execution stages, i.e., physics-only simulation, first-frame inspection, and full recording. These are sensible safeguards against API errors, invalid placements, and expensive rendering of already-broken scenes. The combination of trajectory diagnostics and visual inspection is also well motivated: either alone can miss important failures.
  4. Several evaluation limitations are disclosed clearly. The authors acknowledge the selected benchmark subset, modified judging protocol, shared model between visual feedback and evaluation, and uncontrolled video duration. This transparency is valuable, even though disclosure does not resolve the methodological concerns.
Critical weaknesses

My main concerns about the paper are:

  1. The central comparison does not isolate the proposed contribution. The baselines reported in quantitative results (Table 1) are text-to-video models. PhysWeaver instead receives an explicit physics engine, simulator documentation, curated implementation skills, structured state, and repeated execution-based repair. The comparison therefore mixes the effect of the proposed orchestration with the much larger difference between simulator execution and video generation. The 82.5% versus 52.5% result does not establish that the four-agent design is necessary or superior. A much more informative baseline would use the same coding model, Chrono engine, assets, retrieval resources, and execution budget, but a simpler single-agent workflow. The absence of this comparison is my strongest concern.
  2. Similarly, there is no controlled component ablation in the paper. The paper attributes reliability to planning, context-isolated agents, the skill library, retrieval, static validation, numeric checks, and visual review. However, it does not quantify the contribution of these components. For example, there is no comparison of single-agent versus four-agent execution, first-pass versus repaired programs, or numeric-only versus combined numeric and visual feedback. Consequently, it remains unclear whether the gains come primarily from the agent organization, curated examples, or simply allowing more debugging attempts. Qualitative repair examples demonstrate that repairs can work; they do not measure their average benefit.
  3. The novelty is framed primarily as workflow engineering, without adequate comparative validation. The introduction explicitly acknowledges that earlier systems already generate simulator code, execute it, inspect results, and repair failures. The claimed remaining contribution is a lightweight, inspectable, portable implementation using documents and off-the-shelf coding harnesses.
  4. The benchmark subset limits the scope of the conclusion. The evaluation covers 80 of 350 scenarios. It excludes two categories and selects two scenarios from each remaining subcategory. Excluding deliberately impossible physics is understandable, and the authors explain the character-asset limitation. However, the within-subcategory selection is not established as random or otherwise independent of system capability.
  5. Human assistance and generation budgets are not adequately controlled. PhysWeaver allows an operator to answer planning questions about consequential quantities such as masses, dimensions, simulation duration, and numerical settings. The video baselines are evaluated using previously released outputs from the original prompts. The frequency and content of these human interventions are not quantified.
  6. All 14 externally unsuccessful scenarios were accepted by the internal review agent. Thus, 17.5% of the evaluated accepted outputs still fail at least one benchmark requirement. The authors correctly distinguish internal plan compliance from external scenario correctness, but this also exposes a weakness: an internally generated plan can simplify or misrepresent the original request.
  7. Construction cost and reproducibility are underreported. The manuscript does not provide a benchmark-wide distribution of construction time, repair counts, model calls, token usage, or monetary cost. The city example requires 11 construction iterations, and the authors state that individual simulation-and-rendering runs occupy the available machine’s CPU and GPU resources.
Questions for authors
  1. Do authors have existing comparisons with a single coding agent using the same simulator, tools, and repair budget, or ablations demonstrating the benefit of the multi-agent design?
  2. For the benchmark, how were the 80 scenarios selected, and were any attempted but internally blocked runs excluded from the reported results?
  3. How many scenarios required human clarification, and what information did the operator supply?
  4. What were the average construction time and repair count per scenario, and the maximum allowed attempts?
Reviewer_kXdYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a train-free multi-agent framework that decomposes physically-grounded video generation into scene planning, code generation, execution, and visual rendering. Implementations are documented and results are reported by VLM-as-a-judge.

Critical strengths
  1. The paper presents an interesting approach to physically-grounded video generation through executable code generation.
  2. The implementation is described in detail, covering agent definition, execution protocol, and visual inspections.
Critical weaknesses
  1. The main concern is regarding evaluation design. Current results are reported by VLM-as-a-judge whose quality seems not explicitly guaranteed or verified by human evaluation. Further given that the pipeline design allows for explicit extraction of object trajectories from simulation results, there could also be substantial benefits of introducing rule-based metrics of physical plausibility.
  2. Another concern is regarding the contribution of different components. It is not clear if the performance comes from the main design of planning-coding-excecution, or other components such as human involvement or curated skillsets. A targeted ablation study would help identify the contribution of the core framework.
Questions for authors

Main Questions:

  1. Are measures taken to ensure the results reported by VLM-as-a-judge are trustworthy, or could the authors provide additional supporting evidence such as human evaluation or physical metrics?
  2. Given that many of components involve established techniques, what do the authors consider as the main methodological novelty beyond system integration?

Other Clarifications:

  1. The pipeline seems to rely on a closed vocabulary of predicates and skillsets. Could it be generalized to other scenarios or physical relations?
Reviewer_f55cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhysWeaver uses planning, coding, visual-analysis, and review agents to translate natural-language prompts into executable physics simulations. The framework iteratively repairs generated programs using simulation trajectories and rendered observations. On an 80-scenario subset of PhyWorldBench, it outperforms the evaluated video generation models on joint semantic adherence and physical correctness.

The framework is a useful engineering approach to constructing executable physical worlds. However, I find the contribution incomplete relative to its motivation: it improves physical consistency without sufficiently addressing or evaluating video realism. Together with the limited evidence for methodological novelty, this weakens the case for the paper in its current form.

Critical strengths

The framework is a useful engineering approach to constructing executable physical worlds. However, I find the contribution incomplete relative to its motivation: it improves physical consistency without sufficiently addressing or evaluating video realism. Together with the limited evidence for methodological novelty, this weakens the case for the paper in its current form.

Critical weaknesses
  1. The paper motivates its approach through the physical limitations of video generation models, but does not establish that its outputs combine physical consistency with realistic appearance. Despite asset generation and visual checks, simplified representations, such as discrete spheres for paint and lava, remain a limitation. Integrating simulation guidance with a video model, potentially through fine-tuning, is one possible way to address this gap. The current contribution feels incomplete relative to its video-world-simulation motivation.
  2. Limited methodological novelty and insufficient ablations. The architecture largely combines established components, including planning, code generation, retrieval, and iterative feedback. The methodological advance beyond their integration is not clearly established. Ablations are needed to verify the contributions of the key modules and the benefit over a single agent with the same tools.
  3. Evaluation emphasizes simulation strengths without separately assessing visual quality. Semantic adherence and physical correctness align closely with the strengths of the proposed pipeline. Although these video-based judgments can capture some visible artifacts, neither metric separately measures photorealism or overall perceptual quality. A dedicated visual-quality evaluation is needed to assess the trade-off with video generation models.
Questions for authors
  1. Appendix A.2 lists Claude Opus 5 for the planning, coding, and review agents, while the accompanying text states that the specific backend varied across campaign sessions. Could you clarify the exact model versions and their allocation across the 80 scenarios?
  2. Have you evaluated alternative base models under the same tools and repair budget? How sensitive are success rates and repair costs to the choice of base model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.