acceptodds
Withdrawn from ICLR 2027

WorldAgent: Verification-Guided Agentic Physical World Construction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Constructing complex physical worlds from language requires coordinating extensive 3D environments, detailed structures and objects at different spatial scales, and interacting physical processes under both stated goals and implicit physical constraints. We present WorldAgent, an agentic framework for verification-guided physical world construction from a single natural-language prompt, without iterative user debugging. A world construction layer expands the prompt into a structured world specification and uses physical knowledge to build scenes and run numerical simulations. After every step, a verification layer inspects scene geometry and simulation states alongside rendered views. Failed checks guide automatic revisions to the specification and re-execution of the affected steps. Accepted worlds pass the required checks and remain editable for further inspection and resimulation. We introduce AgenticSimBench, on which WorldAgent achieves the best scores among the evaluated agent-based methods on five of seven metrics. In a 26-participant user study, it receives the highest mean ratings across all four criteria.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Physics-based simulation, Computer graphics, Physics-based animation, 3D world construction, Multi-physics simulation, LLM agents, Verification-guided refinement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mvyPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldAgent, an agentic framework for constructing complex, physically simulated 3D worlds from natural language, it combines structured planning, simulation skills, and step-by-step verification to detect and correct errors during construction. The authors also introduce AgenticSimBench, a benchmark of 13 scenarios. Experiments show improvements over the selected agentic baselines on five of seven metrics, supported by ablation studies and a 26-participant user study.

Critical strengths

The work addresses a practically relevant problem and demonstrates a reasonably complete system for constructing complex, editable, and physically simulated 3D worlds from natural-language descriptions. The visual results are also quite impressive, which adds to the practical appeal of the work.

Critical weaknesses

The method seems more like an engineering integration than a novel methodological contribution. Most components, including structured planning, simulation skills, and verification-based refinement, have been explored in prior work. The paper does not clearly establish what new technical insights or mechanisms it brings beyond existing agentic simulation systems.

The experimental evidence is not sufficiently convincing. The evaluation relies on only 13 self-curated cases, without testing on existing benchmarks or comparing against closely related methods such as SimWorlds and GS-Agent. The unclear agent harness used for the backbone-only comparisons further complicates the interpretation of the reported gains. These limitations make it difficult to establish whether WorldAgent offers a meaningful improvement over existing approaches.

Questions for authors

Given the strong capabilities of GPT-6 Astra in its native Codex agent environment, how does WorldAgent compare with GPT-6 Astra running in Codex directly. Is the Astra-only baseline in Table 3 using the native Codex harness. If not, what harness was used.

What would the authors consider the main technical novelty compared with existing agentic simulation methods such as SimWorlds and GS-Agent. In particular, what benefits come specifically from the proposed verification mechanism rather than the domain-specific skills.

Were the 13 AgenticSimBench cases used during the development or refinement of WorldAgent and its simulation skills? How were these cases selected?

How much additional computational cost does WorldAgent introduce. For example, could the authors provide the average number of verification iterations, execution time, or tool calls compared with the baselines. (and under similiar budget, will the proposed method prevail?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mvyP

Thank you for recognizing the practical importance of the problem, the completeness of the system, and the visual results.

Q1: Comparison with Codex.

The GPT-6 Astra baseline in Table 3 uses the Codex execution environment and can invoke tools to construct and execute scenes. WorldAgent outperforms GPT-6 Astra running directly in Codex. The paired comparison improves geometric consistency, event completion, shot planning, and composition completeness. Specifically, WorldAgent reduces Normalized Maximum Penetration Depth from 0.2248 to 0.0415, increases Causal Beat Completion from 0.6031 to 0.6825, improves Shot Planning Quality from 0.5308 to 0.6042, and increases Scene Composition Completeness from 0.9615 to 1.0000.

Q3: Benchmark Construction

AgenticSimBench tasks were generated by randomly combining Blender’s existing simulation capabilities to produce natural-language descriptions of complex scenes, covering both everyday and cinematic scenarios. These tasks were not used to develop the skills or verification rules. In the revised paper, we will describe the task generation procedure. We will release AgenticSimBench as an open-source benchmark in the future.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer mvyP,

Thank you again for your time and feedback. We posted our response and would appreciate your thoughts on whether it addresses your main concerns. As the discussion deadline approaches, we would be happy to clarify any remaining questions.

Thank you for your consideration.

Best regards, The authors

AuthorsOpenReview ↗

Follow-up on the Codex comparison and added construction evidence

Dear Reviewer mvyP,

Following up on our response, we clarified that the GPT-6 Astra baseline runs directly in Codex with tool access. In the same-backbone comparison, WorldAgent reduces normalized maximum penetration depth from 0.2248 to 0.0415 and increases causal beat completion from 0.6031 to 0.6825.

To further explain the framework’s contribution, we added component-removal evidence and an anonymous 16-iteration walkthrough, documenting how skills and verification guide construction and correction.

Could you let us know whether the harness clarification and added evidence address your concerns, or which specific points still require clarification? With discussion closing on November 18, timely feedback would allow us to respond before the discussion ends.

Thank you for your time.
The authors

Reviewer_JodYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose WorldAgent, a verification-guided agentic framework for constructing complex 3D physical worlds from one natural-language prompt, without iterative user debugging. A world construction layer expands the prompt into a structured specification and instantiates the scene and its simulation using rule-based construction and category-specific solver knowledge. After each step, a verification layer runs eight checks over static scene properties and simulated behavior, and failed checks trigger revisions to the specification and re-execution of the affected steps.

For evaluation, the authors introduce AgenticSimBench with 13 long-form tasks. Using GPT-5.6 Sol for all agent methods, WorldAgent ranks first on five of seven metrics, with the clearest gains on penetration depth and causal beat completion. A 26-participant user study gives it the highest mean rating on all four criteria. Table 3 repeats the comparison with GPT-5.6 Sol, GPT-6 Astra, and Opus 5, and Figure 6 ablates individual checks qualitatively. Rendered output is also compared with Wan 2.2 and MiniMax H3.

Critical strengths

The problem is worth working on. Building one executable world that has to satisfy scene layout, causal event order, and several interacting physical processes at once is harder than reproducing a single physical effect, and AgenticSimBench is a more demanding setting than the short, mechanism-centered prompts used by GS-Agent and SimWorlds. Checking geometry and simulator state after every step, instead of judging only the rendered video, is also a reasonable response to failures that are invisible from the camera, such as occluded interpenetration or liquid volume loss. The controlled comparison is set up fairly in one important respect: all agent baselines share the GPT-5.6 Sol backbone, and the user study is blinded and randomized.

Critical weaknesses

The gains are not attributed to the proposed mechanism. Most of what the system does is integration. The structured specification is a schema, and the execution layer is a body of hand-written rules: Priority-Flood and D8 routing for terrain, PIC/FLIP and advection settings per solver, and failure diagnosis that maps symptoms such as wall-adhering liquid or exploding rigid stacks to specific parameter fixes. Section 3.1.2 states that these skills come from the authors' simulation expertise. That is honest, but it means a large part of the system is injected prior rather than a method someone else could extend to a new physical regime. The verification layer has the same character. Of the eight checks, only scale coherence and interpenetration are given a criterion (Eqs. 1 and 2); contact reuses Eq. 2, and support, navigability, causal order, and category checks are described in prose with no decision rule, so in practice several of them fall back on visual judgment. This is a finer-grained loop than the rollout-level feedback in GS-Agent and SimWorlds, and the single-prompt, no-user-debugging scope is a real difference, but it is a difference of engineering within the same paradigm, and the paper never ablates it on its own. Table 3 makes the attribution problem concrete. GPT-6 Astra alone already reaches 0.6031 causal beat completion, and the full framework moves this to 0.6825; the same comparison on GPT-5.6 Sol goes from 0.3004 to 0.4081. Meanwhile, on GPT-6 Astra the framework lowers alignment (16.22 to 16.12), imaging quality (0.730 to 0.713), and production quality (0.847 to 0.832). The identifiable benefit is concentrated on penetration and causal completion, and even that is not tied to any specific check. I cannot tell whether the mechanism works, or whether a strong model executing a large amount of hand-written graphics knowledge works.

The evaluation does not separate these explanations. Figure 6 removes checks one at a time but only visually, and Table 3 removes the entire framework at once, so there is no evidence for which of the eight checks produce the gains in Table 1. Thirteen tasks with only a macro average is a thin basis for a general claim: one or two scenes can move the number, and no per-task scores or variance are given. For a system whose own conclusion flags the cost of full-trajectory verification as a limitation, I would also expect construction time, token usage, simulation retries, and verification failure rate. None are reported. Finally, acceptance is defined as passing all required checks, with failed or unverified candidates kept as incomplete, but the experiments never say how many of the 13 tasks met that bar and how many were rendered anyway. Without that, the headline numbers are hard to interpret.

Questions for authors

On attribution: Table 3 shows that on GPT-6 Astra the full framework improves penetration and causal completion but slightly hurts alignment, imaging quality, and production quality. Could you provide a quantitative ablation that adds the structured specification, the hand-written skills, and the verification loop one at a time, on the same backbone, so the gain from each can be separated from the gain from the backbone itself?

On the verification layer: Only scale coherence and interpenetration have an explicit criterion. For support validity, navigability, causal order, and the category-specific checks, what exactly is computed, what threshold decides pass or fail, and is the decision made from geometry and simulator state or from a VLM looking at renders? A per-check quantitative ablation, rather than the qualitative examples in Figure 6, would show which of these checks are doing the work.

On evaluation: Of the 13 tasks, how many candidates passed all required checks, and how many were rendered while still failed or unverified? Could you also report per-task scores and the construction cost, including wall-clock time, token usage, number of simulation retries, and verification failure rate? With N=13 and no variance, it is difficult to judge whether the reported margins would hold on a larger set.

3 comments
AuthorsOpenReview ↗

Response to Reviewer JodY

Thank you for your detailed analysis of the method, performance attribution, and experimental settings. We respond to your questions below.

Q2: Verification Computing.

Support and navigability checks use geometric evidence. Semantic completion, key-event visibility, and surface appearance are assessed visually. Causal checks use both geometric or simulation-state evidence and visual evidence. Examples include checking whether river water passes through a specified location and whether water flows out of a tank only after the glass breaks.

Specifically, support checks inspect contact and relative positions between entities and supporting surfaces; navigability checks inspect path continuity, clearance, and obstacles; causal checks inspect the order and relationships among triggers, contacts, and responses; and category-specific checks assess behavior according to the physical system and task requirements.

For example, the waterfall records require tree-root positions to lie between -0.8 and +0.05 m relative to the terrain, use a 0.002 m measurement tolerance for debris-terrain penetration, and flag isolated liquid components larger than 2 cubic meters outside the fall zone. The drift event requires median displacement above 3 m, with at least 50% of composite debris objects rotating by more than 45 degrees. Aggregation and redispersal use changes of at least 25% in nearest-neighbor distances.

These thresholds are associated with specific entity and event requirements. We will further specify the criteria, applicable intervals, and evidence sources in the revision.

Q3: Passing Rate.

13/13 cases in AgenticSimBench passed all required checks.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer JodY,

Thank you again for your time and feedback. We posted our response and would appreciate your thoughts on whether it addresses your main concerns. As the discussion deadline approaches, we would be happy to clarify any remaining questions.

Thank you for your consideration.

Best regards, The authors

AuthorsOpenReview ↗

Follow-up on verification criteria and quantitative component evidence

Dear Reviewer JodY,

We would appreciate your feedback on the additional verification and attribution evidence in our response and revision. Our response distinguishes geometric and simulation-state checks from visual assessments; Appendix A.3 provides task-specific verification criteria for the recorded waterfall case. We have also added quantitative component-removal results and motion/contact diagnostics to examine the complementary roles of skills and verification.

The anonymous 16-iteration walkthrough makes the intermediate states, identified issues, and resulting corrections available for inspection.

Could you identify which concerns about the verification criteria and component attribution remain after considering these additions? As discussion closes on November 18, we would welcome the opportunity to clarify them before the discussion ends.

Thank you for your time.
The authors

Reviewer_rcVGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorldAgent, an agentic framework for automatically constructing 3D worlds and their numerical simulations form a single natural language prompt. The proposed work addresses the challenge of translating high-level descriptions into detailed 3D environments. To evaluate the proposed framework, the authors introduce a benchmark called AgenticSimBench, comprising 13 complex physical-world construction tasks involving diverse environments and interacting physical processes. WorldAgent achieves the best performance on five of seven evaluation metrics. A user study involving 26 participants further shows that WorldAgent obtains the highest average ratings across physical plausibility, shot planning, content alignment, and aesthetics.

Critical strengths
  • This paper proposes a comprehensive framework for physical world construction from high-level natural language description.
  • This paper also introduces a benchmark for evaluating the performance, and the task scope is much harder than previous benchmarks.
  • The empirical results are strong, although the proposed method does not outperform in all the situations.
Critical weaknesses
  • The novelty may be limited, the used iteratively constructing, verifying, and refining simulated environments using execution feedback, are very common in existing agentic simulation and world-generation frameworks.
  • The proposed method may heavily rely on handcrafted domain-specific knowledge
  • No details about how the new benchmark is constructed. I cannot evaluate how fair the evaluation is. Also, the paper does not demonstrate generalization to independently designed or previously unseen task distributions.
Questions for authors
  • What is the key methodological novelty compared to existing agentic simulation frameworks?
  • How much of the improvement comes from handcrafted simulation skills versus verification-guided refinement?
  • Can you report quantitative ablations of individual verification components?
  • Why are GS-Agent, SimWorlds, and ChronoAgentic not included in baselines?
3 comments
AuthorsOpenReview ↗

Response to Reviewer rcVG

Thank you for recognizing the comprehensiveness of the framework, the difficulty of the benchmark, and the experimental results. We address your specific questions below.

Q3: Quantitative Ablations.

Thank you for this insightful suggestion. We separately disabled skill-guided execution and verification-guided refinement and recorded the progressive refinement process described in Comment 2. The component ablations show the effects of removing each component, while the per-iteration record shows how individual verification criteria guide revisions.

We separately removed skill-guided execution and verification-guided refinement on the autumn-valley task. Without skills, the raft exhibits rigid-body motion: its timbers move together instead of articulating. With skills alone, spring-linked timbers move relative to one another, producing a compliant raft, but contact defects remain. Across every delivered film frame, the skills-only output has timber–timber or timber–bed/rock intersections in 301/336 frames (89.6%), compared with 0/152 frames for verification alone; maximum adjacent-timber spacing variation is 5.5 cm and 1.5 cm, respectively. The corresponding clips last 14.00 s and 6.33 s. Production quality (Prod.) is 0.8750 with the full framework, 0.8711 without skills, and 0.8095 without verification. The revised paper reports the geometric diagnostic procedure and complete-clip scope, and the supplementary video compares temporal behavior with the full framework.

Because removing a skill changes the entire refinement trajectory and the skill set contains many entries, individually ablating every entry is computationally impractical. The per-iteration results instead show how verification contracts identify problems at different iterations and guide subsequent revisions. The accompanying visual results show the concrete geometric or visual issues addressed by those revisions, providing evidence for the role of each contract in the recorded process.

Q4: Baseline Comparison.

Thank you for highlighting these closely related works. Their code was not publicly available when we conducted our experiments, which prevented direct comparison in the paper.

GS-Agent emphasizes multi-agent collaboration for configuring physical entities and using rollout feedback; SimWorlds studies staged dynamic scene construction and physical verification; and ChronoAgentic focuses on executable simulation through code generation and repair. These works provide valuable approaches to dynamic scene generation and simulation construction. However, WorldAgent brings scene construction and interacting physical processes together through a shared world specification and multidimensional verification, supporting complex world construction and local revisions. Our rendered examples illustrate the greater physical complexity and visual detail of the worlds we construct. We will discuss the connections and distinctions more carefully in the revision.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer rcVG,

Thank you again for your time and feedback. We posted our response and would appreciate your thoughts on whether it addresses your main concerns. As the discussion deadline approaches, we would be happy to clarify any remaining questions.

Thank you for your consideration.

Best regards,
The authors

AuthorsOpenReview ↗

Follow-up on the added component ablations and refinement evidence

Dear Reviewer rcVG,

Following up on our response, we would appreciate your assessment of the added evidence on the respective roles of simulation skills and verification. The revised paper includes separate component removals, quantitative motion and contact diagnostics, and production-quality scores, with the diagnostic procedure detailed in Appendix F.1.

We have also provided an anonymous 16-iteration walkthrough of the canyon-waterfall task, documenting intermediate results, verification findings, and corresponding construction revisions.

Could you let us know whether these additions clarify the component contributions, and which attribution questions remain? With the discussion closing on November 18, your feedback would help us address outstanding concerns while there is still time for an exchange.

Thank you for your time.
The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their insightful feedback. The main concerns center on the methodological contribution, performance attribution, and time/token costs. We first address these shared concerns in this comment, then respond to each reviewer separately.

Methodological Contribution

WorldAgent builds a complex, executable 3D physical world from a single prompt. As all three reviewers acknowledge, this requires scene structure and simulated behavior to jointly satisfy the task requirements. For example, a flood scene needs rooms, doors, windows, and furniture, together with water that follows plausible routes, interacts with objects, and produces the required outcomes at the appropriate times. As JodY notes, satisfying these requirements together is more challenging than reproducing an isolated physical effect. Previous work has largely focused on either simple dynamic scene generation or individual simulation construction; our work brings the two together to construct coherent, complex physical worlds.

WorldAgent coordinates construction through a structured world specification, combining skill-guided execution for scene construction and solver configuration with verification-guided refinement based on geometry, simulation states, and rendered views. Failed checks guide specification revisions and re-execution of the affected steps. This pipeline achieves strong empirical results: our quantitative evaluation and user study demonstrate its effectiveness, while the rendered examples show detailed environments, coherent physical interactions, and complex event sequences. The ablations below examine the contributions of these components.

Performance Attribution

To clarify how our method works and what its components contribute, we have documented the construction of the canyon waterfall scene iteration by iteration. We present this record in Comment 2 and in the revised paper. An anonymized record is also available at this anonymous link. For each iteration, we document the verification contracts triggered and the skills used to address the identified issues, showing how our framework constructs a complex 3D physical world step by step.

We separately removed skill-guided execution and verification-guided refinement on the autumn-valley task. Without skills, the raft exhibits rigid-body motion: its timbers move together instead of articulating. With skills alone, spring-linked timbers move relative to one another, producing a compliant raft, but contact defects remain. Across every delivered film frame, the skills-only output has timber–timber or timber–bed/rock intersections in 301/336 frames (89.6%), compared with 0/152 frames for verification alone; maximum adjacent-timber spacing variation is 5.5 cm and 1.5 cm, respectively. The corresponding clips last 14.00 s and 6.33 s. Production quality (Prod.) is 0.8750 with the full framework, 0.8711 without skills, and 0.8095 without verification. The revised paper reports the geometric diagnostic procedure and complete-clip scope, and the supplementary video compares temporal behavior with the full framework.

These results illustrate the complementary roles of the two components: skills help the agent determine how to construct and correct the world, while verification identifies where the current result needs correction. The per-iteration record further shows how they work together during construction. Please see the updated paper and video for the ablation results and the visible problems caused by removing each component.

Time/Token Costs

For the canyon-waterfall task in AgenticSimBench, the GPT-6 Astra run with xhigh reasoning effort totalled 25.64 hours of recorded production rounds, and the associated sessions logged approximately 325.6 million tokens. This task ran on a single NVIDIA A800 SXM4 GPU with 80 GB of memory, hosted on a server equipped with two Intel Xeon Platinum 8358 CPUs.

Constructing a scene of this complexity often takes a human artist several days or more than a week, depending on the scene and the artist's experience. Much of this work involves labor-intensive, repetitive tasks, such as modelling trees and rocks and placing them in the scene. Our approach substantially reduces this manual effort and accelerates construction.

AuthorsOpenReview ↗

Waterfall Per-Iteration Results

Thank you all for asking how individual checks guide the agent's revisions. We traced the canyon waterfall through 16 iterations using Opus 5.5, from initial construction to the final render. The revised paper includes the observed issues, associated verification contracts, and resulting changes at each iteration. The accompanying visual record includes intermediate states and reconstructed failed or rejected variants, such as coupling probes, explosive debris motion, and rejected camera framings, showing how the revisions affect the result.

The table below summarizes recorded physical-check issues and the execution skill categories used to address them. Issue counts were retrospectively classified from the retained records. Skill counts refer to distinct execution skill categories and exclude cinematography, appearance adjustments, and engineering fixes.

IterationRecorded verification issues (count)Skill categories used
01Causal (1), Category (1), Numerical (2)2
02No recorded verification issues1
03No recorded verification issues1
04Interpenetration (1), Causal (1), Numerical (2), Contact (1)2
05Interpenetration (1), Navigability (1), Causal (1), Numerical (2)3
06Interpenetration (1), Support Validity (1), Causal (1), Category (1)3
07Interpenetration (1), Category (1), Contact (1)3
08Causal (1), Contact (1)3
09Numerical (1)1
10Contact (1)0
11Category (1)1
12No recorded verification issues1
13Category (1)1
14Category (1)0
15Navigability (1)0
16No recorded verification issues0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.