acceptodds
Under review as a conference paper at ICLR 2027

InfiniDream: Open-Ended Photorealistic 3D Worlds Through Procedural Generation

Abstract

Procedural generation plays a key role in synthetic data creation and game development, offering greater controllability, editability, and diversity than static mesh representations. However, VLM-based code generation often produces relatively simple 3D assets, while sophisticated photorealistic procedural systems require extensive manual development. Existing VLM-guided procedural methods remain largely constrained by predefined factories and APIs, limiting their ability to create novel asset types. We introduce InfiniDream, a VLM-driven framework for open-ended procedural 3D world generation. First, we propose Symbolically Constrained Asset Generation, which leverages existing procedural algorithms to create novel, photorealistic assets while preserving editability and seed-based diversity. Second, we develop an end-to-end text-to-world pipeline that combines semantic scene planning with Deterministic Frustum-Guided Grid Placement to construct large-scale 3D worlds with precise object placement, procedural population, and animation. Extensive evaluations on 100 object and 20 scene generation tasks demonstrate improvements across four perceptual metrics. Diversity analysis, ablation studies, and human preference evaluations further validate the effectiveness of our framework.

Primary area:
generative models
Keywords:
agentic 3D, 3D scene editing, procedural generation, code generation, controllable 3D generation, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZgUJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InfiniDream, a VLM-driven framework for generating editable procedural assets and composing them into 3D worlds. The asset side relies on Symbolically Constrained Editing (SCE), which starts by retrieving an Infinigen factory, builds a structured code summary and editing boundary, and then extends the factory through planning and render-guided refinement. For scene generation, the key idea is to separate semantic planning from deterministic geometric placement, using frustum-guided candidate search together with constraint filtering.

Critical strengths

1.The authors cleverly address the limitation of existing VLM-guided procedural generation. Instead of forcing the VLM to directly predict continuous 3D coordinates (which is error-prone), they introduce a Deterministic Frustum-Guided Grid Placement mechanism. By delegating semantic reasoning (where to put it, what to put) to the VLM and continuous geometric computation (exact 3D coordinates, collision checking) to a deterministic algorithm, they achieve much more reliable object placement.

2.The proposed Symbolically Constrained Editing (SCE) represents a meaningful step forward for open-ended asset generation. By allowing the VLM to edit the underlying procedural code while maintaining the original factory's function signatures and seed-based randomness, InfiniDream successfully generates novel, photorealistic assets that were previously impossible with predefined factories.

Critical weaknesses

I find the pipeline quite heavy. It relies on multiple VLM calls (Planner, Director, Reviser) plus up to six rounds of iterative code refinement with Blender rendering, but the paper doesn't say much about the actual computational cost or wall-clock time. If it takes tens of minutes or hours to generate a single scene, the practical scalability is severely limited.

Questions for authors

1.Can you test InfiniDream without the 70-scene retrieval library and compare it against a baseline using the same initial coarse scenes? I want to know how much of the improvement is genuinely generated versus just picked from Infinigen.

2.Can you break down the benchmark by difficulty (parameter changes, new structures, major extensions) and report success rates for each? This would clarify where the "open-ended" claim actually holds up.

3.What's the end-to-end generation time and model-call cost, split across library prep, asset code generation, placement, and rendering? With multiple VLM calls and up to six rendering rounds, a breakdown is important to assess practical scalability.

If the authors can address these questions, I would be happy to raise my score.

5 comments
AuthorsOpenReview ↗

Response to W1 & Q3: Computational Cost and Scalability (1/3)

We thank the reviewer for the suggestion. To clarify the computational overhead and practical scalability of InfiniDream, we provide a breakdown of wall-clock time, VLM calls, and API costs across different pipeline stages.

1. Asset Generation: Generating a new procedural asset takes approximately 38 minutes, 15.3 VLM calls, and $10.24 on average (including up to six refinement rounds). Generated assets are stored in a reusable procedural asset library, making this a one-time construction cost that does not need to be repeated for subsequent scenes.

2. Scene Generation: When all required assets are available in the library, scene selection, planning, object placement, and final rendering take approximately 24 minutes, 5 VLM calls, and $1.07.

3. Library Preparation: Preparing a coarse scene takes approximately 29 minutes on average. The 70-scene library is built once offline and its scenes are directly retrieved at generation time, so this cost is not incurred when generating a scene.

4. End-to-End: Each asset is generated independently, so new assets can be generated in parallel across GPUs, and assets already in the library are simply retrieved. The wall-clock time of a full scene is therefore about 24 minutes when its assets are retrieved, and about one hour (38 + 24 minutes) when new assets are generated in parallel, rather than growing with the number of assets.

The per-stage breakdown is included in Appendix Q (Table 13). All timings are measured on a single NVIDIA RTX 4090 GPU.

MetricAsset PlanningCodingRefinementSelection & QAAsset TotalScene PlanningPlacementRenderingScene Total
Time (min)5.48.320.24.037.94.86.212.723.7
Calls1.32.011.01.015.35005
Cost ($)1.452.676.070.0510.241.07001.07
AuthorsOpenReview ↗

Response to Q1: Scene Picker Ablation and Same-Initialization (2/3)

To distinguish the contributions of scene retrieval and actual generation, we conducted two additional experiments on the same 20 test scenes.

1. Scene Picker Ablation: We compare three initializations: (1) Full, selecting the initial scene from the 70-scene library; (2) Random, generating scenes of the same biome on the fly with five new random seeds outside the library; and (3) Minimal Biome, keeping only the terrain and environment of the selected scene. InfiniDream remains robust in both alternative settings (CLIP 29.45 and 28.60 vs. 29.64 for Full), showing that the richness and diversity of the final scenes come from the high-quality assets that InfiniDream generates, rather than from what is picked from Infinigen.

2. Same-Initialization Comparison: We provide VLM Harness and Code2Worlds with the exact same initial scenes selected by InfiniDream, including their Blender files and procedural code. InfiniDream outperforms both baselines across all four metrics, so its advantage holds under identical initialization and is not attributable to the Scene Picker alone. A general VLM agent has difficulty handling the large procedural code of the initial scene, and Code2Worlds is restricted to adjusting scene-level and object-level parameters, which limits what it can edit.

Detailed experimental settings and visual comparisons are included in Appendix O (Table 11, Figs. 21–23).

Scene Picker Ablation

MethodInitializationCLIP ↑SigLIP2 ↑DINOv3 ↑IR ↑
Ours (Minimal)Basic biome only29.450.1420.350−0.736
Ours (Random)5 new random seeds28.600.1390.339−1.042
Ours (Full)70-scene library29.640.1430.361−0.877

Same-Initialization Comparison

MethodInitializationCLIP ↑SigLIP2 ↑DINOv3 ↑IR ↑
VLM HarnessMatched initial scenes28.260.1360.351−1.157
Code2WorldsMatched initial scenes24.150.0820.157−1.980
Ours (Full)Matched initial scenes29.640.1430.361−0.877
AuthorsOpenReview ↗

Response to Q2: Difficulty-Stratified Evaluation of Open-Ended Generation (3/3)

To further evaluate InfiniDream's open-ended procedural generation capability, we categorize the 100 assets from our original evaluation into three difficulty levels, based on the modifications required to extend the original Infinigen factories:

  • D1 (38 assets): Parameter adjustments to existing factories.
  • D2 (40 assets): Addition of local geometric structures.
  • D3 (22 assets): Major extensions to the original geometry generation rules.

Importantly, the difficulty levels are defined by the expressive limitations of the original Infinigen factories, rather than the outputs of individual methods. For baselines that generate code or geometry without relying on Infinigen factories, these levels do not necessarily reflect their intrinsic generation difficulty. Code2Worlds is evaluated with its factory coverage manually extended to all 100 objects (Appendix M).

We use a single evaluator, GPT-Sol-5.6, to perform binary feature checks and report Feature Pass Rate to quantify generation performance across all methods.

Table 9 (Appendix M). Feature Pass Rate across three levels of procedural generation difficulty.

VLM HarnessInfinigenCode2WorldsVIGABlenderRAGInfiniDream (Ours)
D10.840.020.260.660.740.97
D20.850.000.000.780.650.93
D31.000.000.000.820.821.00
Overall0.880.010.100.740.720.96

The results show that InfiniDream achieves Feature Pass Rates of 97%, 93%, and 100% on D1, D2, and D3, respectively, with the highest overall score of 96%. Notably, InfiniDream maintains high performance on D2/D3, which require new geometric structures or major extensions to generation rules, whereas Code2Worlds achieves 0% on both. These results indicate that InfiniDream can overcome the expressive limitations of existing procedural factories, rather than being restricted to predefined parameter adjustments.

The evaluation procedure and visual comparisons across difficulty levels are included in Appendix M (Table 9, Fig. 18).

Reviewer_YZmeOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces InfiniDream, an agentic method for creating procedural scenes provided a text or image prompt. Unlike past methods, which compose scenes from a static set of primitives (e.g. Blender primitives, Infinigen programs, or retrieved assets), InfiniDream is able to expand the underlying asset/primitive sources. The authors demonstrate a low-level agentic pipeline for novel assets reconstruction based on Infinigen, and a higher level agentic pipeline for orchestrating scene construction. Their method achieves better accuracy and semantic correctness for both object and scene-level generation compared to their chosen baselines.

Critical strengths
  • The strength of the idea is very clear; and limited expressivity of the underlying primitive set/procedural programs were a clear limitation of past generation methods. VIGA can construct new programs but its “seed set” of basic primitives is typically too weak.

  • The methods introduced in the object-level synthesis pipeline seem to be reasonably general and could be used for general expansion of procedural generator diversity.

  • The qualitative results are good and show better alignment to the prompt and better realism than existing methods;

  • I think the results for natural objects are quite impressive - a notable limitation of even frontier models like GPT-6 is procedural program construction of complex natural objects that don’t have simple, flat geometry. The idea of editing and composing several existing, human-made factories therefore seems promising (at least given current model capabilities)

Critical weaknesses
  • This is not specific to this paper, but I am generally skeptical of most quantitative metrics (e.g. SIGLIP, DINO). They are difficult to interpret, typically easy to saturate, and entangle different factors. For example, it is difficult to tell from a CLIP metric whether a method scores better from conceptual alignment to the text prompt vs has a more “in-distribution image” (one can imagine perfect alignment to the prompt, but poor realism). However, the authors also provide a reasonable human preference study that dis-entangles these two axes.

  • It seems all VIGA examples were done without asset retrieval enabled, and only Blender primitives. However, I understand that since this setting is concerned with procedural generation, this is the correct apples-to-apples comparison. It would still strengthen the paper to compare against the best possible version of VIGA.

Questions for authors

I have no major concerns/clarifications necessary.

4 comments
AuthorsOpenReview ↗

Response to W1: Limitations of Automated Metrics (1/2)

We thank the reviewer for the positive feedback and valuable suggestion. We agree that image-based metrics such as CLIP, SigLIP2, and DINOv3 cannot fully disentangle semantic alignment from visual realism. To address this limitation, we also conduct a human preference study that separately evaluates semantic alignment and visual realism, providing a more comprehensive and interpretable assessment. We have added a discussion of the limitations of automated metrics and the complementary role of human evaluation to the revised paper (Appendix R).

AuthorsOpenReview ↗

Response to W2: VIGA with External Assets (2/2)

We thank the reviewer for the suggestion and for acknowledging the fairness of our original comparison. To provide a more comprehensive evaluation, we additionally evaluate VIGA with Meshy asset retrieval, allowing it to use external 3D assets instead of relying solely on Blender primitives.

While Meshy improves VIGA's generation quality, InfiniDream still outperforms it on CLIP (29.64 vs. 28.89), SigLIP2 (0.143 vs. 0.134), and ImageReward (−0.877 vs. −1.100), while VIGA with Meshy achieves a slightly higher DINOv3 score (0.365 vs. 0.361). InfiniDream thus remains competitive even when VIGA uses external assets.

Detailed results and visual comparisons are included in Appendix P (Table 12, Fig. 24).

MethodCLIPSigLIP2DINOv3IR
VIGA (code)27.140.1210.246-1.433
VIGA (Meshy)28.890.1340.365-1.100
InfiniDream (Ours)29.640.1430.361-0.877

Thank you for the reply. I otherwise have no major concerns and maintain my recommendation.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the professional and constructive feedback throughout the review process, which has helped us strengthen the paper.

Reviewer_JjmpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents InfiniDream, a VLM-driven framework for open-ended photorealistic 3D world generation. The framework introduces Symbolically Constrained Editing (SCE) to modify and extend existing Infinigen procedural generators while preserving editability and seed-based diversity. It also proposes Deterministic Frustum-Guided Grid Placement, which combines semantic scene planning with geometric constraints to achieve precise object placement.

Critical strengths

The paper presents a well-structured end-to-end framework that integrates procedural asset generation, semantic scene planning, object placement, and scene population. The overall pipeline is clearly presented and demonstrates practical value for 3D world generation.

Critical weaknesses
  1. The overall 3D scene generation framework proposed in this paper is relatively similar to existing works. Previous methods, such as WorldCraft (2025) and Code2Worlds (ICML 2026), have already explored the use of LLMs/VLMs for scene planning, procedural asset generation, and scene layout. Although the authors introduce mechanisms such as code structure mapping, editable module constraints, and iterative refinement, these designs primarily integrate existing code analysis techniques with VLM-based agentic code editing workflows, making the contribution largely engineering-oriented.
  2. Appendix I explicitly states that InfiniDream selects an initial scene from a library of 70 pre-generated environments, whereas the competing baselines do not use this library. This introduces differences in scene initialization conditions across methods, and the quality of the initial scenes may directly influence the visual quality and evaluation results of the final generated scenes. But the paper does not provide an ablation study that removes the Scene Picker, nor does it conduct sufficient controlled comparisons under identical initial scene conditions. Consequently, it remains unclear whether part of InfiniDream's performance advantage over competing methods comes from the pre-generated environments rather than the effectiveness of the proposed methods themselves.
  3. The paper emphasizes that its approach can overcome the limitations of existing procedural generators, enabling Open-Ended Procedural Asset Generation while preserving the procedural editability of generated assets. However, the current experiments primarily evaluate generation quality using image-based metrics such as CLIP, SigLIP2, DINOv3, and ImageReward. Although these metrics can reflect visual quality and semantic alignment, they cannot directly demonstrate that SCE is capable of generating novel procedural structures beyond the expressive capabilities of the original Infinigen factories. Furthermore, the paper lacks direct quantitative evaluations of the procedural editability of the generated assets.
  4. There are some minor inconsistencies in terminology and presentation. For example, SigLIP2 is referred to as "SigLIP" in Tables 1 and 2, while the proposed method is labeled as "Ours" in Table 1 but "InfiniDream" in Table 2. The authors should ensure consistent terminology throughout the paper.
Questions for authors

See Weakness

6 comments
AuthorsOpenReview ↗

Response to W1: Contribution Beyond Predefined Generators (1/4)

We thank the reviewer for the feedback. While prior works have explored LLM/VLM-based procedural generation, our key contribution is to overcome the expressive limitations of existing procedural factories, rather than merely operating predefined generators. WorldCraft primarily invokes existing Infinigen generators, while Code2Worlds introduces a Coding Agent but struggles to extend the underlying geometry generation logic within Infinigen's codebase of over 40,000 lines. Our key insight is to enable VLMs to identify and modify task-relevant generation logic without processing the entire codebase, while preserving procedural executability and editability.

Evidence 1: extension beyond existing factories. We categorize the 100 evaluated assets into three difficulty levels: D1 (38, parameter adjustments), D2 (40, new geometric structures), and D3 (22, major extensions to generation rules). Code2Worlds achieves a 0% Feature Pass Rate on D2/D3, whereas InfiniDream achieves 93%/100%, even though we manually extended Code2Worlds' factory coverage to all 100 objects. General code-generation agents can also produce diverse objects that satisfy the requested features (VLM Harness: 85%/100%), but they mainly rely on constructive solid geometry (CSG)-style composition of simple geometric primitives, which limits them to simplified geometry and lower visual quality (Table 1). Details are in Appendix M (Table 9, Fig. 18).

Evidence 2: not a plain agentic coding workflow. In our object-level ablation (Table 5), Agent (Infinigen) gives the same VLM the entire Infinigen codebase with Bash access and the same six-round budget, but without our code structure mapping and editable-module constraints. It performs clearly worse than the full method (CLIP 29.01 vs. 30.37, DINOv3 0.387 vs. 0.494, ImageReward −0.968 vs. −0.580), showing that the proposed symbolic constraints, rather than the agentic workflow itself, make extending a large procedural codebase reliable.

AuthorsOpenReview ↗

Response to W2: Scene Picker and Same-Initialization (2/4)

To assess the impact of Scene Picker on generation quality and comparison fairness, we conducted two additional experiments on the same 20 test scenes.

1. Scene Picker Ablation: We compare three initializations: (1) Full, selecting the initial scene from the 70-scene library; (2) Random, generating scenes of the same biome on the fly with five new random seeds outside the library; and (3) Minimal Biome, keeping only the terrain and environment of the selected scene. InfiniDream remains robust in both alternative settings (CLIP 29.45 and 28.60 vs. 29.64 for Full), showing that the richness and diversity of the final scenes come from InfiniDream’s open-ended generation of high-quality assets rather than from the pre-generated library.

2. Same-Initialization Comparison: To ensure a fair comparison, we provide VLM Harness and Code2Worlds with the exact same initial scenes selected by InfiniDream, including their Blender files and procedural code. InfiniDream outperforms both baselines across all four metrics, indicating that its advantage persists under identical initialization and does not come solely from the Scene Picker. VLM Harness struggles to operate on the large procedural code of the initial scene, while Code2Worlds, which edits a scene by adjusting scene-level and object-level parameters, has limited editing capability.

Detailed experimental settings and visual comparisons are included in Appendix O (Table 11, Figs. 21–23).

Scene Picker Ablation

MethodInitializationCLIP ↑SigLIP2 ↑DINOv3 ↑IR ↑
Ours (Minimal)Basic biome only29.450.1420.350−0.736
Ours (Random)5 new random seeds28.600.1390.339−1.042
Ours (Full)70-scene library29.640.1430.361−0.877

Same-Initialization Comparison

MethodInitializationCLIP ↑SigLIP2 ↑DINOv3 ↑IR ↑
VLM HarnessMatched initial scenes28.260.1360.351−1.157
Code2WorldsMatched initial scenes24.150.0820.157−1.980
Ours (Full)Matched initial scenes29.640.1430.361−0.877
AuthorsOpenReview ↗

Response to W3: Procedural Structures and Editability (3/4)

We agree that image-based metrics alone cannot adequately show structural extension and procedural editability. We therefore conducted two additional evaluations.

1. Novel Procedural Structures: As discussed in W1, we categorize the 100 evaluated assets into three difficulty levels (D1/D2/D3) based on the modifications required to extend the original Infinigen factories. For D2/D3, which require new geometric structures or major extensions to generation rules, InfiniDream achieves Feature Pass Rates of 93%/100%, compared to 0%/0% for Code2Worlds. We further analyze the generated code directly: all 100 InfiniDream assets modify the factory's generation code rather than only its parameters (median 165 added lines), 86% introduce new functions or classes, and 87% of the D2/D3 assets add new functions or classes to the geometry-generation code. Infinigen and Code2Worlds use the original factory code with its native parameters only (0%).

2. Procedural Editability: Following 3DCodeEdit [1], we evaluate five types of object-level edits on 15 objects (75 tasks in total): parameter changes (G1), single-component modifications (G2), multi-component modifications (G3), structural reorganization (G4), and style changes (Style). A single evaluator model (GPT-Sol-5.6) judges each edit, and we report the Feature Pass Rate. InfiniDream reaches a pass rate comparable to VLM Harness and above VIGA (83% vs. 80% and 75%), with the highest post-edit CLIP score (29.86 vs. 29.26/26.37), while editing much longer procedural programs (median 1,732 lines vs. 293 and 175) that produce more detailed geometry (Fig. 19).

To further assess procedural diversity and feature preservation, we select one representative case from each editing category and regenerate assets using five different random seeds per case. All generated variations preserve the requested features while exhibiting geometric diversity, indicating that the edited factories remain generative rather than producing fixed geometry (Fig. 20, Appendix N).

These results support InfiniDream's ability to extend existing procedural factories while preserving procedural editability and generative diversity. Detailed results and visual comparisons are included in Appendix M (Table 9, Fig. 18) and Appendix N (Table 10, Figs. 19–20).

Reference

[1] Wang et al., 3DCodeEdit: Benchmarking Instruction-Guided 3D Program Editing, arXiv:2610.38231, 2026.

Feature Pass Rate

MethodG1G2G3G4StyleAllCLIP after edit
VLM harness1.000.800.800.730.670.8029.26
VIGA0.930.600.800.670.730.7526.37
InfiniDream (Ours)1.000.730.870.730.800.8329.86

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.