acceptodds
Desk rejected at ICLR 2027

WorldWeave: Growing Persistent Geometric Worlds for Video Generation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite rapid progress, world models still lack explicit, persistent structural memory, making it difficult to preserve consistent world structure during continual scene expansion and cross-view revisits. To address this limitation, we present WorldWeave, a world generation framework that decouples world-state maintenance from visual rendering. Specifically, WorldWeave combines continual elevation-map generation with agent-guided scene organization and stitching to build an expandable explicit 3D world that incrementally extends structural memory while preserving existing structure. First, its terrain module uses diffusion-based image outpainting to generate continuous metric elevation maps under neighborhood conditioning and boundary constraints. Next, an agent integrates user intent, terrain evidence, and cross-region connectivity constraints to construct scenes through hierarchical semantic planning, deterministic geometry compilation, and local revision. Finally, during visual generation, planned camera trajectories query world geometry through a read-only interface, producing depth sequences that guide video synthesis without writing the generated results back into the world state. As a result, structural memory remains independent of short-window video generation, enabling continual expansion without predefined map boundaries and providing a consistent geometric basis for observations across trajectories and repeated visits.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Generation, Agentic Scene Construction, Terrain Generation, Geometry-Guided Video Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eEZHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework, WorldWeave, that separates persistent 3D world construction from video generation. The method first uses a pretrained image editing model to complete terrain based on neighboring regions. Subsequently, an agent hierarchically plans region roles, district layouts, and asset relations. Deterministic geometry tools perform 3D asset placement, constraint checks, and local revisions, and validated additions are saved to the world state. During video generation, the system queries this world along a specified camera trajectory and renders depth sequences to guide the video model in generating frames. Across 120 tasks involving exploration and revisitation, the method demonstrates better structural consistency and camera compliance than the evaluated baselines.

Critical strengths
  1. The work constructs a relatively complete world construction and generation system, and demonstrates stronger consistency than existing world models on the benchmark. In addition, the visualizations also demonstrate the advantages of this agent-guided world construction method, especially its consistency in long-horizon tasks.

  2. Using an image generation model as a terrain generator, and then leveraging the capabilities of a VLM agent to construct a complete world, is a reasonable and novel idea. The experimental results also verify that this approach is feasible and effective. This coarse-to-fine world design is insightful.

Critical weaknesses
  1. The generated worlds still appear coarse and insufficiently detailed. In addition, since the agent system selects buildings, vegetation, rivers, and other assets to place from a predefined asset catalog, the generation diversity is limited by the asset catalog, and the system cannot generate complex scenes in the way that video generation methods can. In addition, the time cost of using an agent to construct such scenes is likely to be relatively high.

  2. During video rendering, only depth information is provided, which leads to the loss of texture and material details of the world scene. Specifically, when an object disappears from view for a relatively long period and then reappears, the consistency of its details cannot be guaranteed.

Questions for authors

What are the computational and time costs of agent-guided scene construction, and what is its success rate? How does the system handle failed construction cases, such as interpenetrating objects or poorly arranged layouts?

Reviewer_zXnaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldWeave, a world generation framework that decouples world-state maintenance from visual rendering. The framework keeps an explicit 3D world with terrain, object locations, region layouts, and connectivity. New regions are added to this world over time, while old regions are kept fixed. When generating a video, the camera queries the stored geometry and gets depth maps, which are then used to guide the video model. The paper evaluates WorldWeave on 120 tasks involving exploration, camera movement, occlusion, and revisiting earlier regions. The method performs well on structural consistency, revisit memory, and camera control, while keeping visual quality competitive.

Critical strengths
  1. Straight-forward and useful formulation of the method I think the main idea of this paper is clear and effective. It keeps an explicit world state instead of letting the video model to remember everything through its history. This addresses challenging problems such as revisiting.

  2. Strong experiments and ablations The authors compare several recent representative world models with their own, demonstrating that the proposed method is better. Also, experiments on terrain encoding, terrain joining, planner choice, and random-seed sensitivity, are also tested. I also think that the terrain ablation studies are very useful since they show that the propose residual encoding and boundary handling clearly improve metric terrain continuity.

Critical weaknesses
  1. The main baseline comparison is not exactly fair. WorldWeave gets depth from the constructed 3D world, but the baselines get the same first frame, camera path, and prompt but not the same geometric world representation. For the reported gains, many of the gains are from camera control or geometric consistency, so my concern is that stronger geometric input may explain a large portion of the gain.

  2. Human study is based on a small subset. The paper do the comparison on representative world models and video world models, which is good. But for the user study, the paper uses 20 scene-trajectory cases, which is a bit small considering the main experiments contain 120 tasks. I am a bit concerned as whether the small-scale human study is representative of the full benchmark.

Questions for authors
  1. How were the 20 user-study cases selected?

  2. How well does the model scale to much longer and larger worlds?

4 comments
AuthorsOpenReview ↗

Additional Human Evaluation with Newly Randomly Sampled Cases

Thank you for raising the question of case selection and representativeness in our user study. To further assess the robustness of the findings to case selection, we randomly sampled 20 scene–trajectory cases from the full set used in the main experiments and evaluated WorldWeave alongside six baseline methods, covering 140 videos. More than 42 participants took part in this evaluation. Each video received three valid evaluations, yielding 420 video evaluations and 1,260 individual criterion scores.

We retained the original three criteria—camera consistency, object consistency, and structure/texture consistency—using the same 1–5 rating scale. We first averaged the ratings for each video and then averaged across the 20 cases with equal weight. The results are shown below.

MethodCamera consistency ↑Object consistency ↑Structure/texture consistency ↑
SANA-WM3.001.731.75
Zing3.303.533.65
SolarWM-5B2.652.272.60
AlayaWorld3.573.173.05
EVOKE-Turbo3.502.823.07
Echo-WM4.354.023.93
WorldWeave4.524.534.52

With newly randomly sampled cases and a new round of human evaluation, WorldWeave again achieves the highest average score on all three criteria, consistent with the main findings of the original study. At the case level, WorldWeave achieves the highest or joint-highest score in 15/20 cases for camera consistency, 17/20 for object consistency, and 18/20 for structure/texture consistency. These results show that the observed advantages extend across most evaluated cases.

This additional experiment reproduces the main human-evaluation findings under renewed random case selection, providing further evidence of their robustness. We hope these new results address your concern about case selection and representativeness, and we would welcome your feedback on the additional evidence.

AuthorsOpenReview ↗

Final Clarifications and Additional Experimental Evidence

Thank you for your comments; as we have not yet received further feedback, we summarize our clarifications and additional evidence for your consideration.

  1. Comparison fairness. At the complete-system level, WorldWeave and all other compared methods share the same task inputs and output requirements: generating videos for the given tasks and evaluating them using the same criteria; depth is an intermediate result generated within WorldWeave, rather than additional ground-truth geometry supplied to our method (Sections 3.5 and 4.1).
  2. Additional human evaluation. The original 20 cases were sampled uniformly at random from the test set used for the main results table (Appendix C).To strengthen the human evaluation, we sampled 20 additional cases uniformly at random from the test set used for the main results table, bringing the total to 40 cases across two rounds, and evaluated the same seven methods on these cases with more than 42 participants in the additional round, yielding 140 videos and 420 valid video evaluations; WorldWeave again achieved the highest average scores on all three criteria—camera consistency 4.52, object consistency 4.53, and structure/texture consistency 4.52.
  3. Continuous world expansion. World construction is independent of the 15-second video observation window, and our added four-region experiment demonstrates a 512 m × 512 m world with 12,447 instances, all four internal seams validated, all 12 declared cross-region connections uniquely matched, and previously committed region records preserved (Section 4.3; Appendix H, Figure 12).

Thanks for the clear response, and I would like to raise my score.

Thank you for taking the time to read our response and reconsider your assessment. We appreciate your feedback, which helped us improve the paper.

Reviewer_BrsVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors notice that it is challenging to preserve a consistent world structure while continually expanding it and revisting previous region. Therefore, they propose WorldWeave, a framework for generating expandable worlds with persistent geometric structure. t separates world-state construction from video synthesis by maintaining terrain, scene layouts, and asset instances in a shared metric coordinate system. The terrain module uses diffusion-based outpainting, multiscale elevation encoding, and boundary constraints to extend terrain while preserving existing geometry. An agent further perform planing, and a solver compiles the plan into a spatial layout, with a geometric check and agent review to validate the quality. Experiments on 120 tasks demonstrate improved structural consistency, revisit memory, and camera compliance while maintaining competitive visual quality.

Critical strengths
  1. The paper addresses an essential problem for world models. Its separation of world structure from visual synthesis is well-motivated to mitigate inconsistency. Continual expansion and revisitation require a stable spatial reference. By maintaining terrain, building placements, and connectivity in an explicit 3D world, WorldWeave provides a concrete and verifiable mechanism for structural persistence.

  2. The framework is reasonable. Multiscale elevation-residual encoding helps preserve both subtle terrain variations and larger elevation changes, while boundary correction enforces height and slope continuity without modifying committed terrain; and the agent plans regions, districts, and asset relations, while deterministic tools enforce constraints on placement, support, collision clearance, and road connectivity.

  3. The evaluation is comprehensive. Apart from comparisons of generated videos, the paper examines elevation encoding, neighborhood input organization, and terrain joining, and includes planning-model substitution, random-seed sensitivity tests, and a user study.

Critical weaknesses
  1. Persistent geometry does not by itself ensure consistency in the generated videos. Although the world state preserves building placements and geometry, the video generator is primarily conditioned on depth. Identical depth guidance can still produce different textures, colors, and detaiersistent geometry not ensure video consistency. Video model mainly conditioned on depth, same depth can give different texture/color/detail when revisit. Authors admit depth is only guidance, not hard constraint.ls, allowing appearance to change across revisits. As the authors acknowledge, depth serves as guidance rather than a hard rendering constraint.

  2. Long-term expansion remains insufficiently evaluated. The main video evaluation comprises 120 tasks, each lasting 15 seconds, providing limited evidence about global connectivity, failure rates, and resource growth as many regions are added sequentially. Successful local terrain and road connections do not necessarily ensure the global coherence of a large, continually expanding world.

Questions for authors

The reliability of agent-based semantic validation remains unclear. Although deterministic checks enforce support, collision clearance, and connectivity, they do not fully establish whether regional roles, asset relations, and user intent have been interpreted correctly. Incorrect semantic decisions may therefore survive both geometric validation and agent review. It is better to show the mechanism to mitigate these types of errors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Discussion

WorldWeave separates world-state maintenance from video synthesis, enabling persistent structural memory beyond a single generation window. Its shared 3D world preserves existing geometry and spatial relationships during expansion, allowing cameras to revisit the same locations from different trajectories and viewpoints.

All three reviewers now support acceptance (Weak Accept), with zXna raising their score after the discussion. zXna recognized the explicit 3D representation and supporting terrain experiments; BrsV highlighted the structural preservation enabled by separating world representation from video synthesis; and eEZH appreciated the coarse-to-fine construction and long-range consistency.

During the discussion, we clarified the evaluation conditions and added experiments addressing the reviewers' concerns. WorldWeave achieved the highest mean ratings across all three consistency criteria on a fresh random sample. Layout experiments demonstrated conflict resolution while preserving existing structures, with all 14 cases satisfying regional roles and 13 satisfying asset relationships and overall intent. Landmark revisit experiments further showed improvements over the baseline on both appearance metrics, including a cross-segment revisit from a new viewpoint.

We sincerely thank the Area Chair and reviewers for their time, constructive feedback, and careful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.