acceptodds
Under review as a conference paper at ICLR 2027

Map-Grounded World Model for Urban Street Generation

Abstract

World models have made remarkable progress in generating realistic and interactive environments, yet long-horizon urban generation remains difficult because both scene structure and travel direction must remain consistent as the viewpoint moves. Maps naturally provide these two forms of naturally: a map describes the spatial structure of an environment, while a route provides vectors that define how the viewpoint should move through it. We introduce Vision–Map–Language (VML), a map-grounded framework for urban street generation. Given a map and a route, VML uses the map's structure to determine what should appear around the viewpoint and route vectors to determine where the viewpoint should move next. These signals are translated into coordinated camera motion, language, and visual conditions for a pretrained video generator. VML further projects map structure into the current view to guide generation, allowing the map to remain active throughout long-horizon exploration. We evaluate VML through 3 capabilities: exploring a selected map location, following a specified route, and generating surrounding scenes that reflect the mapped structure along that route. Our results demonstrate that maps can serve as a simple and reusable representation for organizing and navigating generative urban worlds.

Primary area:
generative models
Keywords:
spatial intelligence, 3d world model

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_vNouOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors present VML, a framework that generates long street-level videos from a semantic map, a route, and an appearance description, using a frozen pretrained video DiT. The method relies on two mechanisms:

(i) A map compiler compiles the route into camera motions, and the surrounding map structure into segment-level text descriptions, a map-compatible initial image, and optional visual anchors, which are given as conditioning to the generator.

(ii) A map-guided velocity correction at inference time. The map is projected into the current camera view to obtain a road/building/vegetation occupancy target, a small pretrained structural predictor estimates occupancy from the predicted clean latent, and the gradient of the mismatch is used to correct the DiT's predicted velocity at each denoising step.

Long videos are generated autoregressively by chaining 5-second segments while refreshing the map-derived conditions. The method is evaluated on route following, map-specified scene content, and structural alignment.

Critical strengths
  • Using maps as a persistent structural reference for long-horizon street generation is an interesting and practical idea, since maps are widely available and remain valid however far the viewpoint travels.
  • The guidance is lightweight and simple: it only requires a relatively small structural predictor.
Critical weaknesses
  • There are no external baselines: all results compare VML to its own variants without guidance or anchors, although map- and layout-conditioned street generation methods exist.
  • The evaluation is small-scale: structural alignment uses four procedural layouts with three seeds, scene realization is reported as success counts on a few seeds, and route following is measured by mean pixel displacement.
  • It's unclear which model computes the occupancy used for evaluation. If it is the same structural predictor used for guidance, the guidance directly optimizes the evaluation metric.
  • Visual quality is not evaluated (e.g. no FVD/FID).
Questions for authors
  • Could the authors compare against existing map- or layout-conditioned street generation methods?
  • Could the authors measure route following with a camera-pose estimator, e.g., the trajectory error between the intended and recovered camera paths?
  • Which model do the authors use to compute the occupancy for evaluation, and is it the same structural predictor used for guidance?
  • Could the authors report structural alignment measured with an independent segmentation model?
  • Could the authors report FVD/FID comparing guided and unguided generations?
  • Figure 6 is an empty placeholder. Could the authors provide the Figure?
Reviewer_zXjXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VML, a map-grounded framework that uses map structure and route vectors to condition a pretrained video generator and guide long-horizon urban street generation.

Critical strengths

1.The paper is clearly written and easy to follow, with a straightforward presentation of the overall framework.

2.The core idea of using maps as a persistent structural prior for long-horizon street generation is intuitive and practically motivated.

Critical weaknesses
  1. The main paper lacks comparisons against existing methods and relies almost entirely on internal ablations, making it unclear whether the gains hold under matched conditions.

  2. I think the contribution appears overstated. The method is essentially inference-time map-structure guidance plus condition compilation on a frozen video DiT, yet it is framed as a map-grounded world model that turns any street map into an explorable world, with claims exceeding the evidence.

  3. Figure 6 in the appendix has a compilation issue.

Questions for authors

See the Weaknesses.

Reviewer_nfsbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes VML, a map-assisted video generation framework for urban scene generation. The motivation is that foundation video generation models provide strong priors for realistic scene generation conditioned on camera motion. However, the generated content is not inherently constrained to be consistent with realworld. Realworld maps are thus introduced to extract semantic and spatial information to improve the consistency of generated scenes.

Critical strengths

None

Critical weaknesses
  1. Method Novelty and Rigorous.
  • Fair positioning. The idea that scene generation can benefit from information extracted from maps is widely explored[a][b]. However, none of these are discussed or compared, which raises concerns about the positioning of this work.
  • Limited novelty. The technical novelty of the proposed map-guided diffusion mechanism is limited.
  • Lack of technical details. The author doesn't clearly explain the data construction pipeline. It remains unclear how the initial image is established and be compatible with the starting location on the map. Also, how are the BEV semantic map and camera pose produce street-view semantic maps without height maps for all semantic classes.
  • Incomplete evaluation. The paper doesn't conduct any comparison with other video generation methods such as CameraCtrl[c] and MotionCtrl[d]. The evaluation lacks widely used content generation metrics such as FID and FVD. The paper claims to support long-horizon generation of up to 150 seconds, but doesn't provide qualitative results to support the claim.
  • Poor Results. The qualititive results exhibit blurness and unrecognizable content.
  1. Presentation.
  • The paper is full of typos (e.g., line 047 and 054);
  • Unexplained terms and confusing expressions (e.g., "paper" at line 040, "We demonstrate route and scene responses..." around line 82)
  • Confusing illustration in Figure 1. It's unclear why both block "Map Compiler" and "Map-Guided Diffusion Sampling" point to "Long-Horizon Generation" while only "Map-Guided Diffusion Sampling" points to "Output". It's unclear why the last segment points to the first segment in block "Long-Horizon Generation" which seems to indicate a recurrent generation. The structural predictor is not shown in this figure.
  • The content in related work subsection "Maps and Spatial Representations" is not closely related to this work. None of the map-assisted scene generation methods are discussed.
  • Figure 6 is missing.
  • Hallucinated reference of MotionCtrl.

Reference

  • [a] CityDreamer: Compositional Generative Model of Unbounded 3D Cities. In CVPR, 2024.
  • [b] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion. In SIGGRAPH, 2024.
  • [c] CameraCtrl: Enabling Camera Control for Video Diffusion Models. In ICLR, 2025.
  • [d] MotionCtrl: A Unified and Flexible Motion Controller for Video Generation. In SIGGRAPH, 2024.
Questions for authors
  • Could the authors provide more technical details of the map compiler?
  • Could the authors conduct complete experiments to support the claims?
Reviewer_msXvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Vision-Map-Language (VML), a framework for generating street-view videos from a semantic map, route, and appearance prompt without fine-tuning the pretrained video generator. VML uses a frozen Wan2.2-TI2V-5B / DreamX-World-5B-Cam video DiT together with a separately trained, fixed 0.375M-parameter structural predictor. A map compiler converts route geometry and map elements into camera control, segment-level text, initial visual conditions, and optional anchors. During sampling, map-projected structural targets guide the DiT velocity through gradients from the fixed predictor. In controlled comparisons on four procedural layouts (three seeds per layout), guidance reduces reported occupancy MSE from 0.267 to 0.150 (44.1%). A separate 150-second test on four geometries using one previously unused seed reports a decrease from 0.242 to 0.150 (37.9%). Across five additional seeds on a selected route, the full system satisfies 17/25 map-derived scene requirements.

Critical strengths
  1. The C-versus-G mechanism study is controlled: within each map-seed group, the initial image, camera trajectory, text prompt, sampler, and random seed are matched. A 44.1% reduction in the reported occupancy MSE over four procedural layouts supports the claim that sampling-time guidance improves agreement with the projected map target under the paper's evaluation protocol. This comparison usefully isolates the effect of modifying the denoising velocity.
  2. The modular design is practical. VML refreshes map-derived conditions along a route while keeping the large video backbone frozen, avoiding expensive end-to-end retraining of that backbone. The paper also acknowledges concrete failure modes and quality trade-offs, including reduced sharpness under guidance, larger segment-boundary differences with anchors, and unreliable reversal of building asymmetry in the map-reflection study. These observations help delimit the demonstrated control capabilities.
Critical weaknesses
  1. The main structural metric is not specified independently enough to support the strongest spatial-fidelity claims. Sampling-time guidance uses a fixed 0.375M-parameter predictor trained on SegFormer pseudo-labels, but Eq. (16) does not clearly define the evaluation readout or establish whether it is independent of this predictor. If the same readout is reused, some of the MSE reduction may reflect optimization of the evaluation proxy; decoded-RGB evaluation with an independent segmenter or annotations is needed to establish actual map consistency. The visual trade-offs are also material: in Table 2, G has only 0.750x the sharpness of C, while the anchor configurations increase boundary difference from 0.0125 (C) to 0.1342 (A) or 0.1267 (AG). Under Eq. (14), anchors substitute for the carried-over final frame when initializing the next segment, which breaks cross-segment continuity and explains this sharp jump at segment boundaries. Independent spatial and perceptual checks are needed to judge whether the proxy improvement outweighs these artifacts.
  2. The incremental algorithmic contribution and the generality of the evidence remain unclear. By the chain rule, the velocity gradient in Eq. (7) is proportional to the negative estimated-clean-latent gradient from Eq. (4). When the numerical-stability threshold in Eq. (8) is inactive, RMS normalization removes the noise-scale factor from that gradient's direction. This does not negate novelty in map projection or time-varying targets, but a matched direct clean-latent-guidance baseline is needed to isolate the value of the velocity parameterization; prior general-purpose guidance such as Universal Guidance (ICLR 2024) should be discussed. The paper also omits Streetscapes (SIGGRAPH 2024), which already studies map-conditioned autoregressive street generation, although its trained conditioning differs from VML's frozen-backbone approach. Finally, the main occupancy ablation covers four procedural geometries; the additional route and scene tests do not establish generalization to held-out real city maps or quantify route deviation.
  3. As a minor presentation note, Figure 6 in the submitted PDF renders as a file-path placeholder, so the reported guidance-strength sensitivity cannot be inspected. Table 2 also omits sharpness for A and AG. Please restore the figure and report these values if available.
Questions for authors
  1. How exactly is the own-target occupancy loss in Eq. (16) computed on generated video, and is that evaluation readout independent of the structural predictor used for inference-time guidance? If feasible, please report map-alignment metrics for road, building, and vegetation on decoded RGB frames using an independent evaluator.
  2. Given the chain-rule relationship between the velocity gradient and clean-latent gradient in Eqs. (4), (7), and (8), what benefit does velocity correction provide over direct clean-latent guidance with matched backbone, sampler, denoising steps, and computation? Please compare structural alignment and image sharpness under that control.
  3. How are camera intrinsics, assumed building heights, projection smoothing, structure-visibility intervals, and anchor-screening thresholds selected? On the existing route experiments, can you quantify route adherence or camera-pose deviation, or explain why the reported optical-flow turn contrasts are sufficient to validate following the specified map route?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.