acceptodds
Under review as a conference paper at ICLR 2027

Learning a Structured Latent Space for Vectorized HD Maps

Abstract

Vectorized HD maps provide structured geometric and semantic descriptions essential for autonomous driving. As autonomous systems advance toward generative world models and simulation platforms, maps are increasingly required to function as learnable, generative interfaces rather than static offline assets. However, they naturally take the form of variable-length, heterogeneous, and unordered sets of geometric primitives, such as lane polylines and road polygons. Integrating such irregular vector collections into modern continuous generative models presents a fundamental structural challenge. Existing approaches typically struggle with a trade-off: imposing heuristic one-dimensional sorting on primitives induces semantic drift that collapses outputs into dataset-average straight corridors, whereas uncoordinated set diffusion fails to organize global scene layouts, yielding trivial, mutually parallel paths. To resolve this dilemma, we propose MapVAE, a hierarchical autoencoding framework that decouples local geometric modeling from global scene aggregation into a compact continuous latent space with canonical slot identities. In the first stage, a unified Vector Autoencoder projects heterogeneous primitives onto a shared hyperspherical manifold using canonical local coordinates, faithfully preserving fine-grained curvatures without category-specific backbones. In the second stage, a Map Autoencoder aggregates these element-level features into a fixed-length bottleneck via learnable queries, establishing structured slot representations that anchor global layout and directional diversity for downstream continuous generation. Crucially, distinct from existing works that decode raw geometries, we formulate the Map Autoencoder as a DETR-style set reconstruction directly in feature space, which unifies heterogeneous primitive recovery into a shared metric and improves optimization stability via unit-hypersphere constraints, a discriminative triplet loss, and spherical geodesic denoising. Extensive experiments demonstrate that MapVAE achieves superior reconstruction fidelity and enables our diffusion model, MapLDM, to synthesize diverse, geometrically precise HD maps across complex urban scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
HD Map, Map Representation, Map Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rmwNOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MapVAE, a hierarchical autoencoder that learns a structured latent space for vectorized HD maps, enabling diffusion models to generate realistic road networks while preserving fine-grained geometry and global layout consistency.

Critical strengths

Strong reconstruction results and comprehensive ablation studies.

The method shows substantial improvements over Scenario Dreamer, reducing lane reconstruction RMSE from 0.462 m to 0.180 m and improving curvature fidelity from 0.593 to 0.885. The paper also provides extensive ablations covering most major design choices, which helps demonstrate the effectiveness of the proposed components.

Critical weaknesses
  1. Insufficient comparisons with representative prior methods.

The paper mainly compares against Scenario Dreamer (CVPR 2025), while other relevant HD map generation methods, such as HDMapGen (CVPR 2021) and SLEDGE (ECCV 2024), are discussed in the Related Work section but excluded from the experiments. Since these methods explore different generation approaches, including graph-based generation and raster-based latent diffusion, the limited baseline comparison makes it difficult to assess whether the proposed structured latent representation offers a meaningful advantage over existing alternatives.

  1. Lack of evaluation on public benchmarks.

All experiments are conducted on a private dataset, without any evaluation on established public benchmarks. While the dataset is large and diverse, the authors do not clearly explain why public datasets are excluded. This makes it difficult to independently reproduce the results, verify the reported improvements, or assess how well the proposed method generalizes beyond the authors’ internal data distribution. This is particularly concerning given the paper’s broader claims about learning a general-purpose structured latent representation for vectorized HD maps.

Questions for authors

Q1 Missing baseline comparisons

Given that HDMapGen and SLEDGE (mentioned in related work as previous sota) have publicly available implementations, could the authors explain why these representative graph-based and raster-latent generation methods are excluded from the quantitative comparisons?

Q2 Evaluation on public benchmarks

All experiments are conducted on a private dataset. Could the authors clarify why no experiments are provided on established public benchmarks, such as nuPlan or nuScenes? This makes it difficult to independently verify the reported improvements and compare against existing methods.

Reviewer_MwSbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MapVAE, a two-stage autoencoder that converts vectorized HD maps, i.e., variable-length and unordered sets of lane polylines and road-object polygons, into a fixed-size latent for diffusion-based generation. Stage 1 encodes each map element into a normalized feature vector. Stage 2 compresses these features into a fixed number of learnable slots and reconstructs them with a DETR-style decoder, aided by a spherical denoising branch. A diffusion transformer (MapLDM) is then trained on this latent to generate new maps. On a private dataset, the method reports better reconstruction and generation statistics than Scenario Dreamer.

Critical strengths
  • The paper addresses a relevant problem: turning variable-length, unordered vector maps into a fixed-size latent that standard diffusion models can handle, which is useful for generative driving simulation.

  • Reconstructing elements in a normalized feature space with a DETR-style decoder is a reasonable design that handles polylines and polygons in a unified way. The spherical denoising branch is particularly effective as demonstrated by the results in Table 9.

  • The reconstruction side is thoroughly ablated。

Critical weaknesses
  • Limited generation evaluation. The main paper contains a single quantitative generation table and no qualitative generation result (Fig. 9 appears only in the appendix). Most ablations report reconstruction metrics, so it is unclear how some components, e.g., vMF sampling and the polar parameterization, affect generation quality.

  • Limited discussion of the reported statistics. The paper reports a rich set of generation statistics (Table 1, Tables 14–15), but these are discussed only briefly, mainly through the aggregated Fréchet distances. Several observations in the detailed tables would benefit from further discussion. For example, MapLDM appears to shift the curvature distribution relative to the ground truth (straight lanes: 0.53 vs. 0.78; mildly curved lanes: 0.31 vs. 0.10), whereas the variant without Stage 2 is closer on these categories and achieves a lower F_geo. Similarly, MapLDM's nearest-neighbor angle (0.11 vs. GT 0.07) deviates by a margin comparable to that of the variant without Stage 2 (0.03). In addition, the controllability statement is not accompanied by a count-error measurement.

  • Single baseline on private data. Generation is compared only against Scenario Dreamer, retrained on a private dataset at a larger crop size than its original setting. No results are reported on public datasets such as nuPlan or Waymo.

  • Presentation. Sec. 4.2 lists six latent analyses without stating which claim each one is meant to support. Key results are placed in the appendix, which makes the paper hard to follow.

Questions for authors
  • Could the authors report generation metrics for components whose effect on generation is less obvious, such as vMF sampling (which improves Stage 2 AP but degrades Stage 1 reconstruction, Table 6) and the polar parameterization??

  • Could the authors discuss the curvature shift of MapLDM in Table 14 and its nearest-neighbor angle in Table 15, in comparison with the variant without Stage 2?

  • The controllability statement in Sec. 4.3 refers to "minimal count deviation (Table 1)", but Table 1 reports only the mean and standard deviation of generated counts. Could the authors clarify how the count conditions were set in this evaluation and report the error between the requested and the generated counts?

  • Would it be possible to report results on a public dataset (e.g., nuPlan or Waymo) in Scenario Dreamer's original setting? Even a smaller-scale comparison would be helpful.

  • The paper states that the 500×500 m tiles are sampled at 100-meter intervals along driving trajectories. Could the authors clarify whether tiles overlap with each other, and whether the train/validation split is geographically disjoint so that validation tiles do not overlap with training tiles?

15 comments

We sincerely thank you for your constructive review. We have substantially revised the manuscript (highlighted in blue) to address every point you raised.


Summary of Manuscript Restructuring & Presentation Improvements (Addressing W1 & W4)

Following your feedback on Presentation (W4) and Generation Evaluation in the Main Text (W1), we overhauled the manuscript structure so that the main paper is self-contained and easy to follow:

  1. Reorganized Section 4 & Brought Visuals/Ablations to the Main Text: We restructured the experiments to lead with Map Generation (Sec. 4.2, expanding from one table to Tables 1 & 2 across public and private benchmarks), followed by Reconstruction and Core Ablations (Sec. 4.3). Crucially, we moved the qualitative generation comparisons (new Fig. 2), multi-density reconstruction visualizations (new Fig. 3), and a consolidated core ablation table (new Table 3) from the Appendix directly into the main paper.
  2. Claim-Driven Latent Analysis (Addressing W4): We rewrote Sec. 4.4 (previously Sec. 4.2) by replacing the six fragmented bullet points with three explicit structural claims directly verified by Fig. 4 (previously Fig.3): (1) Automatic Capacity Adaptation via Norm Sparsity, (2) Query Specialization without Mode Collapse, and (3) Scale-Invariant Intrinsic Dimensionality across Scene Densities.
  3. Unified Pipeline Figure & Focused Scope: We merged the previous Figures 1 and 2 into a single architecture overview (new Fig. 1) and condensed the Abstract and Introduction to focus strictly on vectorized HD map generation.

Point-by-Point Answers

1. Public Benchmark (nuPlan) & Additional SOTA Baseline (SLEDGE) [Addressing W3 & Q4]

We evaluated MapLDM on the public nuPlan benchmark in Scenario Dreamer's original 64×64m64\times64\text{m} setting, and added SLEDGE [Chitta et al., 2024] as a second external baseline across both datasets (Sec. 4.2, Tables 1 & 2, Fig. 2):

  • Public nuPlan Benchmark in Scenario Dreamer's Original Setting (Table 1): Following the exact data split (450k training / 50k validation frames, 64×64m64\times64\text{m} FOV) and evaluation protocol of SLEDGE and Scenario Dreamer, we attached lightweight inner-product connectivity heads to our decoders (App. A.2). As shown in Table 1, MapLDM (Small, 234M) outperforms SLEDGE (539M/769M) across all urban planning Fréchet metrics (Conn.=0.12\text{Conn.}=0.12 vs. 1.671.67, Reach=0.13\text{Reach}=0.13 vs. 0.510.51) and endpoint distance (ED=0.28m\text{ED}=0.28\text{m} vs. 0.42m0.42\text{m}) with less than half the parameters. Compared with Scenario Dreamer (377M/679M), MapLDM achieves comparable distribution-level fidelity while substantially reducing the endpoint distance standard deviation (±0.20m\pm 0.20\text{m} vs. ±0.71∼0.80m\pm 0.71\sim0.80\text{m}), reflecting more consistent topological alignment at segment boundaries.
  • Why 1D Sorting Works on nuPlan but Fails on Real Physical Maps: In Sec. 4.2, we explain why Scenario Dreamer does not collapse on nuPlan. First, nuPlan preprocesses lane graphs into a few long, merged centerlines per patch, keeping 1D coordinate order relatively stable, whereas our real-world physical maps contain dozens of short, fragmented markings and polygons whose sorted indices frequently swap under minor spatial shifts. Second, nuPlan patches are predominantly ego-aligned along Manhattan-like grids, whereas our physical maps exhibit arbitrary road headings and are trained with random planar rotations, which breaks fixed-axis 1D sorting and requires our permutation-invariant slot bottleneck.
  • Additional External Baseline (SLEDGE) on the Private Dataset (Table 2 & Fig. 2): We trained SLEDGE (DiT-XL, 673M) on our physical map dataset. Because SLEDGE encodes vectors into a 2D raster latent grid, it avoids 1D sorting collapse and achieves better directional diversity than Scenario Dreamer (Fdiv=0.86F_{\text{div}}=0.86 vs. 1.521.52). However, rasterization introduces spatial discretization errors (Fgeo=2.81F_{\text{geo}}=2.81 vs. our 1.181.18) and lane over-population (Flay=1.10F_{\text{lay}}=1.10, generating 82.7382.73 lanes vs. GT 48.8048.80).

2. Discussion of Curvature Shift & Nearest-Neighbor Angle vs. "DiT without Stage 2" [Addressing W2 & Q2]

We expanded our statistical analysis in Sec. 4.2 and Appendix B.4 (Tables 16 to 18) to explain both observations in detail:

  • Why "DiT without Stage 2" has a lower FgeoF_{\text{geo}} (0.630.63) and 0.03 rad0.03\text{ rad} NN Angle: Diffusing uncompressed Stage 1 tokens without a scene-level bottleneck lacks global layout coordination. As shown visually in Fig. 2(c), the element-level denoiser takes an optimization shortcut by generating locally smooth, mutually parallel straight lines regardless of global road structure. This parallel-line collapse causes its mean nearest-neighbor angle to drop to 0.03 rad0.03\text{ rad} (nearly parallel, vs. GT 0.07 rad0.07\text{ rad}) and artificially inflates its straight-lane ratio (0.690.69), yielding a lower FgeoF_{\text{geo}} while failing to form valid intersections or multi-directional layouts (Fdiv=1.09F_{\text{div}}=1.09, Flay=0.28F_{\text{lay}}=0.28). In contrast, MapLDM actively generates multi-branch intersections, ramps, and turning roadways (Fig. 2d), resulting in a slightly higher nearest-neighbor angle (0.11 rad0.11\text{ rad}) and substantially better global diversity (Fdiv=0.71,Flay=0.13F_{\text{div}}=0.71, F_{\text{lay}}=0.13).
  • Why MapLDM Shifts Between "Straight" and "Mild" Curvature Bins: In Table 16, our curvature categories use a strict hard threshold of <0.1 rad<0.1\text{ rad} (≈5.7∘\approx 5.7^\circ) cumulative turning angle across the entire polyline for the "Straight" bin, and [0.1,0.5) rad[0.1, 0.5)\text{ rad} for the "Mild" bin. Because Stage 2 predicts continuous Stage 1 latents h^i∈Sd−1\hat{\boldsymbol{h}}_i \in \mathbb{S}^{d-1} trained with spherical perturbation κ\kappa, tiny angular residuals in feature space decode into slight, smooth geometric undulation that crosses the strict 0.1 rad0.1\text{ rad} threshold from "Straight" (0.530.53 vs. GT 0.780.78) into "Mild" (0.310.31 vs. GT 0.100.10). Crucially, when summing the "Straight + Mild" ratios, MapLDM matches ground truth closely (0.840.84 vs. GT 0.880.88), and its mean maximum turning angle is identical to ground truth (0.31±0.88 rad0.31 \pm 0.88\text{ rad} vs. GT 0.31±0.88 rad0.31 \pm 0.88\text{ rad} in Table 16).

3. How Power Spherical Sampling and Polar Parameterization Affect Generation [Addressing W1 & Q1]

Thank you for this thoughtful observation regarding the trade-off between Stage 1 and Stage 2 metrics. Because training separate MapLDM models from scratch for each autoencoder variant is computationally expensive and time-consuming, we could not complete additional DiT training runs within the short rebuttal period. More importantly, we would like to clarify the exact roles of these two components, as both are designed to ensure a well-behaved Stage 2 autoencoder bottleneck, which acts as the performance upper bound for downstream generation, rather than directly tuning the DiT itself:

  • Power Spherical Sampling Builds Error Tolerance Between Stage 1 and Stage 2. As you astutely noted in Table 3 (formerly Table 6), deterministic Stage 1 training achieves a lower standalone Stage 1 RMSE (0.096m0.096\text{m} vs. 0.180m0.180\text{m}), yet it causes Stage 2 reconstruction AP to drop sharply by 27.4%27.4\% (94.04%→66.62%94.04\% \to 66.62\% for lanes, and 83.44%→56.73%83.44\% \to 56.73\% for objects). Without spherical perturbation during Stage 1 training, the frozen decoder DvecD_{\text{vec}} overfits to exact encoder features and fails whenever Stage 2 outputs contain minor prediction residuals. Because MapLDM generates maps strictly by decoding sampled latents through the frozen Stage 2 and Stage 1 decoders, a bottleneck that already loses over a third of map elements during reconstruction (66.62%66.62\% AP) directly caps the completeness of any generated map.
  • Polar Parameterization Is Designed for Latent Sparsity Rather Than Direct Generation Gains. We want to candidly clarify that our polar mean decoupling (μ=r⋅μˉ/∥μˉ∥2\boldsymbol{\mu} = r \cdot \bar{\boldsymbol{\mu}}/\|\bar{\boldsymbol{\mu}}\|_2) was not introduced as a trick to boost downstream DiT generation metrics, but rather to achieve explicit norm sparsity and automatic capacity adaptation within the fixed-size bottleneck. Because real-world maps have highly variable element counts, a fixed-slot autoencoder needs a clean mechanism to deactivate redundant slots in simpler scenes. As shown in the comparison against a standard Cartesian VAE (Appendix B.3.2 and Fig. 7), Cartesian parameterization leaves unused slots scattered at arbitrary non-zero norms. By decoupling the radius rr from the unit direction, our polar formulation simplifies the Gaussian KL mean penalty directly to r2r^2, naturally driving redundant slots to the origin (r→0r \to 0, Fig. 4a, d) while slightly improving Stage 2 reconstruction AP (94.04%94.04\% vs. 91.97%91.97\% for lanes). Having a clean, sparse latent representation where active and inactive slots are clearly separated is an inherent structural property of MapVAE.

4. Count Conditioning Protocol & Requested-vs-Generated Count Analysis [Addressing W2 & Q3]

We have clarified the conditioning protocol in Sec. 4.2 and added a dedicated scatter and marginal distribution analysis comparing requested ground-truth counts against generated counts (Fig. 8):

  • How Count Conditions Are Set During Evaluation. When generating the validation samples in Table 2, each sample's target condition (clane,cobj)(c_{\text{lane}}, c_{\text{obj}}) is sampled directly from the ground-truth element counts of the validation scenes. This ensures that the requested condition distribution matches the true validation distribution shown in the top marginal histograms of Fig. 8.
  • Relationship Between Requested and Generated Counts. As shown in Fig. 8(a, b), across the primary data regime where the vast majority of scenes reside (clane≤100c_{\text{lane}} \le 100 and cobj≤50c_{\text{obj}} \le 50, indicated by the top histograms), the generated counts tightly track the ideal y=xy=x identity line, with the sample means intersecting almost exactly on the diagonal (44.7944.79 vs. GT 48.7948.79 for lanes, and 9.669.66 vs. GT 10.4310.43 for road objects). In the extreme high-density tail (clane>150c_{\text{lane}} > 150), the generated counts exhibit a smooth sub-linear saturation below the diagonal, which accounts for the slight difference in overall mean and standard deviation (σ=31.92\sigma=31.92 vs. GT 39.4739.47). Notably, this saturation mirrors the active-slot recruitment curve in our latent bottleneck analysis (Fig. 4d): as total scene complexity approaches the K=256K=256 slot capacity, active slot recruitment gradually plateaus, naturally bounding the maximum decoded instance count in ultra-dense scenes.
Reviewer_JizkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MapVAE, a hierarchical latent framework for generating vectorized HD maps. A Vector Autoencoder maps each lane polyline or road polygon onto a unit hypersphere in a canonical local frame; a Map Autoencoder aggregates element latents into a fixed-length bottleneck with learnable queries and a DETR-style feature-space decoder; MapLDM then diffuses this bottleneck with a DiT (v-prediction), conditioned on lane/object counts via adaLN.Ablations are thorough, but validation is narrow: no public benchmark, one external baseline, and local lane geometry degrades relative to the bottleneck-free ablation.

Critical strengths

1.The problem is well motivated. Heuristic sorting causes index drift and straight-road collapse, while unordered set diffusion fragments global layout; the two internal comparison arms match these failure modes.

2.The element representation is thoughtful. The canonical local frame separates pose from shape, closed polygons are invariant to cyclic shifts and reversal, and curvature-weighted smoothness and connectivity losses target failure modes that raw MSE misses.

3.The model handles road objects in addition to lane polylines, which the baseline lacks; count conditioning and qualitative results cover ramps, junctions, and object-rich scenes.

4.The presentation is organized, with an AI use statement, a reproducibility statement, and most loss definitions moved to the appendix.

Critical weaknesses

1.Evaluation rests on a private dataset with one external baseline. There is no public benchmark result, no comparison with other generative map works, and no alternative latent formulation as a controlled comparison

2.The Stage-2 bottleneck worsens local generation geometry versus the bottleneck-free ablation. MapLDM Fgeo is 1.23–1.21 against 0.63 without Stage 2; MapLDM over-produces mildly curved lanes, under-produces straight ones, and falls short on mean lane length. The paper does not diagnose whether information is lost in compression, diffusion, or hierarchical decoding

3.Novelty is mostly compositional. Hyperspherical normalization, the Power Spherical posterior, DETR matching, denoising queries, anchor queries, polar sparsity, and latent DiT diffusion are established techniques combined for maps

4.The world-model and long-horizon motivation outruns the delivered scope. The model produces isolated single frames conditioned only on counts, while temporal continuity is future work.

Questions for authors

1.MapLDM Fgeo is 1.18–1.23 against 0.63 for the bottleneck-free arm. Please split the gap between bottleneck compression, diffusion sampling, and hierarchical decoding, and report reconstruction metrics from ground-truth versus reconstructed latent. Would a skip path restoring local geometry preserve layout coordination?

2.Generalization and robustness are not characterized in the current evaluation. Validation scenes come from the same metropolitan regions and scene distribution as training, so in-distribution Fréchet scores and AP do not establish robustness under shift. Could you define a robustness protocol along natural axes—held-out metropolitan areas, unseen topology classes such as roundabouts, ramp interchanges, and dense urban blocks, and element-count extremes—and report how Fgeo, Fdiv, Flay, and reconstruction AP degrade on these splits? This would also test whether the fixed-slot bottleneck generalizes beyond the training distribution rather than memorizing local layouts

3.The present metrics measure distributional closeness but not downstream utility. Could you evaluate generated maps inside a planner or simulator, reporting planning success and trajectory violations caused by hallucinated topology, missing junctions, and disconnected lanes? Please compare MapLDM with other SOTA generators under the same distribution shifts on both these planner-level metrics and the existing Fréchet statistics.

6 comments

Dear Reviewer,

We sincerely thank you for your constructive feedback. To thoroughly address your comments on Presentation, Scope, and Experimental Validation, we have conducted a major restructuring of the main manuscript alongside new experiments on the public nuPlan benchmark and an additional representative baseline (SLEDGE). All revisions are highlighted in blue in the updated PDF.


Summary of Manuscript Restructuring & Presentation Improvements

Following your feedback, we overhauled the manuscript structure to make it self-contained, focused, and easy to follow:

  1. Refocused Abstract & Introduction (Addressing W4): We substantially condensed the Abstract and Introduction, removing overclaimed "world model" and "long-horizon rollout" motivations. The narrative now focuses strictly on our core contribution: resolving the dilemma between local geometric precision and global layout coordination when bridging variable-length vector sets with continuous diffusion models.
  2. Unified Framework Overview (Figure 1): We redrew and merged the previous Figures 1 and 2 into a single, comprehensive architecture diagram (new Fig. 1).
  3. Self-Contained Main Text with Visuals & Core Ablations: We reorganized Section 4 to lead with generation results (Sec. 4.2), followed by a new Reconstruction and Core Ablations section (Sec. 4.3) and a streamlined Latent Structure Analysis (Sec. 4.4) organized around three explicit claims rather than fragmented bullet points. Crucially, we moved the generation and reconstruction visualizations (new Fig. 2 & Fig. 3) and a consolidated core ablation table (new Table 3) from the Appendix directly into the main paper.

Point-by-Point Responses to Weaknesses & Questions

1. Public Benchmark (nuPlan) & Additional Baseline (SLEDGE) [Addressing W1]

We have added two major comparisons in Sec. 4.2 (Tables 1 & 2, Fig. 2):

  • Evaluation on Public nuPlan Benchmark (Table 1): Following the exact protocol of SLEDGE and Scenario Dreamer, we trained and evaluated MapVAE and MapLDM on the public nuPlan benchmark (64×64m64\times64\text{m} FOV, 450k train / 50k val frames). As shown in Table 1, MapLDM (Small, 234M) outperforms SLEDGE (539M/769M) across all urban planning Fréchet metrics (e.g., Conn.=0.12\text{Conn.}=0.12 vs. 1.671.67, Reach=0.13\text{Reach}=0.13 vs. 0.510.51) while using less than half the parameters. Against Scenario Dreamer (377M/679M), MapLDM achieves competitive distributional fidelity while substantially reducing the endpoint distance (ED) standard deviation (±0.20m\pm 0.20\text{m} vs. ±0.71∼0.80m\pm 0.71\sim0.80\text{m}), demonstrating more consistent boundary alignment.
  • Comparison with Raster-Latent Formulation (SLEDGE) on the Internal Dataset (Table 2 & Fig. 2): To compare against an alternative latent formulation, we trained SLEDGE (DiT-XL, 673M), which projects vectors onto a regular 2D raster latent grid, on our large-scale physical map dataset. As shown in Table 2 and Fig. 2, SLEDGE avoids 1D sorting collapse and achieves better directional diversity than Scenario Dreamer (Fdiv=0.86F_{\text{div}}=0.86 vs. 1.521.52), but suffers from raster discretization errors (Fgeo=2.81F_{\text{geo}}=2.81 vs. our 1.181.18) and severe lane over-population (Flay=1.10F_{\text{lay}}=1.10, generating 82.7382.73 lanes vs. GT 48.8048.80).
  • Why 1D Sorting Works on nuPlan but Fails on Real Physical Maps: In Sec. 4.2, we explicitly analyze this contrast: (1) nuPlan preprocesses lane graphs into a few long, merged centerlines, keeping 1D sorting relatively stable, whereas physical maps contain dozens of short, fragmented markings and polygons whose sorted indices frequently swap; (2) nuPlan patches are predominantly ego-aligned along Manhattan-like grids, whereas our physical maps span arbitrary headings with random planar rotations, breaking fixed-axis 1D sorting and necessitating our permutation-invariant slot bottleneck.

2. Diagnosing the FgeoF_{\text{geo}} Trade-off & Stage 2 Bottleneck [Addressing W2 & Q1]

This is an insightful question. To isolate whether the geometric shift occurs during Stage 2 bottleneck compression/hierarchical decoding or during DiT diffusion sampling, we evaluated the reconstructed maps decoded directly from ground-truth MapVAE Stage 2 latents, which is reported as MapVAE (upper bound) in the updated Tables 2, 16, and 17:

  1. Why "DiT without Stage 2" has a lower FgeoF_{\text{geo}} (0.630.63) but fails globally: As shown in Table 17 and Fig. 2(c), diffusing uncompressed Stage 1 tokens without a scene bottleneck yields a collapsed mean nearest-neighbor angle of 0.03 rad0.03\text{ rad} (vs. GT 0.07 rad0.07\text{ rad}) and fragmented layouts. Without global coordination, the element-level denoiser takes the optimization shortcut of generating isolated, mutually parallel straight lines, which artificially inflates the straight-lane ratio (0.690.69) and lowers FgeoF_{\text{geo}} at the expense of valid intersection topologies (Fdiv=1.09F_{\text{div}}=1.09).
  2. Splitting the Gap (Stage 2 Compression/Decoding vs. DiT Diffusion Sampling): Comparing Ground Truth, MapVAE (upper bound), and MapLDM across Tables 2, 16, and 17 cleanly separates the two sources of error:
    • Effect of Stage 2 Compression & Hierarchical Decoding (GT →\to MapVAE upper bound): Reconstructing from ground-truth Stage 2 latents preserves global layout, directional diversity, and scale almost perfectly: mean lane length (24.97m24.97\text{m} vs. GT 24.53m24.53\text{m}), max turning angle (0.32 rad0.32\text{ rad} vs. GT 0.31 rad0.31\text{ rad}), and nearest-neighbor angle (0.07 rad0.07\text{ rad} vs. GT 0.07 rad0.07\text{ rad}, Fdiv=0.27,Flay=0.19F_{\text{div}}=0.27, F_{\text{lay}}=0.19) match ground truth with minimal loss. However, FgeoF_{\text{geo}} moves from 00 to 0.920.92 primarily because the "Straight" bin (<0.1 rad≈5.7∘<0.1\text{ rad} \approx 5.7^\circ cumulative turn) shifts from 0.780.78 to 0.590.59 while the "Mild" bin ([0.1,0.5) rad[0.1, 0.5)\text{ rad}) rises from 0.100.10 to 0.240.24. This shows that 78% of MapLDM's FgeoF_{\text{geo}} (0.920.92 out of 1.181.18) stems from hierarchical decoding across the strict 0.1 rad0.1\text{ rad} threshold: small angular residuals on Sd−1\mathbb{S}^{d-1} decode into tiny, smooth polyline undulations that cross the 0.1 rad0.1\text{ rad} boundary without altering true road curvature (Straight + Mild sum remains 0.830.83 vs. GT 0.880.88).
    • Effect of DiT Diffusion Sampling (MapVAE upper bound →\to MapLDM): Moving from reconstructed Stage 2 latents to diffused latents accounts for the remaining 0.260.26 gap in FgeoF_{\text{geo}} (0.92→1.180.92 \to 1.18), a slight further shift in Straight/Mild ratios (0.59→0.540.59 \to 0.54 and 0.24→0.300.24 \to 0.30), and the reduction in mean lane length (24.97m→21.84m24.97\text{m} \to 21.84\text{m}) as well as the increase in FdivF_{\text{div}} (0.27→0.710.27 \to 0.71). Thus, lane shortening is not caused by the Stage 2 bottleneck (which preserves full length at 24.97m24.97\text{m}), but rather by the smoothing variance of latent diffusion sampling.
  3. Would a Skip Path Restoring Local Geometry Preserve Layout Coordination? Because generation begins strictly from pure Gaussian noise in the K×DK \times D Stage 2 space (where ground-truth Stage 1 element tokens do not exist at inference time), a standard encoder-to-decoder skip connection bypassing Stage 2 cannot be directly used during generation. However, in our ongoing exploration, we have found that injecting richer auxiliary control signals during the generation process effectively helps restore fine-grained local geometry while preserving global layout coordination. We are actively investigating this direction and will present these findings in our future work.

3. Novelty & Domain-Specific Technical Contributions [Addressing W3]

While individual building blocks draw inspiration from established literature, adapting DETR-style set prediction to serve as a continuous generative autoencoder bottleneck in feature space is non-trivial and fails without our domain-specific designs: As proven in Table 3, standard DETR query matching in latent feature space collapses to 47.35%47.35\% lane AP and 35.70%35.70\% object AP. To make feature-space set autoencoding viable, we had to co-design: (1) Spherical Geodesic Denoising along tangent arcs on Sd−1\mathbb{S}^{d-1} (which boosts AP by +46.7%+46.7\% to 94.04%94.04\%), coupled with (2) Stage 1 Power Spherical perturbation and triplet margin separation (without which Stage 2 AP drops by 27.4%27.4\% due to zero error tolerance during feature matching), and (3) Polar mean decoupling (μ=r⋅μˉ/∥μˉ∥2\boldsymbol{\mu} = r \cdot \bar{\boldsymbol{\mu}}/\|\bar{\boldsymbol{\mu}}\|_2), which reduces the KL mean penalty to r2r^2 to naturally deactivate redundant slots (r→0r\to 0) while preserving directional subspace specialization (verified via RBF-MMD in Fig. 4).

4. Robustness Across Geographically Disjoint Splits & Scene Complexity Regimes [Addressing Q2]

To evaluate generalization and robustness without layout memorization, we first clarified our geographic splitting protocol in Sec. 4.1 and then conducted a stratified evaluation across element-count extremes in Appendix B.4 (Tables 18, 19, and 20):

  • Geographically Disjoint Validation Split. As clarified in Sec. 4.1, our training (149,428 scenes) and validation (29,263 scenes) splits are partitioned by geographically disjoint urban trajectories and districts with zero spatial overlap. Because our vectorized physical map dataset does not contain semantic scene-level tags (such as explicit labels for roundabouts or ramp interchanges), we established a stratified complexity protocol along the natural axis of lane density: Simple (10≤clane<3010 \le c_{\text{lane}} < 30, predominantly straight arterial roads with mean length 40.79m40.79\text{m}), Moderate (30≤clane<6030 \le c_{\text{lane}} < 60, standard urban intersections with mean length 25.26m25.26\text{m}), and Complex (clane≥60c_{\text{lane}} \ge 60, averaging 100.19100.19 lanes and 23.4523.45 objects per scene, capturing dense multi-way intersections, fragmented markings, and ramp-like junctions with short 19.88m19.88\text{m} segments and higher directional entropy 1.291.29).
  • Bottleneck Robustness Across Complexity Regimes (MapVAE upper bound). As reported in Tables 18 to 20, the fixed-slot bottleneck (K=256K=256) generalizes smoothly from sparse to extreme-density scenes without memorizing fixed layouts. In Simple and Moderate regimes, MapVAE (upper bound) achieves near-zero layout discrepancy (Flay=0.02F_{\text{lay}}=0.02 and 0.040.04) and matches ground-truth lane lengths accurately (41.28m41.28\text{m} vs. GT 40.79m40.79\text{m} in Simple; 25.31m25.31\text{m} vs. GT 25.26m25.26\text{m} in Moderate). In the Complex regime (clane≥60c_{\text{lane}} \ge 60), although active slot saturation slightly increases FgeoF_{\text{geo}} (0.61→1.030.61 \to 1.03) and FlayF_{\text{lay}} (0.02→0.370.02 \to 0.37, reconstructing 91.0091.00 vs. GT 100.19100.19 lanes), MapVAE (upper bound) actually achieves its best directional diversity (Fdiv=0.18F_{\text{div}}=0.18 vs. 0.390.39 in Simple) while preserving exact mean segment length (19.83m19.83\text{m} vs. GT 19.88m19.88\text{m}) and nearest-neighbor angle (0.07 rad0.07\text{ rad} vs. GT 0.06 rad0.06\text{ rad}).
  • Generation Stability Under Extreme Density Shifts (MapLDM). Across all three complexity splits, MapLDM (B) maintains remarkably stable geometric and directional fidelity, with FgeoF_{\text{geo}} remaining virtually invariant (1.271.27 in Simple, 1.281.28 in Moderate, and 1.221.22 in Complex) and FdivF_{\text{div}} staying within 0.67∼0.900.67\sim0.90. Moreover, we observe an instructive structural trend in Table 19: in the Simple regime (dominated by long 40.79m40.79\text{m} highways), diffusion sampling tends to subdivide long corridors into shorter segments (27.88m27.88\text{m}), whereas in the Complex regime (dominated by short 19.88m19.88\text{m} intersection markings), MapLDM matches ground-truth segment length (18.83m18.83\text{m} vs. GT 19.88m19.88\text{m}), max turning angle (0.33 rad0.33\text{ rad} vs. GT 0.34 rad0.34\text{ rad}), and scene coverage (0.310.31 vs. GT 0.320.32) closely, with only a moderate layout count saturation (87.3987.39 vs. GT 100.19100.19 lanes, Flay=0.46F_{\text{lay}}=0.46) consistent with the active-slot capacity curve in Fig. 4(d).

5. Downstream Utility & Topological Traversability Metrics [Addressing Q3]

To evaluate downstream topological utility and traversability for planning, we evaluated MapLDM on the nuPlan Urban Planning benchmark (Table 1) using the standard graph-traversability and topological validity metrics established by SLEDGE and Scenario Dreamer.

  • Multi-hop Traversability and Path Reachability (Reach\text{Reach} and Convenience\text{Convenience}). These metrics construct directed lane graphs from generated polylines and connections to measure whether a planner can find valid, uninterrupted multi-hop routes across intersections. MapLDM achieves Reach=0.06\text{Reach}=0.06 and Conve.=0.26\text{Conve.}=0.26 (outperforming SLEDGE at 0.510.51 and 1.681.68), alongside an average traversable Route Length (RL) of 35.18m35.18\text{m} within a 64×64m64\times64\text{m} patch, which closely matches ground truth and baselines (35.8∼37.0m35.8\sim37.0\text{m}).
  • Junction and Boundary Disconnection Error (ED\text{ED}). Endpoint Distance directly measures topological violations such as disconnected lane junctions along predicted successor edges. MapLDM achieves 0.28±0.20m0.28 \pm 0.20\text{m}, substantially outperforming Scenario Dreamer (±0.71∼0.80m\pm 0.71\sim0.80\text{m}).

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period proceeds, we would like to kindly follow up to see if our rebuttal and additional results have adequately addressed your concerns. If you have any remaining questions, we remain fully available to provide further clarifications. If our response has resolved your concerns, we would deeply appreciate it if you could consider reflecting that in your assessment.

Thank you again for your valuable time and constructive feedback!

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.