acceptodds
Under review as a conference paper at ICLR 2027

OmniCore: Heterogeneous Conditioning for Controllable 3D Scene Generation

Abstract

Controllable outdoor 3D scene generation is central to autonomous driving and urban simulation, yet existing methods remain limited in scalability and flexibility. They typically depend on dense semantic annotations or rigid conditioning signals that are rarely available in practice. We introduce OmniCore, to our knowledge the first unified framework for controllable 3D scene generation in which roadmaps define the primary road layout, while natural language and heightmaps further specify its semantic layout and vertical structure. This design enables both high-fidelity 3D semantic scene synthesis and region-specific editing. OmniCore features a novel windowed spectral-attention block (WSAB), which integrates local spatial attention with scene-adaptive spectral filtering within each window to improve fine-grained geometric continuity across multimodal representations. To facilitate systematic evaluation, we present OmniControl-3D, a large-scale benchmark for controllable 3D scene generation that enriches SemanticKITTI, KITTI-360, and nuScenes with diverse annotations, including roadmaps, natural language descriptions, and heightmaps, each paired with 3D semantic scenes. Extensive experiments show that OmniCore achieves state-of-the-art performance across all conditioning configurations, demonstrating strong flexibility and spatial precision for controllable 3D scene generation. In addition, OmniCore can serve as a scalable data engine, generating diverse and semantically consistent 3D urban scenes that improve downstream 3D perception, as demonstrated in 3D semantic scene completion. A subset of OmniControl-3D is available here; the full benchmark and code will be released.

Primary area:
generative models
Keywords:
Controllable 3D Scene Generation, Semantic Scene Synthesis, Heterogeneous Conditioning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yD77OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

OmniCore addresses conditional generation of outdoor semantic voxel scenes. A roadmap supplies the primary spatial layout, while optional text and heightmaps provide semantic and vertical constraints. The method combines a semantic VAE, modality-specific conditioning encoders, and a diffusion backbone built around a windowed spectral-attention block (WSAB). The accompanying OmniControl-3D benchmark augments three existing driving datasets with paired conditioning signals. The paper reports improved generation metrics, cross-dataset transfer, conditional editing examples, and an application to semantic scene completion.

Critical strengths
  1. Useful conditioning design. Making road layout the common input and treating text and height as complementary controls is a practical formulation for semantic scene authoring (Sections 3–4).
  2. Substantial reported improvements within the adapted benchmark. Under roadmap-only conditioning, Table 2(a) reports FID values of 38.27 for OmniCore, versus 83.21 for 3D DiT and 112.51 for CymbaDiff. These are encouraging differences if the shared evaluation and adaptation protocols are confirmed.
  3. Meaningful ablations and supplementary evidence. Tables 2(c), 3, 6, and 7 examine the attention block, transform/gating choices, fusion, and encoders. The appendix also includes boundary-continuity measurements, direct condition-alignment metrics, VAE reconstruction results, and a downstream experiment. These additions should be credited rather than described as missing.
  4. Evaluation beyond in-domain generation. Cross-dataset results, unseen-map examples, and the SSA-SC experiment help assess possible uses of the generated scenes, even though some conclusions need tighter controls.
Critical weaknesses
  1. Experimental assessment

1.1 The most consequential issue is comparison provenance, not a lack of result tables

Section 5 specifies a learned 3D geometry/semantic feature space for FID and MMD, whereas Table 2(a) includes historical-looking rows whose original evaluation protocols differ (pp. 7–8). PrITTI reports FID 71.028 on KITTI-360 using top-down semantic renderings; Table 2 lists 71.03 in its upper SemanticKITTI block [P1, Sections 4 and 4.2, Table 3]. CymbaDiff explicitly identifies the SSD 112.82 and SemCity 56.55 values as 2D FID, and these values also appear in Table 2 [P2, Table 2].

Concern: The numerical matches raise a concrete provenance question; they do not prove whether a row was copied or independently recomputed. Unless these methods were rerun on a common dataset and rescored with the same feature extractor, the historical rows cannot establish one common 3D-FID ranking. This does not automatically invalidate the separate adapted-baseline results on OmniControl-3D, which compare matching input modalities and show substantial reported improvements.

1.2 Condition adherence is demonstrated, but preservation during editing is not yet established Table 8/A.10 (p. 21) reports roadmap IoU 0.485 against a GT reference of 0.503, text Jaccard improving from 0.211 to 0.595 when text is added, and heightmap MAE decreasing from 4.095 to 2.146 voxel units with height conditioning. These are meaningful quantitative results; it would be inaccurate to describe controllability as unevaluated.

Existing WSAB ablations are useful, but do not fully distinguish the proposed mechanism

Table 2(c) (p. 7) compares vanilla, global-scope, shifted-window, and default blocks. Table 3(a) (p. 8) crosses fixed/learned transforms with full-spectrum/low-band gating: fixed-transform plus LF yields FID 54.44, versus 38.27 for learned-transform plus LF. Table 5/A.9 (pp. 20–21) additionally reports boundary-discontinuity improvement on 815 validation samples. Thus, the paper already has substantive architectural and continuity ablations.These results support effectiveness within the reported setup, but not every part of the proposed explanation. Equations (3)–(4) retain a fixed set of transform indices while learning a general unitary basis; unitarity alone does not guarantee that these indices remain physically low-frequency. Section 4.2 explicitly makes operations independent within each window, so the block itself does not create a cross-window latent communication path. Improved decoded continuity can still arise through local regularization, shared conditioning, or other network components. Moreover, BDR only considers non-empty voxel pairs and cannot by itself settle occupied–empty cracks or the possibility of excessive smoothing. 2. ### Task/interface contribution: useful, but narrower than a new controllable-generation paradigm

Combining roadmaps, heightmaps, and text is a practically appealing interface. However, controllable urban-scene generation, semantic/geometric guidance, and editing are not new in themselves. X-Scene already integrates textual intent with geometric layout control and occupancy generation [P3, Sections 3.1–3.2 and 4.2]; PrITTI supports controllable generation and editing [P1, Sections 3.3 and 4.3]. These systems are not identical to OmniCore and do not refute the novelty of its exact input combination. The differentiator should be the accessibility and complementarity of these particular controls, with demonstrated spatial edit preservation—not a broad priority claim for controllable scene generation. Established encoders and FiLM-style fusion do not make the system worthless, but are not independently new mechanisms (manuscript Section 4.2).

  1. WSAB’s empirical benefit is better supported than its frequency and cross-window interpretation First, the transform is initialized as a DFT but is subsequently allowed to move on the unitary group. Unitarity preserves invertibility, not the physical frequency ordering of basis vectors. Consequently, a fixed low-index set is not necessarily a low-frequency band after learning. Please either constrain or diagnose the learned basis to justify this interpretation, or describe it more conservatively as a learned subspace gate. The reported fixed/learned and full-spectrum/band-selective ablations are useful, but do not establish that frequency ordering is retained. The explicit real-part projection in Eq. (4) is acknowledged; this concern is not an allegation that the stated unitary inverse is incorrect or that the implementation contains a bug.

Second, p. 7 explicitly states that all subsequent operations are independent within each non-overlapping window. Such a block can encourage smoother outputs, but does not itself exchange latent information between adjacent windows. Please distinguish local regularization from cross-window communication and identify any communication supplied by other network components.

Importantly, the appendix does provide positive boundary evidence: Table 5 reduces partition-specific excess discontinuity from 0.0267 to 0.0095, compared with 0.0141 for shifted windows and 0.0070 for ground truth. The limitation is narrower: A.9 evaluates disagreement between non-empty voxel pairs, which can overlook occupied-to-empty cracks and does not alone separate continuity from oversmoothing. An occupancy-aware check on existing outputs, or a suitably limited interpretation, would address this without requiring an entirely new benchmark.

Additional clarifications and reporting improvements

Architecture and VAE usage: A.4 and Table 9 report two VAEs with different category coverage and training durations. Identify which checkpoint is used in each experiment, whether they are combined, how absent classes are handled, and how this choice affects baseline fairness. Specify the condition feature’s 2D-to-3D lifting and essential network dimensions.

Computational claims: The p. 4 comparison with O(nw³) window attention needs correction or a different definition of w. For n cubic windows of side w, dense within-window attention has a token-pair term proportional to nw⁶, before accounting for channel dimensions. A learned dense unitary transform also does not automatically inherit FFT complexity. Report actual parameter count, memory, and sampling latency under a common setup, or moderate the efficiency claim.

Questions for authors

see the above

5 comments
AuthorsOpenReview ↗

Response to Reviewer yD77 Q1-Q2

Thank you for recognizing our practical conditioning design, the substantial gains on our benchmark, the meaningful ablations and supplementary evidence, and the evaluation beyond in-domain generation. We confirm the shared evaluation and adaptation protocols: 3D DiT, CymbaDiff, and OmniCore share the VAE-full latent, training split, and 3D evaluation, differing only in the denoiser.

Q1: Several Table 2(a) values (PrITTI 71.03, SSD 112.82, SemCity 56.55) match 2D FIDs reported in prior work, yet Section 5 defines FID/MMD in a 3D feature space. Were these baselines rerun and rescored under the same 3D protocol? If not, these rows cannot support a common 3D-FID ranking.

A1: Thank you for catching this. We have revised it accordingly in Table 2. These results are taken from the original papers, which evaluate on SemanticKITTI (PrITTI: KITTI-360) under their own protocols. They are listed to show the conditioning each prior method uses (column C of Table 2(a)) and should not be ranked against our 3D FIDs. All rankings are now based on the controlled block of Table 2 (the rows on our OmniControl-3D benchmark), where 3D DiT and CymbaDiff are retrained and scored under the same 3D protocol as OmniCore. For a direct comparison, we have also retrained PrITTI, the strongest of the three (it outperforms SemCity, which in turn outperforms SSD, each in its own paper).

Starting from PrITTI's official code. We adapt it so that it takes our roadmap (R) with (optional) text (T) or heightmap (H) as conditioning to generate 3D semantic voxels; all other components are kept unchanged, and we follow OmniCore's data, training, and evaluation setup. Both methods are trained on OmniControl-3D-SemanticKITTI and evaluated on its validation set and zero-shot on OmniControl-3D-KITTI-360 and OmniControl-3D-nuScenes:

DatasetConditionPrITTI FID ↓PrITTI MMD ↓OmniCore FID ↓OmniCore MMD ↓
SemanticKITTIR147.300.1038.270.04
SemanticKITTIR+T142.400.1030.840.04
SemanticKITTIR+H63.040.0615.190.03
KITTI-360R252.800.1480.190.08
nuScenesR321.900.16123.290.08

OmniCore outperforms PrITTI on these three datasets under identical settings. We attribute the gap to the task: PrITTI is designed to generate from scene-level labels, whereas here every method must infer dense 3D voxels from a sparse roadmap. We have added PrITTI to the controlled block of Table 2, marked the reference rows as "reported, not directly comparable" (Table 2 and Sec. 5), and corrected PrITTI's dataset label.

Q2: Condition adherence is quantitatively demonstrated (Table 8/A.10), but preservation of unedited regions during editing is not yet established.

A2: Beyond condition adherence (Table 8), we have further verified (App. A.15) that edits preserve the roadmap-defined road layout, and moderate heightmap edits also largely preserve the top-down semantic layout. The roadmap fixes the road layout, the text gives the coarse placement of one target category, and the heightmap sets the vertical extent at each location. For roadmap + text in particular, the voxel-level layout of the remaining categories is not specified by any input and is left to the generative model. Preservation is therefore expected only for content fixed by the unchanged inputs, such as the road layout under text edits. On all 815 SemanticKITTI validation scenes, we change a single input, either the building location in the prompt (roadmap + text, as in Fig. 4(d–f)) or the heightmap scale (roadmap + heightmap, as in Fig. 4(g–h)), with the roadmap and initial noise fixed:

EditRoad IoU with the input roadmap, before → after the editTop-down IoU between the outputs before and after the edit ↑
Text: building location changed0.489 → 0.482Roads: 0.907
Heightmap scaled to 0.8×0.501 → 0.496Mean over all categories: 0.727
Heightmap scaled to 0.5×0.501 → 0.493Mean over all categories: 0.514
Ground-truth scenes (reference)0.503–
  • All edits preserve the road layout. The road IoU with the roadmap changes by at most 0.008 and stays close to that of real scenes (0.503, the attainable ceiling, since the roadmap draws every road at a fixed width; App. A.10).
  • Text edits move the buildings as requested (fraction of building voxels in the requested region: 0.286 → 0.863), while the roads stay where they were (IoU with the pre-edit output: 0.907).
  • Heightmap edits retain most of the top-down layout at 0.8× (0.727). The lower value at 0.5× (0.514) is expected, as halving all heights also changes the categories seen from above.

We have revised Secs. 1 and 4.2 to state this precisely, and clarified in Sec. 5.1 and Fig. 4 that, in all text edits, the prompt keeps only the sentence for the edited category, extracted from the full multi-category description.

AuthorsOpenReview ↗

Response to Reviewer yD77 Q3-Q4

Q3: Eqs. (3)–(4) keep fixed low-index positions, but the DFT-initialized transform is learned on the unitary group, and unitarity alone does not ensure these indices remain physically low-frequency after training. Please constrain or diagnose the learned basis, or call it a learned subspace gate instead.

A3: The kept indices remain low-frequency after training, both by construction and empirically: the learned basis is constrained to stay close to the DFT, and we have diagnosed the trained model as suggested. We do not rely on unitarity for frequency ordering; it only ensures invertibility. The transform is W_DFT = U·F, where F is the normalized DFT matrix. At U = I, the kept indices (the constant term and the six axis-aligned unit frequencies) are exactly the lowest frequencies, so what matters is how far training moves U from I. In our implementation, the Cayley generator Q of U is additionally bounded, ‖Q‖₂ ≤ 0.1, by projected gradient descent following COFT (Qiu et al., 2023). Hence ‖U − I‖₂ ≤ 0.2, and each kept basis vector provably keeps at least (1 − 2·0.1²)⁶ ≈ 0.885 of its energy in its original DFT component. Empirically, the purity (the fraction of a kept vector's energy in the canonical low frequencies, over all WSAB layers) confirms this:

BasisPurity ↑
DFT (initialization)1.00
Trained OmniCore (guaranteed ≥ 0.885)0.93
Random unitary (expected)0.11

We have added the bound, its derivation and this diagnosis to Sec. 4.2 and App. A.18.

Q4: Section 4.2 makes WSAB operate independently within each window, so the block itself provides no cross-window communication path. The improved decoded continuity may also arise from local regularization, shared conditioning, or other network components. Please distinguish local regularization from cross-window communication and identify any cross-window communication supplied by other components.

A4: WSAB acts within each window: it targets continuity at window seams, not latent exchange between windows. Cross-window context comes from two other components, as now clarified in Sec. 4.2: (i) the VAE decoder, whose convolutions span seams; and (ii) the condition encoder, whose convolutions run over the full scene before windowing, so the condition features that each window reads through cross-attention (Eq. 2) carry context across seams. We attribute the improved continuity to WSAB's spectral gate, which keeps the low-frequency structure and attenuates high-frequency deviations within each window (Eq. 4), anchoring every window to this shared condition so that adjacent windows agree at seams.

Table 5 varies only the block (vanilla windowed cross-attention, WSAB + shifted window, or WSAB), so shared conditioning and the other network components (the condition encoder and VAE decoder) are identical across all variants. Even with them, vanilla windowed cross-attention shows an excess seam discontinuity (ΔBDR, App. A.9) of 0.0267; WSAB reduces it to 0.0095, outperforming "WSAB + shifted window" (0.0141), which adds latent communication through shifted windows. The gain is also not explained by uniform smoothing, the simplest form of local regularization: it would lower seam and interior disagreement alike, which ΔBDR normalizes out, and WSAB does not oversmooth (A5, App. A.17). The gain thus comes from this local but content-adaptive spectral gating that aligns each window with the shared conditioning.

AuthorsOpenReview ↗

Response to Reviewer yD77 Q5-Q6

Q5: Table 5 shows positive boundary evidence (excess discontinuity 0.0267 → 0.0095; shifted windows 0.0141, GT 0.0070). However, BDR (A.9) considers only non-empty voxel pairs, so it can miss occupied-to-empty cracks and cannot distinguish continuity from oversmoothing. Please add an occupancy-aware check on the existing outputs, or interpret the result more narrowly.

A5: Thank you for suggesting this check. We have run it and an oversmoothing check on the existing outputs (App. A.17), and neither cracks nor oversmoothing explains the gain. BDR is already normalized by neighboring interior planes, so uniform oversmoothing, which lowers seam and interior disagreement alike, does not by itself reduce it.

  • Occupancy-aware ΔBDR: occupied/empty disagreement across seam planes, over pairs with at least one occupied voxel, normalized and offset-controlled as in A.9. Values above ground truth (GT) indicate cracks at seams.
  • Interior transition rate: the fraction of adjacent occupied voxel pairs with different labels away from seams. Values well below GT would indicate oversmoothing.
MethodOccupancy-aware ΔBDR (gap to GT)Interior transition rate (gap to GT)
Ground truth−0.01600.0266
Vanilla windowed cross-attention0.0019 (0.0179)0.0471 (0.0205)
WSAB + shifted window−0.0062 (0.0098)0.0448 (0.0182)
WSAB (ours)−0.0142 (0.0018)0.0313 (0.0047)

GT has no window partition, so its values reflect the scene layout alone (hence ΔBDR can be negative), and the target is GT, not zero. WSAB shows almost no seam-specific occupancy excess over GT (0.0018, vs. 0.0179 for the vanilla block), so the gain in Table 5 is not explained by cracks. No variant has a transition rate below GT, which oversmoothing would cause, and WSAB is the closest to GT.

Q6: Controllable generation and editing are not new (X-Scene, PrITTI), and the encoders and FiLM fusion are established. The contribution should be framed as accessible, complementary controls with demonstrated edit preservation, not as a broad priority claim.

A6: We have further clarified in the Abstract and Sec. 1 that our priority claim is restricted to the specific combination of an abstract roadmap with optional text and/or heightmap for 3D semantic scene generation, not controllable generation or editing in general. We have adopted the suggested framing, which matches our three differentiators:

  1. Accessibility. No condition requires manual labeling, and the only required input is a public roadmap, which enables generation at unseen real-world locations (Fig. 5). In contrast, PrITTI needs scene-level labels, and X-Scene controls geometry through layout maps with 3D boxes.
  2. Complementarity. Each signal primarily governs one factor: the roadmap sets the horizontal layout, the text the semantic placement, and the heightmap the vertical structure (Table 8: adding text raises text Jaccard from 0.211 to 0.595; adding the heightmap reduces height MAE from 4.095 to 2.146).
  3. Edit preservation. Edits preserve the road layout, and moderate heightmap edits also largely preserve the top-down semantic layout, as quantified in A2.

We do not claim the encoders or FiLM as contributions (Sec. 1). Still, using the roadmap as the backbone of the roadmap–text fusion outperforms concatenation and cross-attention (Table 3(b): FID 30.84 vs. 56.25 and 44.52). We have also narrowed the statement that heightmaps give vertical control "that existing methods lack", and added a comparison with X-Scene to Sec. 2.

Reviewer_k3GJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniCore, a latent diffusion model for 3D semantic voxel scene generation. It decomposes scene control into three accessible conditions: roadmaps for horizontal layout, heightmaps for vertical structure, and text for semantic object placement. The architecture uses a 3D VAE with a diffusion transformer based on a Windowed Spectral Attention Block (WSAB), which combines windowed cross-attention with a learnable unitary discrete Fourier transform. The authors also contribute the OmniControl-3D dataset across SemanticKITTI, KITTI-360, and nuScenes, demonstrating conditional generation, cross-dataset transfer, and downstream semantic scene completion gains.

Critical strengths

Decomposing 3D scene synthesis into low-cost roadmaps, heightmaps, and text is practical and scalable compared to expensive HD maps. The aligned multi-dataset construction also provides a solid resource for 3D scene generation.

Critical weaknesses

Mechanism Mismatch in WSAB: The claims about windowed spectral modeling improving cross-window continuity conflict with the actual architecture. Because patchification, windowed cross-attention, and Fourier transforms operate strictly inside non-overlapping windows without shifted windows, no information is exchanged across windows during denoising. Additionally, because the block appears to omit latent self-attention, the spectral projection operator serves as the only spatial token mixer, which may explain the performance gains rather than the frequency design itself.

Questions for authors
  1. Does WSAB include latent self-attention and residual connections? If not, how much of the performance gain comes from basic spatial token mixing rather than spectral filtering?
  2. With only 1024 latent tokens, what is the speed, memory, and generation quality if WSAB is replaced with standard global self-attention?
  3. How is the R+H+T condition combination implemented in Table 2, and how does your R+H fusion method differ from the concatenation baseline in Table 3(b)?
4 comments
AuthorsOpenReview ↗

Response to Reviewer k3GJ Q1-Q2

Thank you for recognizing the practicality and scalability of decomposing 3D scene synthesis into low-cost roadmaps, heightmaps, and text, as well as the value of OmniControl-3D as a resource for 3D scene generation.

Q1: Mechanism Mismatch in WSAB: The claims about windowed spectral modeling improving cross-window continuity conflict with the actual architecture. Because patchification, windowed cross-attention, and Fourier transforms operate strictly inside non-overlapping windows without shifted windows, no information is exchanged across windows during denoising. Additionally, because the block appears to omit latent self-attention, the spectral projection operator serves as the only spatial token mixer, which may explain the performance gains rather than the frequency design itself.

A1: WSAB acts within each window, as you note: it targets continuity at window seams, not latent exchange between windows. Cross-window context comes from two other components, as now clarified in Sec. 4.2: (i) the VAE decoder, whose convolutions span seams; and (ii) the condition encoder, whose convolutions run over the full scene before windowing, so the condition features that each window reads through cross-attention (Eq. 2) carry context across seams. We attribute the improved continuity to WSAB's spectral gate, which keeps the low-frequency structure and attenuates high-frequency deviations within each window (Eq. 4), anchoring every window to this shared condition so that adjacent windows agree at seams.

Table 5 varies only the block (vanilla windowed cross-attention, WSAB + shifted window, or WSAB), so the condition encoder and VAE decoder are shared by all variants. Even with them, vanilla windowed cross-attention shows an excess seam discontinuity (ΔBDR, App. A.9) of 0.0267; WSAB reduces it to 0.0095, outperforming "WSAB + shifted window" (0.0141), which adds latent communication through shifted windows. Our framework exchanges information across windows through condition features from a full-scene encoding and the convolutional decoder. WSAB's spectral gate preserves the low-frequency layout anchored to the shared condition and suppresses the window-specific high-frequency deviations that appear as seams, thereby avoiding the unnecessary all-frequency exchange introduced by shifted windows.

A2 tests directly whether generic token mixing explains the FID gain.

Q2: Does WSAB include latent self-attention and residual connections? If not, how much of the performance gain comes from basic spatial token mixing rather than spectral filtering?

A2: WSAB contains no latent self-attention: each window applies cross-attention, followed by spectral filtering. It does include residual connections around both the cross-attention and the final LayerNorm–MLP block; we have made them explicit after Eqs. 2 and 4 and in Fig. 2.

To separate spatial token mixing from spectral filtering, we replace WSAB's spectral branch with two generic token mixers that act inside the same windows, keeping everything else unchanged:

Token mixer inside each windowFID ↓MMD ↓
None (vanilla windowed cross-attention)73.820.06
Latent self-attention54.990.06
3D depthwise convolution68.430.06
Spectral gate (WSAB, ours)38.270.04

The better of the two generic mixers recovers 18.83 of the 35.55-point FID gain (73.82 → 38.27); the spectral gate improves FID by a further 16.72 points over it. This is consistent with existing ablations: with the same tokens mixed in each window and the same gating rule, switching from a fixed to a learnable Fourier basis alone lowers FID from 54.44 to 38.27 (Table 3(a)), and Global-scope WSAB mixes far more tokens yet reaches only 63.18 (Table 2(c)). We have added this to App. A.20.

AuthorsOpenReview ↗

Response to Reviewer k3GJ Q3-Q4

Q3: With only 1024 latent tokens, what is the speed, memory, and generation quality if WSAB is replaced with standard global self-attention?

A3: Replacing WSAB with standard global self-attention and fusing the condition through a single cross-attention layer, with all else fixed, gives FID 85.54 (vs. 38.27 with WSAB), at 4.60 GiB peak inference memory (vs. 3.68) and 15.67 s per scene (vs. 12.01). Both models use roadmap-only conditioning and the same protocol (one A100, batch 1, 100 denoising steps):

ModelParametersPeak inference memory (GiB)Sampling time per scene (s)FID ↓MMD ↓
OmniCore with global self-attention751M4.6015.6785.540.09
OmniCore with WSAB (ours)618M3.6812.0138.270.04

We adopt windows for locality. 3D scenes have non-stationary statistics (Sec. 4.2), so the spectral gate needs a fixed local support, which the shared window partition provides; applying the gate globally degrades FID from 38.27 to 63.18 (Table 2(c)). Windowing also keeps the cost linear in scene size, which matters for larger scenes or finer latents. We have added this comparison to App. A.21, and Sec. 4.2 now states both reasons.

Q4: How is the R+H+T condition combination implemented in Table 2, and how does your R+H fusion method differ from the concatenation baseline in Table 3(b)?

A4: R+H+T. The two pathways of Sec. 4.2 are composed of: the roadmap (R) and heightmap (H) are concatenated and encoded by the lightweight 2D CNN (as in R+H), and the resulting features are scaled and shifted channel-wise (FiLM) using the OpenCLIP text (T) embedding (as in R+T), F_RH ⊙ (1 + γ) + β, before being lifted and injected into WSAB via cross-attention.

R+H vs. concatenation. Both concatenate R and H and inject the result into the denoiser in the same way. The only difference is that we first encode the concatenation with the lightweight 2D CNN, whereas the baseline uses the raw channels (FID 15.19 vs. 21.21).

Thank you for raising this; the combination details about R+H+T are now stated in Sec. 4.2, and the explanation of Table 3(b) in Sec. 5.2 has been revised accordingly.

Dear Reviewer k3GJ,

Thank you again for your constructive suggestions. We have carefully addressed the concerns you raised and conducted the requested experiments. With the discussion period ending soon, we hope these updates have addressed your concerns. Please feel free to let us know if you have any further questions.

Best regards,

The Authors

Thanks for the rebuttal, I would like to keep my score to weak accept.

Reviewer_iiEaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces omniCore, a latent diffusion approach for generation of 3d semantic voxel scenes conditioned on roadmaps, text and height maps. The main idea is to combine windowed cross-attention with adaptive filtering in a learned basis. The work also proposes OminControl-3D and report improvements in generation metrics, transfer across-dataset, downstream semantic scene completion and adherence to the conditions provided.

Critical strengths
  • work reports improvements over adapted baselines and provides useful information from components ablations.
  • the direct condition-alignment and boundary-continuity evals (Window scope, learnable transforms, gating, fusion strategies, and condition encoder) provide complete information
  • Cross-dataset transfer and synthetic-data augmentation extend the results to more than just qualitative results in a meaningful way
Critical weaknesses
  • Ground-truth-derived heightmaps and captions weaken the broad claims of readily obtainable, annotation-free conditioning
  • prompt based generation changes do not show localized editing with preservation of unrelated content
  • Dataset construction, joint conditioning, baseline adaptation, and evaluation protocols require more detail to be clear
  • Important related work that needs to be discussed better for placing novelty claims [1,2]

[1] Huang, Zhipeng, et al. "Adaptive frequency filters as efficient global token mixers." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023. [2] Xie, Haozhe, et al. "Generative Gaussian splatting for unbounded 3D city generation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors

see weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer iiEa Q1-Q3

Thank you for the constructive review. All changes are in the revised paper.

Q1: Ground-truth-derived heightmaps and captions weaken the broad claims of readily obtainable, annotation-free conditioning

A1: We have further clarified in Secs. 1 and 3 that "annotation-free" means no condition requires manual labeling: roadmaps come from OpenStreetMap, while captions and heightmaps are generated automatically by our designed pipeline. Deriving training conditions from the target scenes is standard practice in conditional generation (e.g., ControlNet extracts edge and depth maps from its training images). To generate a new scene, a user simply provides a public roadmap, optionally with a text prompt and a heightmap (user-specified), as demonstrated on unseen real-world roadmaps (Fig. 5) and user-edited prompts and heightmaps (Fig. 4). All conditions therefore remain readily obtainable in practice.

Q2: Prompt-based generation changes do not show localized editing with preservation of unrelated content.

A2: We localize each edit by changing only the target category's location in the prompt, with all other inputs fixed. Since the fixed roadmap defines the road layout (Sec. 4.2) and relocating a category necessarily changes the content around it, what is preserved is the layout, not every voxel. We verify both on all 815 SemanticKITTI validation scenes, fixing the roadmap and initial noise and editing the building location (e.g., West → East, as in Fig. 4(d–f)):

PromptFraction of building voxels in the new region ↑Road IoU with the input roadmap ↑
Original0.2860.489
Edited0.8630.482
Ground-truth scene (reference)–0.503

Buildings move to the requested region (0.286 → 0.863), while the generated roads still match the roadmap (0.489 vs. 0.482, close to 0.503 for real scenes). Compared with the output before editing, the road IoU is 0.907 and the total building volume changes by only 7.6%, i.e., buildings are relocated rather than removed. We have further clarified this in Sec. 1 and added the analysis as App. A.15.

Q3: Dataset construction, joint conditioning, baseline adaptation, and evaluation protocols require more detail to be clear

A3: We have added all four items to the revised paper (App. A.16):

  • Dataset construction: each scene is paired automatically with a roadmap queried from OpenStreetMap by GPS, a caption generated with a fixed template over seven categories, and a heightmap of the maximum occupied height per ground location.
  • Joint conditioning: the roadmap and heightmap (when given) are concatenated and encoded by a 2D CNN, and the text (when given) rescales these features channel-wise (FiLM). In all settings, the features are lifted to 3D by a strided convolution and fed to every block by cross-attention.
  • Baseline adaptation: 3D DiT and CymbaDiff are retrained using their official code on the same data split, resolution, labels, schedule, and conditions as OmniCore, with minimal adaptations to their input interfaces so that all methods receive the same roadmap, heightmap, and text conditions.
  • Evaluation protocol: every method generates one sample per validation scene (100 steps), and FID/MMD are computed with the same fixed 3D feature encoder (trained on the training split, following UrbanDiffusion).

All methods are thus compared under identical data, conditions, and evaluation protocol.

AuthorsOpenReview ↗

Response to Reviewer iiEa Q4

Q4: Important related work that needs to be discussed better for placing novelty claims [1,2]; [1] Huang, Zhipeng, et al. "Adaptive frequency filters as efficient global token mixers." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023. [2] Xie, Haozhe, et al. "Generative Gaussian splatting for unbounded 3D city generation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

A4: Thanks for the suggestion. We have discussed both works in Sec. 2; our contributions remain distinct:

  • AFF [1] applies a content-adaptive mask to the global FFT spectrum in 2D image backbones. WSAB instead filters within local 3D windows, in a learnable basis initialized at the DFT, and both choices matter (FID, lower is better): global instead of local, 63.18 vs. 38.27 (Tab. 2(c)); fixed instead of learnable basis, 54.44 vs. 38.27 (Tab. 3(a)).
  • GaussianCity [2] renders images from dense top-down maps (height field, semantic map, density map) lifted to 3D Gaussians; its authors note that this lifting cannot represent hollow structures. OmniCore needs no dense semantic map, generates a full 3D semantic scene, and uses the heightmap only as optional guidance, not as geometry.

We have further clarified our claim in the Abstract and Sec. 1: to our knowledge, the first unified framework combining an abstract roadmap (horizontal layout) with optional natural language (semantic placement) and/or heightmaps (vertical structure) to enable both high-fidelity 3D semantic scene synthesis and region-specific editing.

Dear Reviewer iiEa,

As the discussion period ends soon, we would like to follow up gently. We have carefully addressed the concerns you raised and hope our clarifications and revisions have resolved them. If anything remains unclear, we would be very happy to continue the discussion. Thank you again for your thoughtful feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.