- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OmniCore addresses conditional generation of outdoor semantic voxel scenes. A roadmap supplies the primary spatial layout, while optional text and heightmaps provide semantic and vertical constraints. The method combines a semantic VAE, modality-specific conditioning encoders, and a diffusion backbone built around a windowed spectral-attention block (WSAB). The accompanying OmniControl-3D benchmark augments three existing driving datasets with paired conditioning signals. The paper reports improved generation metrics, cross-dataset transfer, conditional editing examples, and an application to semantic scene completion.
Critical strengths
- Useful conditioning design. Making road layout the common input and treating text and height as complementary controls is a practical formulation for semantic scene authoring (Sections 3–4).
- Substantial reported improvements within the adapted benchmark. Under roadmap-only conditioning, Table 2(a) reports FID values of 38.27 for OmniCore, versus 83.21 for 3D DiT and 112.51 for CymbaDiff. These are encouraging differences if the shared evaluation and adaptation protocols are confirmed.
- Meaningful ablations and supplementary evidence. Tables 2(c), 3, 6, and 7 examine the attention block, transform/gating choices, fusion, and encoders. The appendix also includes boundary-continuity measurements, direct condition-alignment metrics, VAE reconstruction results, and a downstream experiment. These additions should be credited rather than described as missing.
- Evaluation beyond in-domain generation. Cross-dataset results, unseen-map examples, and the SSA-SC experiment help assess possible uses of the generated scenes, even though some conclusions need tighter controls.
Critical weaknesses
- Experimental assessment
1.1 The most consequential issue is comparison provenance, not a lack of result tables
Section 5 specifies a learned 3D geometry/semantic feature space for FID and MMD, whereas Table 2(a) includes historical-looking rows whose original evaluation protocols differ (pp. 7–8). PrITTI reports FID 71.028 on KITTI-360 using top-down semantic renderings; Table 2 lists 71.03 in its upper SemanticKITTI block [P1, Sections 4 and 4.2, Table 3]. CymbaDiff explicitly identifies the SSD 112.82 and SemCity 56.55 values as 2D FID, and these values also appear in Table 2 [P2, Table 2].
Concern: The numerical matches raise a concrete provenance question; they do not prove whether a row was copied or independently recomputed. Unless these methods were rerun on a common dataset and rescored with the same feature extractor, the historical rows cannot establish one common 3D-FID ranking. This does not automatically invalidate the separate adapted-baseline results on OmniControl-3D, which compare matching input modalities and show substantial reported improvements.
1.2 Condition adherence is demonstrated, but preservation during editing is not yet established Table 8/A.10 (p. 21) reports roadmap IoU 0.485 against a GT reference of 0.503, text Jaccard improving from 0.211 to 0.595 when text is added, and heightmap MAE decreasing from 4.095 to 2.146 voxel units with height conditioning. These are meaningful quantitative results; it would be inaccurate to describe controllability as unevaluated.
Existing WSAB ablations are useful, but do not fully distinguish the proposed mechanism
Table 2(c) (p. 7) compares vanilla, global-scope, shifted-window, and default blocks. Table 3(a) (p. 8) crosses fixed/learned transforms with full-spectrum/low-band gating: fixed-transform plus LF yields FID 54.44, versus 38.27 for learned-transform plus LF. Table 5/A.9 (pp. 20–21) additionally reports boundary-discontinuity improvement on 815 validation samples. Thus, the paper already has substantive architectural and continuity ablations.These results support effectiveness within the reported setup, but not every part of the proposed explanation. Equations (3)–(4) retain a fixed set of transform indices while learning a general unitary basis; unitarity alone does not guarantee that these indices remain physically low-frequency. Section 4.2 explicitly makes operations independent within each window, so the block itself does not create a cross-window latent communication path. Improved decoded continuity can still arise through local regularization, shared conditioning, or other network components. Moreover, BDR only considers non-empty voxel pairs and cannot by itself settle occupied–empty cracks or the possibility of excessive smoothing. 2. ### Task/interface contribution: useful, but narrower than a new controllable-generation paradigm
Combining roadmaps, heightmaps, and text is a practically appealing interface. However, controllable urban-scene generation, semantic/geometric guidance, and editing are not new in themselves. X-Scene already integrates textual intent with geometric layout control and occupancy generation [P3, Sections 3.1–3.2 and 4.2]; PrITTI supports controllable generation and editing [P1, Sections 3.3 and 4.3]. These systems are not identical to OmniCore and do not refute the novelty of its exact input combination. The differentiator should be the accessibility and complementarity of these particular controls, with demonstrated spatial edit preservation—not a broad priority claim for controllable scene generation. Established encoders and FiLM-style fusion do not make the system worthless, but are not independently new mechanisms (manuscript Section 4.2).
- WSAB’s empirical benefit is better supported than its frequency and cross-window interpretation First, the transform is initialized as a DFT but is subsequently allowed to move on the unitary group. Unitarity preserves invertibility, not the physical frequency ordering of basis vectors. Consequently, a fixed low-index set is not necessarily a low-frequency band after learning. Please either constrain or diagnose the learned basis to justify this interpretation, or describe it more conservatively as a learned subspace gate. The reported fixed/learned and full-spectrum/band-selective ablations are useful, but do not establish that frequency ordering is retained. The explicit real-part projection in Eq. (4) is acknowledged; this concern is not an allegation that the stated unitary inverse is incorrect or that the implementation contains a bug.
Second, p. 7 explicitly states that all subsequent operations are independent within each non-overlapping window. Such a block can encourage smoother outputs, but does not itself exchange latent information between adjacent windows. Please distinguish local regularization from cross-window communication and identify any communication supplied by other network components.
Importantly, the appendix does provide positive boundary evidence: Table 5 reduces partition-specific excess discontinuity from 0.0267 to 0.0095, compared with 0.0141 for shifted windows and 0.0070 for ground truth. The limitation is narrower: A.9 evaluates disagreement between non-empty voxel pairs, which can overlook occupied-to-empty cracks and does not alone separate continuity from oversmoothing. An occupancy-aware check on existing outputs, or a suitably limited interpretation, would address this without requiring an entirely new benchmark.
Additional clarifications and reporting improvements
Architecture and VAE usage: A.4 and Table 9 report two VAEs with different category coverage and training durations. Identify which checkpoint is used in each experiment, whether they are combined, how absent classes are handled, and how this choice affects baseline fairness. Specify the condition feature’s 2D-to-3D lifting and essential network dimensions.
Computational claims: The p. 4 comparison with O(nw³) window attention needs correction or a different definition of w. For n cubic windows of side w, dense within-window attention has a token-pair term proportional to nw⁶, before accounting for channel dimensions. A learned dense unitary transform also does not automatically inherit FFT complexity. Report actual parameter count, memory, and sampling latency under a common setup, or moderate the efficiency claim.
Questions for authors
see the above
5 comments
Response to Reviewer yD77 Q3-Q4
Q3: Eqs. (3)–(4) keep fixed low-index positions, but the DFT-initialized transform is learned on the unitary group, and unitarity alone does not ensure these indices remain physically low-frequency after training. Please constrain or diagnose the learned basis, or call it a learned subspace gate instead.
A3: The kept indices remain low-frequency after training, both by construction and empirically: the learned basis is constrained to stay close to the DFT, and we have diagnosed the trained model as suggested. We do not rely on unitarity for frequency ordering; it only ensures invertibility. The transform is W_DFT = U·F, where F is the normalized DFT matrix. At U = I, the kept indices (the constant term and the six axis-aligned unit frequencies) are exactly the lowest frequencies, so what matters is how far training moves U from I. In our implementation, the Cayley generator Q of U is additionally bounded, ‖Q‖₂ ≤ 0.1, by projected gradient descent following COFT (Qiu et al., 2023). Hence ‖U − I‖₂ ≤ 0.2, and each kept basis vector provably keeps at least (1 − 2·0.1²)⁶ ≈ 0.885 of its energy in its original DFT component. Empirically, the purity (the fraction of a kept vector's energy in the canonical low frequencies, over all WSAB layers) confirms this:
| Basis | Purity ↑ |
|---|---|
| DFT (initialization) | 1.00 |
| Trained OmniCore (guaranteed ≥ 0.885) | 0.93 |
| Random unitary (expected) | 0.11 |
We have added the bound, its derivation and this diagnosis to Sec. 4.2 and App. A.18.
Q4: Section 4.2 makes WSAB operate independently within each window, so the block itself provides no cross-window communication path. The improved decoded continuity may also arise from local regularization, shared conditioning, or other network components. Please distinguish local regularization from cross-window communication and identify any cross-window communication supplied by other components.
A4: WSAB acts within each window: it targets continuity at window seams, not latent exchange between windows. Cross-window context comes from two other components, as now clarified in Sec. 4.2: (i) the VAE decoder, whose convolutions span seams; and (ii) the condition encoder, whose convolutions run over the full scene before windowing, so the condition features that each window reads through cross-attention (Eq. 2) carry context across seams. We attribute the improved continuity to WSAB's spectral gate, which keeps the low-frequency structure and attenuates high-frequency deviations within each window (Eq. 4), anchoring every window to this shared condition so that adjacent windows agree at seams.
Table 5 varies only the block (vanilla windowed cross-attention, WSAB + shifted window, or WSAB), so shared conditioning and the other network components (the condition encoder and VAE decoder) are identical across all variants. Even with them, vanilla windowed cross-attention shows an excess seam discontinuity (ΔBDR, App. A.9) of 0.0267; WSAB reduces it to 0.0095, outperforming "WSAB + shifted window" (0.0141), which adds latent communication through shifted windows. The gain is also not explained by uniform smoothing, the simplest form of local regularization: it would lower seam and interior disagreement alike, which ΔBDR normalizes out, and WSAB does not oversmooth (A5, App. A.17). The gain thus comes from this local but content-adaptive spectral gating that aligns each window with the shared conditioning.
Response to Reviewer yD77 Q5-Q6
Q5: Table 5 shows positive boundary evidence (excess discontinuity 0.0267 → 0.0095; shifted windows 0.0141, GT 0.0070). However, BDR (A.9) considers only non-empty voxel pairs, so it can miss occupied-to-empty cracks and cannot distinguish continuity from oversmoothing. Please add an occupancy-aware check on the existing outputs, or interpret the result more narrowly.
A5: Thank you for suggesting this check. We have run it and an oversmoothing check on the existing outputs (App. A.17), and neither cracks nor oversmoothing explains the gain. BDR is already normalized by neighboring interior planes, so uniform oversmoothing, which lowers seam and interior disagreement alike, does not by itself reduce it.
- Occupancy-aware ΔBDR: occupied/empty disagreement across seam planes, over pairs with at least one occupied voxel, normalized and offset-controlled as in A.9. Values above ground truth (GT) indicate cracks at seams.
- Interior transition rate: the fraction of adjacent occupied voxel pairs with different labels away from seams. Values well below GT would indicate oversmoothing.
| Method | Occupancy-aware ΔBDR (gap to GT) | Interior transition rate (gap to GT) |
|---|---|---|
| Ground truth | −0.0160 | 0.0266 |
| Vanilla windowed cross-attention | 0.0019 (0.0179) | 0.0471 (0.0205) |
| WSAB + shifted window | −0.0062 (0.0098) | 0.0448 (0.0182) |
| WSAB (ours) | −0.0142 (0.0018) | 0.0313 (0.0047) |
GT has no window partition, so its values reflect the scene layout alone (hence ΔBDR can be negative), and the target is GT, not zero. WSAB shows almost no seam-specific occupancy excess over GT (0.0018, vs. 0.0179 for the vanilla block), so the gain in Table 5 is not explained by cracks. No variant has a transition rate below GT, which oversmoothing would cause, and WSAB is the closest to GT.
Q6: Controllable generation and editing are not new (X-Scene, PrITTI), and the encoders and FiLM fusion are established. The contribution should be framed as accessible, complementary controls with demonstrated edit preservation, not as a broad priority claim.
A6: We have further clarified in the Abstract and Sec. 1 that our priority claim is restricted to the specific combination of an abstract roadmap with optional text and/or heightmap for 3D semantic scene generation, not controllable generation or editing in general. We have adopted the suggested framing, which matches our three differentiators:
- Accessibility. No condition requires manual labeling, and the only required input is a public roadmap, which enables generation at unseen real-world locations (Fig. 5). In contrast, PrITTI needs scene-level labels, and X-Scene controls geometry through layout maps with 3D boxes.
- Complementarity. Each signal primarily governs one factor: the roadmap sets the horizontal layout, the text the semantic placement, and the heightmap the vertical structure (Table 8: adding text raises text Jaccard from 0.211 to 0.595; adding the heightmap reduces height MAE from 4.095 to 2.146).
- Edit preservation. Edits preserve the road layout, and moderate heightmap edits also largely preserve the top-down semantic layout, as quantified in A2.
We do not claim the encoders or FiLM as contributions (Sec. 1). Still, using the roadmap as the backbone of the roadmap–text fusion outperforms concatenation and cross-attention (Table 3(b): FID 30.84 vs. 56.25 and 44.52). We have also narrowed the statement that heightmaps give vertical control "that existing methods lack", and added a comparison with X-Scene to Sec. 2.
Response to Reviewer yD77 Q1-Q2
Thank you for recognizing our practical conditioning design, the substantial gains on our benchmark, the meaningful ablations and supplementary evidence, and the evaluation beyond in-domain generation. We confirm the shared evaluation and adaptation protocols: 3D DiT, CymbaDiff, and OmniCore share the VAE-full latent, training split, and 3D evaluation, differing only in the denoiser.
Q1: Several Table 2(a) values (PrITTI 71.03, SSD 112.82, SemCity 56.55) match 2D FIDs reported in prior work, yet Section 5 defines FID/MMD in a 3D feature space. Were these baselines rerun and rescored under the same 3D protocol? If not, these rows cannot support a common 3D-FID ranking.
A1: Thank you for catching this. We have revised it accordingly in Table 2. These results are taken from the original papers, which evaluate on SemanticKITTI (PrITTI: KITTI-360) under their own protocols. They are listed to show the conditioning each prior method uses (column C of Table 2(a)) and should not be ranked against our 3D FIDs. All rankings are now based on the controlled block of Table 2 (the rows on our OmniControl-3D benchmark), where 3D DiT and CymbaDiff are retrained and scored under the same 3D protocol as OmniCore. For a direct comparison, we have also retrained PrITTI, the strongest of the three (it outperforms SemCity, which in turn outperforms SSD, each in its own paper).
Starting from PrITTI's official code. We adapt it so that it takes our roadmap (R) with (optional) text (T) or heightmap (H) as conditioning to generate 3D semantic voxels; all other components are kept unchanged, and we follow OmniCore's data, training, and evaluation setup. Both methods are trained on OmniControl-3D-SemanticKITTI and evaluated on its validation set and zero-shot on OmniControl-3D-KITTI-360 and OmniControl-3D-nuScenes:
OmniCore outperforms PrITTI on these three datasets under identical settings. We attribute the gap to the task: PrITTI is designed to generate from scene-level labels, whereas here every method must infer dense 3D voxels from a sparse roadmap. We have added PrITTI to the controlled block of Table 2, marked the reference rows as "reported, not directly comparable" (Table 2 and Sec. 5), and corrected PrITTI's dataset label.
Q2: Condition adherence is quantitatively demonstrated (Table 8/A.10), but preservation of unedited regions during editing is not yet established.
A2: Beyond condition adherence (Table 8), we have further verified (App. A.15) that edits preserve the roadmap-defined road layout, and moderate heightmap edits also largely preserve the top-down semantic layout. The roadmap fixes the road layout, the text gives the coarse placement of one target category, and the heightmap sets the vertical extent at each location. For roadmap + text in particular, the voxel-level layout of the remaining categories is not specified by any input and is left to the generative model. Preservation is therefore expected only for content fixed by the unchanged inputs, such as the road layout under text edits. On all 815 SemanticKITTI validation scenes, we change a single input, either the building location in the prompt (roadmap + text, as in Fig. 4(d–f)) or the heightmap scale (roadmap + heightmap, as in Fig. 4(g–h)), with the roadmap and initial noise fixed:
We have revised Secs. 1 and 4.2 to state this precisely, and clarified in Sec. 5.1 and Fig. 4 that, in all text edits, the prompt keeps only the sentence for the edited category, extracted from the full multi-category description.