OmniCore: Heterogeneous Conditioning for Controllable 3D Scene Generation

Controllable outdoor 3D scene generation is central to autonomous driving and urban simulation, yet existing methods remain limited in scalability and flexibility. They typically depend on dense semantic annotations or rigid conditioning signals that are rarely available in practice. We introduce OmniCore, to our knowledge the first unified framework for controllable 3D scene generation in which roadmaps define the primary road layout, while natural language and heightmaps further specify its semantic layout and vertical structure. This design enables both high-fidelity 3D semantic scene synthesis and region-specific editing. OmniCore features a novel windowed spectral-attention block (WSAB), which integrates local spatial attention with scene-adaptive spectral filtering within each window to improve fine-grained geometric continuity across multimodal representations. To facilitate systematic evaluation, we present OmniControl-3D, a large-scale benchmark for controllable 3D scene generation that enriches SemanticKITTI, KITTI-360, and nuScenes with diverse annotations, including roadmaps, natural language descriptions, and heightmaps, each paired with 3D semantic scenes. Extensive experiments show that OmniCore achieves state-of-the-art performance across all conditioning configurations, demonstrating strong flexibility and spatial precision for controllable 3D scene generation. In addition, OmniCore can serve as a scalable data engine, generating diverse and semantically consistent 3D urban scenes that improve downstream 3D perception, as demonstrated in 3D semantic scene completion. A subset of OmniControl-3D is available here; the full benchmark and code will be released.