LiFTeR: Generating LiDAR Gaussian Fields from Surround-View Cameras

High-fidelity LiDAR simulation requires modeling both scene geometry and sensor-specific return characteristics. Existing LiDAR generation methods often synthesize range, intensity, and ray-drop maps in image space, without an explicit 3D representation underlying these observations. Gaussian-based LiDAR reconstruction offers such a representation, but typically relies on per-scene optimization with real LiDAR measurements. We present LiFTeR, a cross-modal generative framework that bridges these approaches by producing renderable LiDAR Gaussian fields from surround-view images, without requiring target-scene LiDAR measurements or per-scene optimization at inference time. A diffusion transformer generates a latent representation conditioned on multi-view images, capturing scene geometry and LiDAR return characteristics. A ray-anchored Gaussian decoder then maps this latent representation to Gaussian primitives with explicit geometric parameters, intensity attributes, and ray-drop probabilities, which a panoramic Gaussian splatting renderer converts into range, intensity, and ray-drop maps. The framework additionally supports feed-forward conversion of observed LiDAR into Gaussian fields, enabling generation and reconstruction within a shared representation. On nuScenes, LiFTeR achieves the state-of-the-art MMD and JSD among LiDAR generation methods. Without target-scene LiDAR, it outperforms per-scene reconstruction methods on most depth and intensity metrics, while reducing the time to construct a scene representation from about an hour to 3 minutes. With LiDAR input, LiFTeR reduces the depth and intensity RMSE of novel view synthesis relative to baselines within 0.5 minutes per scene.