- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents LidarVerse, a conditional LiDAR world model intended to unify LiDAR generation, forecasting, and reconstruction-grounded resimulation. A LiDAR-native range-image tokenizer based on neighborhood attention maps LiDAR sequences into a shared latent space without temporal downsampling. A 2B-parameter bidirectional flow-matching teacher is trained with independently sampled conditions including a LiDAR prefix, an HD-map and actor layout, and a persistent spatial cache (PSC). The teacher is distilled into a four-step block-causal student using self-forcing distribution matching. During causal rollout, a temporal K/V cache retains recent dynamics, while a rolling world-aligned voxel cache stores geometry generated in previous blocks. The paper evaluates tokenizer reconstruction, 129-frame autoregressive generation, ego-action-conditioned generation, point-cloud forecasting, and sparse-input resimulation. Separately fine-tuned bidirectional variants are used for cross-sensor generation and intensity modeling.
Critical strengths
-
Well-motivated LiDAR-specific design. The attention-based range-image tokenizer handles depth discontinuities, while the temporal K/V cache and world-aligned PSC provide complementary short- and long-term memory.
-
Broad and strong empirical evaluation. The paper covers long-horizon generation, forecasting, resimulation, multiple sensors, intensity, and efficiency. The four-step causal model achieves a strong quality-throughput trade-off against the included baselines.
Critical weaknesses
-
The unified-model claim is overstated and closely related work is missing. Cross-sensor and intensity results use separately fine-tuned bidirectional teachers rather than the four-step causal student, while unseen PandaSet transfer is only qualitative. More importantly, LiDAR-GS++ already combines LiDAR reconstruction with diffusion priors for extrapolated resimulation. LiSTAR covers reconstruction, prediction, and conditional generation. OmniLiDAR study cross-sensor/domain LiDAR generation or transfer. LiDAR-GS, GS-LiDAR, and XSIM are also directly relevant real-time resimulation baselines. These works substantially narrow the claimed novelty.
-
The resimulation evaluation uses privileged annotations and does not test genuine novel trajectories. The frozen-dynamic PSC requires annotated object poses during inference, but it is unclear whether competing methods receive the same information. Quantitative evaluation only predicts held-out frames along the recorded Waymo trajectory, rather than testing the large off-trajectory viewpoint changes advertised by the paper.
Questions for authors
- Which exact checkpoint supports each task/sensor combination? Can the same four-step causal student perform generation, forecasting, resimulation, 32/64/128-beam generation, unseen-pattern transfer, and intensity modeling without further fine-tuning?
- Please add discussion or comparison with the closely related baseline works LiDAR-GS++ (arXiv:2511.12304), LiSTAR (arXiv:2511.16049), OmniLiDAR (arXiv:2605.13815), Towards Foundational LiDAR World Models (arXiv:2506.23434), LiDAR-GS (arXiv:2410.05111), GS-LiDAR (arXiv:2501.13971), and XSIM (arXiv:2602.05617).
- During resimulation, are ground-truth future 3D actor boxes or poses supplied at inference time? Are LiDAR4D, LiDAR-RT, and SplatAD provided with the same object tracks?
- Could you evaluate resimulation quality under genuinely off-trajectory viewpoints, such as lateral translations or trajectories that revisit previously observed regions?
- Regarding the efficiency problem, can a new ego action be injected every frame, or only once per four-frame block? Does the 21-frame forecasting configuration remain above 10 FPS?
The lack of discussion and comparison on the baseline prevents me from giving a higher rating at this stage. I will consider improving my rating after the complete experiment.
6 comments
Q2: Privilege Annotations in Resimulation
A2: We would like to clarify that existing baselines, including LiDAR-RT and SplatAD, also use ground-truth future 3D actor boxes and boxes at inference time. LiDAR4D does not, as intended by their formulation, and thus resulting in lack of actor’s controllability and worse performance compared to ours.
We also want to point out that the usage of ground-truth actor boxes are standard practice in autonomous driving resimulation methods [8,9,10,11,12,13] and thus are not considered as privileged information. We will make this clarification clearer in the experiment setting for resimulation.
Reference:
[8] Yang et al., “UniSim: A neural closed-loop sensor simulator,” CVPR 2023
[9] Tonderski et al., “NeuRAD: Neural Rendering for Autonomous Driving,” CVPR 2024
[10] Yan et al., “Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting,” ECCV 2024
[11] Wu et al., “Dynamic LiDAR Re-simulation using Compositional Neural Fields,” CVPR 2024
[12] Zhou et al., “LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation,” CVPR 2025
[13] Chen et al., “OmniRe: Omni Urban Scene Reconstruction,” ICLR 2025
Q3: Off-trajectory evaluation
A3: To directly test the off-trajectory in conventional sense, we render at different lateral shifts (1m, 2m, and 3m) to the left and right and compute FUD and MMD, following existing methods such as [14,15,16,17]. We use the same evaluation setting as Table 3. We also include comparison with LiDAR-GS as requested, and will include more baselines afterwards. We observe that our methods outperform all resimulation baseline on off-trajectory evaluation, which supports our claim of quality under large off-trajectory viewpoint changes.
Each cell below reports FUD / MMD × 10⁵ (lower is better for all)
| Method | ±1 m | ±2 m | ±3 m |
|---|---|---|---|
| LiDAR-GS | 25.56 / 5.23 | 27.21 / 7.36 | 29.31 / 9.88 |
| LiDAR4D | 34.50 / 6.35 | 37.12 / 7.60 | 39.76 / 9.18 |
| LiDAR-RT | 39.25 / 7.30 | 41.72 / 8.76 | 45.32 / 10.23 |
| SplatAD | 75.94 / 4.46 | 77.82 / 6.16 | 80.06 / 8.36 |
| Ours-Bidir | 11.73 / 4.13 | 12.96 / 5.87 | 13.79 / 8.15 |
| Ours-SF-DMD | 25.73 / 4.43 | 26.24 / 5.92 | 27.41 / 7.86 |
Reference:
[14] Want et al., “FreeVS: Generative View Synthesis on Free Driving Trajectory,” ICLR 2025
[15] Yan et al., “StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models,” CVPR 2025
[16] Che et al., “GenRe: Diffusion-guided Generalizable Enhancer for Urban Scene Reconstruction,” ICRA 2026
[17] Chen et al., “LiDAR-GS++:Improving LiDAR Gaussian Reconstruction via Diffusion Priors,” AAAI 2026
Q4: Which checkpoint can support which task
A4: While cross-sensor transfer and intensity modeling are treated as extension in our paper, a four-step causal student derived from that checkpoint can perform generation, forecasting, resimulation, 32/64/128-beam generation, unseen-pattern transfer, and intensity modeling without further fine-tuning. We deliberately only include 1 dataset in our quantitative experiments to keep the training set fair with the baselines. We plan to release such a checkpoint publicly.
Q5: Efficiency Clarification
A5: If not pre-determined, new ego trajectory can only be injected once after 4-frame generation, resulting in latency of action-conditioning. We provide the full FPS and latency for our 3 modes below.
| Mode | End-to-end FPS | Block (4-frame) latency (ms) |
|---|---|---|
| Generation | 11.18 | 337.78* |
| Forecasting | 10.04 | 405.25 |
| Resimulation | 12.26 | 319.53 |
Thank you for the thoughtful review and valuable feedback. We are pleased that our design is “well-motivated” and that our experimental designs are “broad” and “strong”.
Q1: Overstated Unified Model Claims and Discussion with SOTA models
A1: We would like to clarify that our unified model is targeting multiple different tasks being supported within a single checkpoint and a single inference interface. To our understanding, we are the first unified model that can achieve all 3 tasks of long-horizon 4D LiDAR generation, LiDAR forecasting, and generalizable LiDAR resimulation. We provide the discussion with the requested baseline in the table below.
Notably, LiDAR-GS++ can only generate single-frame and cannot generate long-horizon LiDAR from scratch. LiSTAR’s reconstruction evaluation describes tokenizer’s reconstruction evaluation, without evidence of any capability to support resimulation task. OmniLiDAR studies cross-sensor/domain LiDAR but is limited to single-frame and does not support re-simulation. XSIM is a per-scene resimulation method that cannot support any generative task. In this sense, we are the only method capable of those three LiDAR-specific tasks. We will update the paper with this discussion.
Reference:
[1] Chen et al., “LiDAR-GS++: Improving LiDAR Gaussian Reconstruction via Diffusion Priors,” AAAI 2026.
[2] Liu et al., “LiSTAR: Ray-Centric World Models for 4D LiDAR Sequences in Autonomous Driving,” ECCV 2026.
[3] Liu et al., “OmniLiDAR: A Unified Diffusion Framework for Multi-Domain 3D LiDAR Generation,” arXiv preprint, 2026.
[4] Liu et al., “Towards Foundational LiDAR World Models with Efficient Latent Flow Matching,” NeurIPS 2025.
[5] Chen et al., “LiDAR-GS: Real-time LiDAR Re-Simulation using Gaussian Splatting,” CVM 2026.
[6] Jiang et al., “GS-LiDAR: Generating Realistic LiDAR Point Clouds with Panoramic Gaussian Splatting,” ICLR 2025.
[7] Patakin et al., “Unified Sensor Simulation for Autonomous Driving,” IJCAI 2026.