acceptodds
Under review as a conference paper at ICLR 2027

LiFTeR: Generating LiDAR Gaussian Fields from Surround-View Cameras

Abstract

High-fidelity LiDAR simulation requires modeling both scene geometry and sensor-specific return characteristics. Existing LiDAR generation methods often synthesize range, intensity, and ray-drop maps in image space, without an explicit 3D representation underlying these observations. Gaussian-based LiDAR reconstruction offers such a representation, but typically relies on per-scene optimization with real LiDAR measurements. We present LiFTeR, a cross-modal generative framework that bridges these approaches by producing renderable LiDAR Gaussian fields from surround-view images, without requiring target-scene LiDAR measurements or per-scene optimization at inference time. A diffusion transformer generates a latent representation conditioned on multi-view images, capturing scene geometry and LiDAR return characteristics. A ray-anchored Gaussian decoder then maps this latent representation to Gaussian primitives with explicit geometric parameters, intensity attributes, and ray-drop probabilities, which a panoramic Gaussian splatting renderer converts into range, intensity, and ray-drop maps. The framework additionally supports feed-forward conversion of observed LiDAR into Gaussian fields, enabling generation and reconstruction within a shared representation. On nuScenes, LiFTeR achieves the state-of-the-art MMD and JSD among LiDAR generation methods. Without target-scene LiDAR, it outperforms per-scene reconstruction methods on most depth and intensity metrics, while reducing the time to construct a scene representation from about an hour to 3 minutes. With LiDAR input, LiFTeR reduces the depth and intensity RMSE of novel view synthesis relative to baselines within 0.5 minutes per scene.

Primary area:
generative models
Keywords:
LiDAR Generation, Cross-Modal Generation, Gaussian Splatting, Feed-Forward Reconstruction, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nQpgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LiFTeR generates renderable LiDAR Gaussian fields from surround-view camera images without requiring target-scene LiDAR measurements or per-scene optimization at inference time. A pretrained multi-view diffusion transformer generates LiDAR latents conditioned on camera images. A ray-anchored decoder then converts these latents into 2D Gaussian primitives with geometry, intensity, and ray-drop attributes. A panoramic renderer and a ray-drop refinement U-Net produce the final scans. The same Gaussian decoder also supports feed-forward reconstruction from observed LiDAR scans. Experiments on nuScenes evaluate conditional generation, comparisons with per-scene reconstruction methods, novel-view synthesis, and component ablations. Table 1 reports approximately 35% and 36.5% reductions in MMD and JSD relative to X-Drive. Table 2 shows strong performance on most depth and intensity metrics. Novel-view experiments demonstrate some ability to reuse the generated representation, although the camera-conditioned variant remains behind reconstruction baselines in point-cloud geometry. The appendix provides qualitative results for a denser sensor configuration and downstream object detection.

Critical strengths
  1. A useful unification of camera-conditioned generation and feed-forward reconstruction. LiFTeR produces an explicit, renderable representation without target-scene LiDAR or per-scene optimization, while also accepting observed LiDAR through the same decoder. This provides a more flexible interface than generating isolated scans and is the paper’s most consequential contribution.

  2. Encouraging generation quality and scene-construction efficiency. Table 1 shows meaningful MMD/JSD improvements over the closest camera-conditioned baseline, X-Drive, while Table 2 reports strong depth and intensity results. The LiDAR-input variant also achieves competitive depth and intensity RMSE in Table 3 with approximately 0.5 minutes of processing per scene. These results support the potential of a learned feed-forward approach.

Critical weaknesses
  1. The central benefit of a reusable 3D field is insufficiently established. The most direct controlled comparison—Gaussian rendering versus reprojection of a range image decoded from the same latent—is demonstrated on only one scene with limited object motion (§4.4). This provides preliminary evidence, but does not establish consistent benefits across scenes, larger viewpoint changes, or newly visible regions. In Table 3, the camera-conditioned variant obtains a CD of 1.7357 and an F-score of 0.4733, both worse than all reconstruction baselines. Its lack of LiDAR input is an important distinction, so this does not invalidate the approach. Nevertheless, the evidence leaves a gap between supporting novel-view rendering and producing reliable novel-view geometry, which directly affects the representation’s value for simulation.

  2. The experiments do not sufficiently isolate the proposed technical contributions. Latent-to-Gaussian decoding has precedents such as Lyra, while LiDAR-oriented 2D Gaussian rendering and return-attribute modeling appear in GS-LiDAR. The contribution therefore depends substantially on the effectiveness of the LiDAR-specific adaptations. Table 5 evaluates 2D versus 3D Gaussians, geometric regularization, and ray-drop refinement, but does not directly ablate ray anchoring. The paper also does not report the main generation metrics when the same generated latents are decoded using the original LiDAR VAE decoder. It is consequently difficult to determine whether the gains in Table 1 primarily come from the pretrained generative backbone or the proposed Gaussian decoder.

Questions for authors
  1. Contribution attribution / additional validation: What are the MMD/JSD and paired geometric errors when the same generated latents are decoded using the original LiDAR VAE decoder versus the Gaussian decoder? How does performance change when ray anchoring is removed while other settings are retained? These comparisons would directly affect my assessment of the proposed design.

  2. 3D representation benefit / additional validation: Can the same-latent reprojection comparison in Figure 7 be evaluated across multiple held-out scenes, with geometric error and coverage reported against actual translation and rotation magnitudes? In particular, do the benefits extend to newly visible regions, beyond improved coverage of previously observed surfaces?

  3. Multi-frame mechanism clarification: How many neighboring frames are used for each query in Table 3, and are both past and future frames included? Since §3.3 describes taking the union of per-frame Gaussians in world coordinates, how are moving objects aligned to the query time, and how is duplication across timestamps avoided?

  4. Efficiency protocol clarification: How many input and query frames are included in the timings in Tables 2 and 3? What inference hardware and GPU counts are used? Do the reported times include encoding, diffusion sampling, Gaussian decoding, and final rendering? These details are necessary to interpret the claimed speedups.

3 comments
AuthorsOpenReview ↗

Response to Reviewer nQpg (1/2)

We sincerely thank the reviewer for the careful assessment and constructive feedback. We appreciate the recognition of LiFTeR’s unified framework for camera-conditioned LiDAR generation and feed-forward reconstruction, as well as its generation quality and computational efficiency.

We address the reviewer’s concerns regarding (1) the contribution of the Gaussian decoder, (2) the benefits of an explicit 3D representation, (3) multi-frame consistency, and (4) computational efficiency below.

W2&Q1. Contribution attribution / additional validation: What are the MMD/JSD and paired geometric errors when the same generated latents are decoded using the original LiDAR VAE decoder versus the Gaussian decoder? How does performance change when ray anchoring is removed while other settings are retained? These comparisons would directly affect my assessment of the proposed design.

Response: We thank the reviewer for highlighting the importance of isolating the contributions of the Gaussian decoder and ray anchoring. We have added the following comparisons and clarified the purposes of Tabs. 1–3 in the revised manuscript.

  1. Comparison between the VAE and Gaussian decoder. We have revised the descriptions of all three tables to provide more detailed explanations of their respective purposes and the LiFTeR modules they evaluate. In Tab. 1, the LiDAR scans are decoded by the VAE to evaluate the performance of the diffusion model. Tab. 2 and Tab. 3 aim to evaluate the Gaussian decoder. Under the evaluation protocol of Tab.2, the results obtained with the VAE decoder are reported below:
\hline \text{Decoder} & \text{CD}\downarrow & \text{F-score}\uparrow & \text{Depth RMSE}\downarrow & \text{Depth PSNR}\uparrow & \text{Intensity RMSE}\downarrow & \text{Intensity PSNR}\uparrow \\\\ \hline \text{VAE decoder} & \text{1.6131} & \text{0.6009} & \text{7.2756} & \text{17.0658}& \text{0.0413} & \text{24.7617} \\\\ \text{Gaussian decoder} & \text{1.5955} & \text{0.6118} & \text{7.0761} & \text{17.2053} & \text{0.0449} & \text{24.7396} \\\\ \hline \end{array}$$ 2. **Ray-anchoring ablation.** In LiFTeR, the ray direction of each Gaussian is fixed by the sensor configuration, so its position is described by a single depth along the ray. In the ablation, we replace this depth with a free xyz position and retrain the decoder with all other settings unchanged. The results are shown below.

\begin{array}{lcccccc} \hline \text{Method} & \text{CD}\downarrow & \text{F-score}\uparrow & \text{Depth MedAE}\downarrow & \text{Depth PSNR}\uparrow & \text{Intensity MedAE}\downarrow & \text{Intensity PSNR}\uparrow \\ \hline \text{Full model} & \text{1.5955} & \text{0.6118} & \text{0.1263} & \text{17.2053} & \text{0.0073} & \text{24.7396} \\ \text{w/o ray anchoring} & \text{14.4976} & \text{0.3992} & \text{1.0949} & \text{13.1531} & \text{0.0689} & \text{19.2761} \\ \hline \end{array}

Withoutfixingtheraydirection,thedecoderfailstolearnvalidgeometry.CDincreasesfrom1.59to14.49,whichindicatesthatthepredictedscenegeometryisentirelyincorrect.Wethankthereviewerforsuggestingtheray−anchoringablation.ThisablationisausefulcontrolandhelpsreadersbetterunderstandwhyLiFTeRadoptsray−anchoringinsteadofpredictingafree3Dposition.Wehaveincludedtheresultsintheablationstudyoftherevisedmanuscript. Without fixing the ray direction, the decoder fails to learn valid geometry. CD increases from 1.59 to 14.49, which indicates that the predicted scene geometry is entirely incorrect. We thank the reviewer for suggesting the ray-anchoring ablation. This ablation is a useful control and helps readers better understand why LiFTeR adopts ray-anchoring instead of predicting a free 3D position. We have included the results in the ablation study of the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer nQpg (2/2)

W1&Q2. 3D representation benefit / additional validation: Can the same-latent reprojection comparison in Figure 7 be evaluated across multiple held-out scenes, with geometric error and coverage reported against actual translation and rotation magnitudes? In particular, do the benefits extend to newly visible regions, beyond improved coverage of previously observed surfaces?

Response: We appreciate the reviewer’s observation that the original novel-view comparison was limited to one scene.

In the Appendix, we extend the same-latent reprojection comparison in Figure 7 to multiple held-out scenes, including scenes with dynamic objects, and report the ego-pose translation and rotation magnitudes for each case. We evaluate novel-view rendering quality using PSNR and geometric accuracy using the range RMSE between rendered and ground-truth LiDAR ranges. The results show that the Gaussian representation performs better in both rendering quality and geometric accuracy.

We further clarify the rationale for using a Gaussian representation from the following three perspectives.

First, decoding LiDAR latents with the Gaussian decoder at reference views and rendering Gaussians at novel views yields more reliable results than directly decoding with the pretrained LiDAR VAE and then reprojecting range and intensity maps at novel views, as shown in Figure 7 and by more examples in Appendix.

Second, a Gaussian field serves as a reusable scene representation for close-loop autonomous driving simulation, enabling the LiDAR sensor, in terms of range, intensity, and ray-drop maps, to be simulated via rendering at arbitrary ego pose without relying on any camera priors.

Third, using Gaussian rendering as the backend of LiDAR sensor simulator avoids the need to run the pretrained DiT at every simulation timestamp, which substantially reduces computational efforts.

Q3. Multi-frame mechanism clarification: How many neighboring frames are used for each query in Table 3, and are both past and future frames included? Since §3.3 describes taking the union of per-frame Gaussians in world coordinates, how are moving objects aligned to the query time, and how is duplication across timestamps avoided?

Response: We thank the reviewer for the insightful comments. Each query frame in Tab. 3 is rendered using the preceding and following frames. The visualizations included in the paper do not explicitly isolate dynamic objects. In fact, in our LiDAR simulation application, we use the 3D annotations provided by nuScenes to decompose the predicted LiDAR Gaussians into static and dynamic parts. We aggregate the dynamic Gaussians separately across multiple frames in the canonical coordinate system and then render together with static Gaussians in the world coordinate system at inference time.

Moreover, we include the visualizations of dynamic objects in the revised manuscript now, demonstrating the capability of our method to produce high-quality renderings even in scenes with many dynamic objects. In future work, we will extend our model to predict the perception outputs directly, so that the static–dynamic decomposition no longer relies on annotations.

Q4. Efficiency protocol clarification: How many input and query frames are included in the timings in Tables 2 and 3? What inference hardware and GPU counts are used? Do the reported times include encoding, diffusion sampling, Gaussian decoding, and final rendering? These details are necessary to interpret the claimed speedups.

Response: We agree that a more detailed description of the efficiency protocol is necessary for reproducibility. Thus, we clarify the experimental setup as follows:

  • Hardware: All LiFTeR inference is run on a single NVIDIA H20 GPU.
  • Tab. 2: One in every eight frames is held out for testing. At each test timestamp, LiFTeR (C→L) takes images from all six cameras as input.
  • Tab. 3: We use every other frame as input and render each intervening frame from its two neighbors.
  • Reported time: The reported times exclude model loading only. All remaining steps are included, i.e., VAE encoding, diffusion model denoising, Gaussian decoding, and final rendering.

We provide more detailed implementation details in the revised manuscript to facilitate reproducibility.

Summary:

We update the manuscript to:

  1. Include the controlled VAE-versus-Gaussian decoding comparison and ray-anchoring ablation (W2&Q1).
  2. Expand novel-view evaluation to multiple scenes and viewpoint changes (W1&Q2).
  3. Clarify multi-frame aggregation and dynamic-object limitations (Q3).
  4. Provide reproducible evaluation and runtime details (Q4).

We thank the reviewer again for the valuable suggestions, which help us strengthen the evidence supporting LiFTeR’s contributions.

Thank you for the thoughtful responses and the effort put into the additional experiments and revisions.

I appreciate the VAE-versus-Gaussian decoder comparison, the ray-anchoring ablation, the expanded novel-view evaluation, and the clarifications of the multi-frame and runtime protocols. These additions have addressed most of my concerns and strengthened the empirical support for the proposed framework. I will maintain my positive assessment.

Reviewer_fjx7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a generalizable LiDAR resimulation approach that combines LiDAR generation approach with Gaussian-based resimulation approach. It first generates LiDAR latent tokens conditioned on multi-view camera images, then trains a Gaussian decoder adapted from Long-LRM to outputs ray-anchored 2DGS. The method shows promising results for resimulation, with reasonable and expected failure case.

Critical strengths
  • The paper is well motivated, and the pipeline is well thought-out and technically sound.
  • Quantitative results and ablation studies are thorough, with demonstrated downstream usecases and sufficient discussion of limitations and failure cases.
Critical weaknesses
  • Metrics for LiDAR generation are a bit sparse. Current methods also introduce perceptual metrics (FPVD, FSVD, ... as introduced in [1]) beyond the distributional metrics such as MMD/JSD.
  • Some of the experiment are unclear which causes some confusion and ambiguity. Please refer to the questions section below.

[1] Ran et. al., Towards Realistic Scene Generation with LiDAR Diffusion Models, CVPR 2024

Questions for authors

Question:

  • For table 2 evaluation, do you directly use camera images of the test timestep to generate LiDAR, or use the camera images at the training timestep and then render at the test timestep?
  • For novel view synthesis evaluation, how do you handle dynamic objects? Sec 4.3 mentioned the test frame is rendered from the neighboring training frame, but the location of different actors must differ between frames. Moreover, how are multiple per-frame LiDAR gaussian merged together?
  • Small question, figure 12 shows LiDAR without the typical nuScenes ego mask distribution. Is there a reason for this?

Suggestions:

  • Most visualizations except the first are in range images, which is nice but hard to grasp the geometry of the generated LiDAR. Including more visualization in the 3D space compared with other baselines and GT would be nice to have.
3 comments
AuthorsOpenReview ↗

Response to Reviewer fjx7

We sincerely thank the reviewer for the positive assessment of our motivation, technical design, and experimental evaluation. We appreciate the suggestions for improving the evaluation and presentation of results.

We address the reviewer’s concerns below.

W1: Metrics for LiDAR generation are a bit sparse. Current methods also introduce perceptual metrics (FPVD, FSVD, ... as introduced in [1]) beyond the distributional metrics such as MMD/JSD.

Response: Since our main baseline, X-Drive, does not release its pretrained models, we follow the training and evaluation protocol of its paper and adopt the metrics it reports (MMD and JSD), as shown in Tab.1, which allows a fair and direct comparison. Following the reviewer's suggestion, we additionally report FSVD and FPVD. The results are shown in the table below.

\hline \text{Method} & \text{FSVD} \downarrow & \text{FPVD} \downarrow \\\\ \hline \text{Text2LiDAR} & 69.92 & 62.12 \\\\ \text{UniScene} & 131.74 & 103.28 \\\\ \text{OpenDWM} & 60.09 & 50.39 \\\\ \text{LiFTeR (Ours)} & \mathbf{22.92} & \mathbf{18.18} \\\\ \hline \end{array}$$ LiFTeR achieves the lowest FSVD and FPVD among the compared methods. ***Q1. For table 2 evaluation, do you directly use camera images of the test timestep to generate LiDAR, or use the camera images at the training timestep and then render at the test timestep?*** ***Response:*** We directly use the camera images at the test timestep to generate the LiDAR scans. Those test scenes are excluded from the training dataset used for both the DiT model and the Gaussian decoder. ***Q2. For novel view synthesis evaluation, how do you handle dynamic objects? Sec 4.3 mentioned the test frame is rendered from the neighboring training frame, but the location of different actors must differ between frames. Moreover, how are multiple per-frame LiDAR gaussian merged together?*** ***Response:*** We thank the reviewer for the insightful comments. The visualizations included in the paper do not explicitly isolate dynamic objects. In fact, in our LiDAR simulation application, we use the 3D annotations provided by nuScenes to decompose the predicted LiDAR Gaussians into static and dynamic parts. We aggregate the dynamic Gaussians separately across multiple frames in the canonical coordinate system and then render together with static Gaussians in the world coordinate system at inference time. Moreover, we include the visualizations of dynamic objects in the revised manuscript now, demonstrating the capability of our method to produce high-quality renderings even in scenes with many dynamic objects. In future work, we will extend our model to predict the perception outputs directly, so that the static–dynamic decomposition no longer relies on annotations. ***Q3. Small question, figure 12 shows LiDAR without the typical nuScenes ego mask distribution. Is there a reason for this?*** ***Response:*** Fig. 12 shows only the front-view projection. Because the LiDAR results are generated from all six camera views, regions occluded by the ego vehicle in one view are recovered from neighbors such as the front-left and front-right cameras. We have clarified the dataset and sensor setup in the caption. ***Q4. Most visualizations except the first are in range images, which is nice but hard to grasp the geometry of the generated LiDAR. Including more visualization in the 3D space compared with other baselines and GT would be nice to have.*** ***Response:*** We thank the reviewer for the suggestion. In the Appendix, we additionally provide more 3D visualizations comparing our results with the baselines and the ground truth. ***Summary:*** We revise the manuscript as follows: 1. Add perceptual metrics (FSVD and FPVD) in Tab. 1 alongside MMD and JSD, following the protocol of [1]. LiFTeR achieves the lowest FSVD and FPVD among the compared methods. 2. Clarify the evaluation protocol of Tab. 2 in Sec. 4.2. 3. Clarify the multi-frame mechanism and the handling of dynamic objects (Q2). The corresponding qualitative visualizations are added to the revised paper. 4. Clarify the caption of Fig. 12(Q3). 5. Provide more 3D visualizations comparing our results with the baselines and the ground truth in the Appendix. [1] Ran et. al., Towards Realistic Scene Generation with LiDAR Diffusion Models, CVPR 2024 We thank the reviewer again for the positive assessment and for the suggestions that helped us improve the evaluation and the presentation of the results.

Thank you for the thoughtful rebuttal. The perceptual metrics are convincing.

However, I still have a concern regarding using test images as conditioning. I believe several baselines do not have access to such strong condition for their input, resulting in unfair advantage. I suggest to have an experiment that either extend baselines to use test images, or restrict your method from using such input. A simple point warping from training timestep to test timestep would suffice, even if metrics aren’t as strong.

I will keep my rating in favor of acceptance due to the strong visual and quantitative results of the generation.

Response: We thank the reviewer for raising this concern, and we agree that the baselines are not perfectly equally compared. We address the two parts of the suggestion separately.

On the comparison itself. Tab. 1 evaluates camera-conditioned LiDAR generation, where the camera image of the query timestep is the input by design, while the baselines we compare against are conditioned on text or layout, or reconstructed from LiDAR only. We will state this explicitly in the revised manuscript, so that Tab. 1 is read as a comparison of output realism across condition types rather than a like-for-like comparison under identical inputs.

On the suggested experiment. We agree that the restricted variant is the right control. First, we report LiFTeR (L→L) from Tab. 3, which receives no camera input at all and therefore bounds the accuracy achievable without the camera condition. Second, extending the compared baselines to use test images as conditions is not feasible for us, since each method is designed for a specific input modality. For example, layout-conditioned methods take layouts as input and are not designed to use test images as conditions.

Overall, we regard the camera condition as the appropriate setting for autonomous driving applications, since camera images, unlike text or layout, are commonly available in real scene datasets. It is also the setting this paper aims to study. The results shown in this paper support the potential of this setting.

Reviewer_jQdUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

LiFTeR is a framework for generating LiDAR Gaussian fields, and thus LiDAR scans, from surround-view images. A multi-view diffusion transformer generates LiDAR latents in the RangeLDM VAE space, conditioned on the camera latents. A ray-anchored decoder maps these latents to 2D Gaussian primitives, and a panoramic rasterizer renders the primitives into range, intensity, and ray-drop maps. The same decoder also accepts latents encoded from observed LiDAR scans, which enables feed-forward LiDAR-to-Gaussian reconstruction. Conceptually, the method combines Gaussian-based LiDAR reconstruction (e.g., GS-LiDAR) with diffusion-based generation. As a result, a renderable scene representation can be obtained without target-scene LiDAR or per-scene optimization.

Although most of the individual components are existing techniques, their integration is meaningful and provides practical benefits. On nuScenes, LiFTeR achieves the best MMD and JSD among LiDAR generation methods. It also reduces scene construction time from roughly one hour (per-scene reconstruction) to a few minutes.

Critical strengths
  1. The paper extends Gaussian-based LiDAR reconstruction to camera-conditioned generation. This produces an explicit, renderable 3D representation with intensity and ray-drop attributes that can be rendered directly for LiDAR simulation. A shared latent space allows the same decoder to perform both camera-to-LiDAR generation and feed-forward LiDAR reconstruction. This design is practically useful for scalable LiDAR simulation, and it is the main reason I support acceptance.

  2. LiFTeR reduces MMD and JSD by about 35% relative to the camera-conditioned X-Drive (Tab. 1), and builds a scene representation in about 3 minutes instead of roughly an hour (Tab. 2). In the LiDAR-input setting, LiFTeR (L->L) achieves lower depth and intensity RMSE than SplatAD in 0.5 minutes per scene (Tab. 3). Together, these results show that the generated Gaussian representation is useful in practice.

Critical weaknesses

Overall I support acceptance. The two points below are the main reasons my rating is not higher.

  1. The multi-frame field is constructed as a union of per-frame primitives in the world frame (L224–225). There is no dynamic decomposition or deformation, so moving objects are likely to produce duplicated/inconsistent surfaces when the field is rendered at intermediate timestamps. In the camera-to-LiDAR setting, the latent of each frame also appears to be sampled independently, so even static geometry may be inconsistent across frames. The trajectory extrapolation experiment in Sec. 4.4 uses a single scene that was explicitly chosen for its limited object motion (L420), so it does not test this case.

  2. The pretrained LiDAR VAE can already decode the generated latents into range and intensity maps. The paper itself reports that Gaussian rendering "starts at similar reference-pose accuracy" as reprojecting the VAE-decoded range image (L428–429). The benefit for novel-view rendering is shown (Fig. 7), but it is modest (1.20 dB at T = 30) and measured on a single scene. Because each primitive is anchored to a ray of the reference scan, the field mainly covers surfaces visible from the reference pose. This raises the question of how much the explicit 3D representation adds beyond a range image, which the paper itself criticizes for not recovering newly visible surfaces (L136). Also, why not simply train the diffusion model to generate range, intensity, and ray-drop maps directly, and use those for LiDAR simulation?

Questions for authors
  1. How does the framework keep Gaussian primitives generated at different timestamps consistent, especially for moving objects? Are the latents of different frames sampled independently in the camera-to-LiDAR setting? Could the authors provide quantitative results, or at least qualitative results, on scenes with significant object motion?

  2. Could the LiDAR latent diffusion model directly generate range, intensity, and ray-drop maps, followed by a similar U-Net refinement? How would this baseline compare with the proposed Gaussian pipeline at the reference view in Tab. 1 and Tab. 2? Relatedly, are the scans evaluated in Tab. 1 rendered from the Gaussian field or decoded by the VAE?

  3. In Tab. 2, at each held-out frame, LiFTeR is conditioned on the camera images of that same timestamp. The per-scene methods, in contrast, must interpolate from the LiDAR of other frames. LiFTeR has also been trained on 700 nuScenes scenes. Could the authors discuss how this difference in available information affects the comparison? The claim that LiFTeR "outperforms per-scene reconstruction methods" may need to be stated more carefully. In the novel-view setting of Tab. 3, LiFTeR (C->L) has the highest Chamfer distance and the lowest F-score among all methods.

  4. The range map is a 360° cylindrical projection, but the camera condition consists of six perspective views concatenated with the LiDAR latent as separate views (L190–191). The camera–LiDAR correspondence therefore has to be learned implicitly. Have the authors considered conditioning on a single 360° equirectangular panorama, which is naturally column-aligned with the range map? Moreover, since paired camera–LiDAR data are scarce (L161), could recent panorama generation models (e.g., DiT360 [Feng et al., CVPR 2026]) be used to synthesize additional training data? For example, they could generate diverse appearances (e.g., night or adverse weather) for existing LiDAR scans, which may also help with the failure cases in Fig. 13.

2 comments
AuthorsOpenReview ↗

Response to Reviewer jQdU (1/2)

We sincerely thank the reviewer for the thoughtful feedback and for supporting the acceptance of our paper. We appreciate the recognition of the practical benefits of unifying camera-conditioned LiDAR generation and feed-forward Gaussian reconstruction.

We address the reviewer’s concerns below.

W1&Q1. How does the framework keep Gaussian primitives generated at different timestamps consistent, especially for moving objects? Are the latents of different frames sampled independently in the camera-to-LiDAR setting? Could the authors provide quantitative results, or at least qualitative results, on scenes with significant object motion?

Response: We thank the reviewer for the insightful comments. The visualizations included in the paper do not explicitly isolate dynamic objects. In fact, in our LiDAR simulation application, we use the 3D annotations provided by nuScenes to decompose the predicted LiDAR Gaussians into static and dynamic parts. We aggregate the dynamic Gaussians separately across multiple frames in the canonical coordinate system and then render together with static Gaussians in the world coordinate system at inference time.

Moreover, we include the visualizations of dynamic objects in the revised manuscript now, demonstrating the capability of our method to produce high-quality renderings even in scenes with many dynamic objects. In future work, we will extend our model to predict the perception outputs directly, so that the static–dynamic decomposition no longer relies on annotations.

W2. The pretrained LiDAR VAE can already decode the generated latents into range and intensity maps. The paper itself reports that Gaussian rendering "starts at similar reference-pose accuracy" as reprojecting the VAE-decoded range image (L428–429). The benefit for novel-view rendering is shown (Fig. 7), but it is modest (1.20 dB at T = 30) and measured on a single scene. Because each primitive is anchored to a ray of the reference scan, the field mainly covers surfaces visible from the reference pose. This raises the question of how much the explicit 3D representation adds beyond a range image, which the paper itself criticizes for not recovering newly visible surfaces (L136). Also, why not simply train the diffusion model to generate range, intensity, and ray-drop maps directly, and use those for LiDAR simulation?

Response: We thank the reviewer for this insightful question, which addresses a core design choice in our work. We answer it in three points.

First, decoding LiDAR latents with the Gaussian decoder at reference views and rendering Gaussians at novel views yields more reliable results than directly decoding with the pretrained LiDAR VAE and reprojecting range and intensity maps at novel views, as shown in Figure 7 and by more examples in Appendix.

Second, a Gaussian field serves as a reusable scene representation for close-loop autonomous driving simulation, enabling the LiDAR sensor, in terms of range, intensity, and ray-drop maps, to be simulated via rendering at arbitrary ego pose without relying on any camera priors.

Third, using Gaussian rendering as the backend of LiDAR sensor simulator avoids the need to run the pretrained DiT at every simulation timestamp, which substantially reduces computational efforts.

Q2. Could the LiDAR latent diffusion model directly generate range, intensity, and ray-drop maps, followed by a similar U-Net refinement? How would this baseline compare with the proposed Gaussian pipeline at the reference view in Tab. 1 and Tab. 2? Relatedly, are the scans evaluated in Tab. 1 rendered from the Gaussian field or decoded by the VAE?

Response: Yes, we can obtain the range, intensity and ray-drop maps directly by decoding the DiT generated latents with pretrained LiDAR VAE, which yields a natural baseline without Gaussian decoder. We have revised the descriptions of all three tables to provide more detailed explanations of their respective purposes and the LiFTeR modules they evaluate. In Tab. 1, the LiDAR scans are decoded by the VAE to evaluate the performance of the diffusion model. Tab. 2 and Tab. 3 aim to evaluate the Gaussian decoder. Under the evaluation protocol of Tab.2, the results obtained with the VAE decoder are reported below: \begin{array}{lcccccc} \hline \text{Decoder} & \text{CD}\downarrow & \text{F-score}\uparrow & \text{Depth RMSE}\downarrow & \text{Depth PSNR}\uparrow & \text{Intensity RMSE}\downarrow & \text{Intensity PSNR}\uparrow \\ \hline \text{VAE decoder} & \text{1.6131} & \text{0.6009} & \text{7.2756} & \text{17.0658}& \text{0.0413} & \text{24.7617} \\ \text{Gaussian decoder} & \text{1.5955} & \text{0.6118} & \text{7.0761} & \text{17.2053} & \text{0.0449} & \text{24.7396} \\ \hline \end{array}

AuthorsOpenReview ↗

Response to Reviewer jQdU (2/2)

Q3. In Tab. 2, at each held-out frame, LiFTeR is conditioned on the camera images of that same timestamp. The per-scene methods, in contrast, must interpolate from the LiDAR of other frames. LiFTeR has also been trained on 700 nuScenes scenes. Could the authors discuss how this difference in available information affects the comparison? The claim that LiFTeR "outperforms per-scene reconstruction methods" may need to be stated more carefully. In the novel-view setting of Tab. 3, LiFTeR (C->L) has the highest Chamfer distance and the lowest F-score among all methods.

Response: We agree with the reviewer that LiFTeR (C→L) does not achieve the best performance in the novel-view synthesis setting of Tab. 3. We have revised the manuscript to describe the results more precisely.

Q4. The range map is a 360° cylindrical projection, but the camera condition consists of six perspective views concatenated with the LiDAR latent as separate views (L190–191). The camera–LiDAR correspondence therefore has to be learned implicitly. Have the authors considered conditioning on a single 360° equirectangular panorama, which is naturally column-aligned with the range map? Moreover, since paired camera–LiDAR data are scarce (L161), could recent panorama generation models (e.g., DiT360 [Feng et al., CVPR 2026]) be used to synthesize additional training data? For example, they could generate diverse appearances (e.g., night or adverse weather) for existing LiDAR scans, which may also help with the failure cases in Fig. 13.

Response: We thank the reviewer for the suggestions. We agree that panoramic conditioning and panorama-based data synthesis (e.g., DiT360) are probably valuable for LiDAR representation and scene generation, and we would consider this promising direction in our future work. We have updated both to the Discussion of future work in the revised manuscript.

Summary:

We revise the manuscript as follows:

  1. Clarify the multi-frame mechanism and the handling of dynamic objects (W1&Q1). The corresponding qualitative visualizations are added to the Appendix.
  2. Clarify the motivation for using Gaussian fields (W2).
  3. Clarify the roles of the VAE and Gaussian decoders and add a reference-view comparison between them (Q2).
  4. Revise several imprecise statements in the manuscript (Q3).
  5. Add the discussion that panoramic conditioning and panorama-based data synthesis will be carefully considered in our future work (Q4).

We thank the reviewer again for the careful and constructive review, which has helped us strengthen both the empirical evidence and the clarity of the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has substantially improved our paper. We have uploaded a revised manuscript, in which all changes are highlighted in blue. The main revisions are summarized below.

Experiments

  • Tab. 1: Added the perceptual metrics FSVD and FPVD alongside MMD and JSD. LiFTeR achieves the best results on both. (Reviewer fjx7)
  • Tab. 5: Added two ablations, (i) removing ray anchoring and (ii) comparing the Gaussian decoder with the pretrained LiDAR VAE decoder on the same generated latents. (Reviewers nQpg, jQdU)
  • Appendix E.1: Added a qualitative ablation on dynamic-object decomposition for multi-frame rendering. (Reviewers jQdU, fjx7, nQpg)
  • Appendix E.2: Extended the comparison between Gaussian rendering and range-image reprojection in Fig. 7 to more additional scenes, including scenes with dynamic objects, with the corresponding ego-pose translation and rotation. (Reviewers nQpg, jQdU)
  • Appendix E.3: Added more 3D visualizations comparing LiFTeR with baselines and the ground truth. (Reviewer fjx7)

Clarifications

  • Sec. 3.3: Clarified the motivation for generating an explicit Gaussian field instead of a range image. (Reviewer jQdU)
  • Sec. 4.2: Clarified that the scans in Tab. 1 are decoded by the LiDAR VAE decoder, while Tabs. 2 and 3 evaluate scans rendered from the Gaussian field. (Reviewers jQdU, nQpg)
  • Appendix F: Added the efficiency evaluation protocol, including hardware, frame settings, and the operations included in the reported times. (Reviewer nQpg)
  • Fig. 12: Clarified the caption regarding points generated in regions without real returns. (Reviewer fjx7)
  • Appendix C: Added future work on panoramic conditioning and panorama-based data synthesis (e.g., DiT360). (Reviewer jQdU)

We hope these revisions address the reviewers' concerns, and we are happy to provide further clarification if needed.

AuthorsOpenReview ↗

Key Contributions of LiFTeR

Dear AC and Reviewers,

We thank all reviewers for their constructive feedback and active engagement. To facilitate the final discussion, we briefly summarize the contributions of LiFTeR. The revisions made during the rebuttal are listed in our earlier comment, Summary of Revisions.

LiFTeR generates renderable LiDAR Gaussian fields from surround-view cameras, without target-scene LiDAR or per-scene optimization. A camera-conditioned diffusion transformer generates LiDAR latents, and a ray-anchored decoder converts them into 2D Gaussians carrying geometry, intensity, and ray-drop. The same decoder also reconstructs observed LiDAR scans in a feed-forward manner, so that generation and reconstruction share one explicit scene representation. Once generated, the Gaussian field serves as a reusable scene representation for closed-loop simulation, which renders range, intensity, and ray-drop maps at arbitrary ego poses without camera input or rerunning the diffusion model.

We are grateful that the reviewers recognized the practical value of unifying camera-conditioned generation and feed-forward reconstruction for scalable LiDAR simulation (jQdU, nQpg), and found the pipeline well motivated and technically sound with thorough experiments (fjx7). In summary, LiFTeR

  • achieves state-of-the-art camera-to-LiDAR generation quality on nuScenes,
  • constructs a renderable scene representation in minutes rather than the hour-level optimization required by per-scene reconstruction,
  • supports both camera-conditioned generation and fast feed-forward reconstruction from LiDAR within a single framework, and
  • also performs well on real-world driving data with a denser multi-LiDAR setup, making it applicable to practical LiDAR simulation.

We thank the AC and all reviewers again for their time and valuable suggestions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.