- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LiFTeR generates renderable LiDAR Gaussian fields from surround-view camera images without requiring target-scene LiDAR measurements or per-scene optimization at inference time. A pretrained multi-view diffusion transformer generates LiDAR latents conditioned on camera images. A ray-anchored decoder then converts these latents into 2D Gaussian primitives with geometry, intensity, and ray-drop attributes. A panoramic renderer and a ray-drop refinement U-Net produce the final scans. The same Gaussian decoder also supports feed-forward reconstruction from observed LiDAR scans. Experiments on nuScenes evaluate conditional generation, comparisons with per-scene reconstruction methods, novel-view synthesis, and component ablations. Table 1 reports approximately 35% and 36.5% reductions in MMD and JSD relative to X-Drive. Table 2 shows strong performance on most depth and intensity metrics. Novel-view experiments demonstrate some ability to reuse the generated representation, although the camera-conditioned variant remains behind reconstruction baselines in point-cloud geometry. The appendix provides qualitative results for a denser sensor configuration and downstream object detection.
Critical strengths
-
A useful unification of camera-conditioned generation and feed-forward reconstruction. LiFTeR produces an explicit, renderable representation without target-scene LiDAR or per-scene optimization, while also accepting observed LiDAR through the same decoder. This provides a more flexible interface than generating isolated scans and is the paper’s most consequential contribution.
-
Encouraging generation quality and scene-construction efficiency. Table 1 shows meaningful MMD/JSD improvements over the closest camera-conditioned baseline, X-Drive, while Table 2 reports strong depth and intensity results. The LiDAR-input variant also achieves competitive depth and intensity RMSE in Table 3 with approximately 0.5 minutes of processing per scene. These results support the potential of a learned feed-forward approach.
Critical weaknesses
-
The central benefit of a reusable 3D field is insufficiently established. The most direct controlled comparison—Gaussian rendering versus reprojection of a range image decoded from the same latent—is demonstrated on only one scene with limited object motion (§4.4). This provides preliminary evidence, but does not establish consistent benefits across scenes, larger viewpoint changes, or newly visible regions. In Table 3, the camera-conditioned variant obtains a CD of 1.7357 and an F-score of 0.4733, both worse than all reconstruction baselines. Its lack of LiDAR input is an important distinction, so this does not invalidate the approach. Nevertheless, the evidence leaves a gap between supporting novel-view rendering and producing reliable novel-view geometry, which directly affects the representation’s value for simulation.
-
The experiments do not sufficiently isolate the proposed technical contributions. Latent-to-Gaussian decoding has precedents such as Lyra, while LiDAR-oriented 2D Gaussian rendering and return-attribute modeling appear in GS-LiDAR. The contribution therefore depends substantially on the effectiveness of the LiDAR-specific adaptations. Table 5 evaluates 2D versus 3D Gaussians, geometric regularization, and ray-drop refinement, but does not directly ablate ray anchoring. The paper also does not report the main generation metrics when the same generated latents are decoded using the original LiDAR VAE decoder. It is consequently difficult to determine whether the gains in Table 1 primarily come from the pretrained generative backbone or the proposed Gaussian decoder.
Questions for authors
-
Contribution attribution / additional validation: What are the MMD/JSD and paired geometric errors when the same generated latents are decoded using the original LiDAR VAE decoder versus the Gaussian decoder? How does performance change when ray anchoring is removed while other settings are retained? These comparisons would directly affect my assessment of the proposed design.
-
3D representation benefit / additional validation: Can the same-latent reprojection comparison in Figure 7 be evaluated across multiple held-out scenes, with geometric error and coverage reported against actual translation and rotation magnitudes? In particular, do the benefits extend to newly visible regions, beyond improved coverage of previously observed surfaces?
-
Multi-frame mechanism clarification: How many neighboring frames are used for each query in Table 3, and are both past and future frames included? Since §3.3 describes taking the union of per-frame Gaussians in world coordinates, how are moving objects aligned to the query time, and how is duplication across timestamps avoided?
-
Efficiency protocol clarification: How many input and query frames are included in the timings in Tables 2 and 3? What inference hardware and GPU counts are used? Do the reported times include encoding, diffusion sampling, Gaussian decoding, and final rendering? These details are necessary to interpret the claimed speedups.
3 comments
Response to Reviewer nQpg (2/2)
W1&Q2. 3D representation benefit / additional validation: Can the same-latent reprojection comparison in Figure 7 be evaluated across multiple held-out scenes, with geometric error and coverage reported against actual translation and rotation magnitudes? In particular, do the benefits extend to newly visible regions, beyond improved coverage of previously observed surfaces?
Response: We appreciate the reviewer’s observation that the original novel-view comparison was limited to one scene.
In the Appendix, we extend the same-latent reprojection comparison in Figure 7 to multiple held-out scenes, including scenes with dynamic objects, and report the ego-pose translation and rotation magnitudes for each case. We evaluate novel-view rendering quality using PSNR and geometric accuracy using the range RMSE between rendered and ground-truth LiDAR ranges. The results show that the Gaussian representation performs better in both rendering quality and geometric accuracy.
We further clarify the rationale for using a Gaussian representation from the following three perspectives.
First, decoding LiDAR latents with the Gaussian decoder at reference views and rendering Gaussians at novel views yields more reliable results than directly decoding with the pretrained LiDAR VAE and then reprojecting range and intensity maps at novel views, as shown in Figure 7 and by more examples in Appendix.
Second, a Gaussian field serves as a reusable scene representation for close-loop autonomous driving simulation, enabling the LiDAR sensor, in terms of range, intensity, and ray-drop maps, to be simulated via rendering at arbitrary ego pose without relying on any camera priors.
Third, using Gaussian rendering as the backend of LiDAR sensor simulator avoids the need to run the pretrained DiT at every simulation timestamp, which substantially reduces computational efforts.
Q3. Multi-frame mechanism clarification: How many neighboring frames are used for each query in Table 3, and are both past and future frames included? Since §3.3 describes taking the union of per-frame Gaussians in world coordinates, how are moving objects aligned to the query time, and how is duplication across timestamps avoided?
Response: We thank the reviewer for the insightful comments. Each query frame in Tab. 3 is rendered using the preceding and following frames. The visualizations included in the paper do not explicitly isolate dynamic objects. In fact, in our LiDAR simulation application, we use the 3D annotations provided by nuScenes to decompose the predicted LiDAR Gaussians into static and dynamic parts. We aggregate the dynamic Gaussians separately across multiple frames in the canonical coordinate system and then render together with static Gaussians in the world coordinate system at inference time.
Moreover, we include the visualizations of dynamic objects in the revised manuscript now, demonstrating the capability of our method to produce high-quality renderings even in scenes with many dynamic objects. In future work, we will extend our model to predict the perception outputs directly, so that the static–dynamic decomposition no longer relies on annotations.
Q4. Efficiency protocol clarification: How many input and query frames are included in the timings in Tables 2 and 3? What inference hardware and GPU counts are used? Do the reported times include encoding, diffusion sampling, Gaussian decoding, and final rendering? These details are necessary to interpret the claimed speedups.
Response: We agree that a more detailed description of the efficiency protocol is necessary for reproducibility. Thus, we clarify the experimental setup as follows:
- Hardware: All LiFTeR inference is run on a single NVIDIA H20 GPU.
- Tab. 2: One in every eight frames is held out for testing. At each test timestamp, LiFTeR (C→L) takes images from all six cameras as input.
- Tab. 3: We use every other frame as input and render each intervening frame from its two neighbors.
- Reported time: The reported times exclude model loading only. All remaining steps are included, i.e., VAE encoding, diffusion model denoising, Gaussian decoding, and final rendering.
We provide more detailed implementation details in the revised manuscript to facilitate reproducibility.
Summary:
We update the manuscript to:
- Include the controlled VAE-versus-Gaussian decoding comparison and ray-anchoring ablation (W2&Q1).
- Expand novel-view evaluation to multiple scenes and viewpoint changes (W1&Q2).
- Clarify multi-frame aggregation and dynamic-object limitations (Q3).
- Provide reproducible evaluation and runtime details (Q4).
We thank the reviewer again for the valuable suggestions, which help us strengthen the evidence supporting LiFTeR’s contributions.
Thank you for the thoughtful responses and the effort put into the additional experiments and revisions.
I appreciate the VAE-versus-Gaussian decoder comparison, the ray-anchoring ablation, the expanded novel-view evaluation, and the clarifications of the multi-frame and runtime protocols. These additions have addressed most of my concerns and strengthened the empirical support for the proposed framework. I will maintain my positive assessment.
Response to Reviewer nQpg (1/2)
We sincerely thank the reviewer for the careful assessment and constructive feedback. We appreciate the recognition of LiFTeR’s unified framework for camera-conditioned LiDAR generation and feed-forward reconstruction, as well as its generation quality and computational efficiency.
We address the reviewer’s concerns regarding (1) the contribution of the Gaussian decoder, (2) the benefits of an explicit 3D representation, (3) multi-frame consistency, and (4) computational efficiency below.
W2&Q1. Contribution attribution / additional validation: What are the MMD/JSD and paired geometric errors when the same generated latents are decoded using the original LiDAR VAE decoder versus the Gaussian decoder? How does performance change when ray anchoring is removed while other settings are retained? These comparisons would directly affect my assessment of the proposed design.
Response: We thank the reviewer for highlighting the importance of isolating the contributions of the Gaussian decoder and ray anchoring. We have added the following comparisons and clarified the purposes of Tabs. 1–3 in the revised manuscript.
- Comparison between the VAE and Gaussian decoder. We have revised the descriptions of all three tables to provide more detailed explanations of their respective purposes and the LiFTeR modules they evaluate. In Tab. 1, the LiDAR scans are decoded by the VAE to evaluate the performance of the diffusion model. Tab. 2 and Tab. 3 aim to evaluate the Gaussian decoder. Under the evaluation protocol of Tab.2, the results obtained with the VAE decoder are reported below:
\hline \text{Decoder} & \text{CD}\downarrow & \text{F-score}\uparrow & \text{Depth RMSE}\downarrow & \text{Depth PSNR}\uparrow & \text{Intensity RMSE}\downarrow & \text{Intensity PSNR}\uparrow \\\\ \hline \text{VAE decoder} & \text{1.6131} & \text{0.6009} & \text{7.2756} & \text{17.0658}& \text{0.0413} & \text{24.7617} \\\\ \text{Gaussian decoder} & \text{1.5955} & \text{0.6118} & \text{7.0761} & \text{17.2053} & \text{0.0449} & \text{24.7396} \\\\ \hline \end{array}$$ 2. **Ray-anchoring ablation.** In LiFTeR, the ray direction of each Gaussian is fixed by the sensor configuration, so its position is described by a single depth along the ray. In the ablation, we replace this depth with a free xyz position and retrain the decoder with all other settings unchanged. The results are shown below.\begin{array}{lcccccc} \hline \text{Method} & \text{CD}\downarrow & \text{F-score}\uparrow & \text{Depth MedAE}\downarrow & \text{Depth PSNR}\uparrow & \text{Intensity MedAE}\downarrow & \text{Intensity PSNR}\uparrow \\ \hline \text{Full model} & \text{1.5955} & \text{0.6118} & \text{0.1263} & \text{17.2053} & \text{0.0073} & \text{24.7396} \\ \text{w/o ray anchoring} & \text{14.4976} & \text{0.3992} & \text{1.0949} & \text{13.1531} & \text{0.0689} & \text{19.2761} \\ \hline \end{array}
Withoutfixingtheraydirection,thedecoderfailstolearnvalidgeometry.CDincreasesfrom1.59to14.49,whichindicatesthatthepredictedscenegeometryisentirelyincorrect.Wethankthereviewerforsuggestingtheray−anchoringablation.ThisablationisausefulcontrolandhelpsreadersbetterunderstandwhyLiFTeRadoptsray−anchoringinsteadofpredictingafree3Dposition.Wehaveincludedtheresultsintheablationstudyoftherevisedmanuscript.