acceptodds
Withdrawn from ICLR 2027

LiDAR-Anchored Splat-and-Distill: Sensor-Grounded 3D-Aware Distillation for Vision Foundation Encoders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision foundation encoders capture strong 2D semantics but lack the explicit 3D geometry essential for autonomous vehicle (AV) applications and embodied perception. While recent feed-forward "splat-and-distill" approaches lift camera pairs into Gaussian feature fields to distill 3D structure, they rely on inferred, scale-ambiguous geometry from frozen stereo networks, built for short-baseline, high-overlap scenes. In contrast, standard AVs are equipped with synchronized camera-and-LiDAR rigs that provide metric geometry directly. We therefore take a geometry-first approach and introduce LiDAR-Anchored Splat-and-Distill, a framework leveraging this native sensor setup to distill 3D-aware features from measured rather than inferred geometry. Adapting feed-forward distillation to AV environments raises two challenges. First, the views of a surround rig differ substantially in viewpoint, so geometry cannot be inferred from them; anchoring 3D Gaussian feature fields directly to LiDAR surfels removes any need for geometry estimation and lets distillation operate across arbitrary viewpoint changes. Second, LiDAR is sparse, confined to a narrow horizontal band, and subject to occlusion, so the lifted-and-rendered features leave most of every target view without a distillation target; a coverage-masked semantic-anchor loss preserves feature quality in those regions. An auxiliary metric-depth head supervised by the same LiDAR complements the distillation objective. We train on in-house multi-camera-and-LiDAR driving logs and evaluate zero-shot on KITTI, which uses a different sensor rig, across two backbones, DINOv2-Base and C-RADIOv4-Huge. Our method improves depth estimation and cross-view correspondence over the off-the-shelf encoders while also improving 2D semantic segmentation. Against the public SnD, FiT3D, and MEF checkpoints under the same protocol, it gives the largest depth gain and is the strongest on KITTI-pairs, a rotation-stratified driving correspondence benchmark we introduce. Our resulting encoder requires no LiDAR and adds no computational overhead at test time: measured geometry serves only as training-time scaffolding for a general-purpose 2D encoder.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D-aware distillation, vision foundation models, Gaussian splatting, LiDAR, autonomous driving, feature distillation, self-supervised representation learning, monocular depth estimation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xdTBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LiDAR-Anchored Splat-and-Distill for adapting vision foundation encoders to autonomous-driving geometry. During training, LiDAR points are converted into Gaussian surfels, painted with frozen teacher features from context cameras, and rendered into a held-out target camera. A coverage-masked anchor loss constrains regions without rendered supervision, while an auxiliary LiDAR depth head provides metric supervision. LoRA updates are merged into the backbone, so inference remains a standard single-image forward pass without LiDAR.

Critical strengths
  1. Replacing fragile short-baseline stereo geometry with measured LiDAR is a practical extension of splat-and-distill to surround-view driving rigs, with no added inference cost.
  2. The paper clearly diagnoses sparse supervision: 76.4% of the target view receives no rendered signal, motivating a simple coverage-masked anchor loss.
Critical weaknesses
  1. The evidence for the core 3D-aware distillation claim is weak. Distillation alone degrades depth and correspondence; depth gains mainly come from direct depth supervision, while the anchor restores correspondence. Final r@10 gains are only 0.27/0.64 points, with no improvement in the largest-rotation bin.
  2. Baseline coverage is incomplete and uncontrolled. SnD, FiT3D, and MEF use mismatched training domains, while two highly related works LiDAR-Anchored Collaborative Distillation (arXiv:2602.12524) and DDMS (arXiv:2608.23850) are not discussed. The former particularly weakens the claim that prior LiDAR-to-camera distillation is limited to task-specific BEV models.
  3. The method is only evaluated on one dataset, which seems insufficient. More datasets such as nuScenes and Waymo are required.
Questions for authors
  1. I wonder what is contributed by cross-view surfel distillation, given that distillation alone degrades geometry, depth-only gives substantially better depth, and the anchor accounts for most correspondence recovery?
  2. Why does the method not improve the 10–90° rotation bin and fall below the off-the-shelf encoder beyond 5 m translation? Should the claim of handling arbitrary viewpoint changes be weakened?
  3. The ablation studies are insufficient. It's encouraged to compare controlled alternatives on the same driving corpus and backbone, including direct LiDAR-to-image feature distillation and measured-geometry SnD without the proposed surfel design.
  4. Please add discussion about the baselines mentioned above.
  5. Could you evaluate the method on other public driving datasets (e.g., waymo, nuscenes) or any downstream tasks such as detection or occupancy?
  6. How sensitive are the results to LiDAR density, vertical field of view, and the uncovered fraction? Can a controlled sparse-LiDAR experiment be provided?
Reviewer_bCpAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper adapts feed-forward splat-and-distill to autonomous driving by replacing stereo-inferred geometry with LiDAR surfels used only as training-time scaffolding. It adds a coverage-masked anchor loss, an auxiliary depth head, and rank-8 LoRA, and evaluates DINOv2-Base and C-RADIOv4-Huge on KITTI depth, segmentation, and a new KITTI-pairs correspondence benchmark.

Critical strengths
  1. Figure 1 shows that stereo-based SnD fails on surround-view rigs, and using LiDAR only during training is a sensible remedy that adds no test-time cost.
  2. The coverage analysis (76.4% of each target view unsupervised, Appendix A.5) motivates the masked anchor loss, and the ablations and negative results (Tables 2–5, Appendix A.1) are transparent.
Critical weaknesses
  1. The claims overstate the results. Distillation alone degrades depth and correspondence on both backbones (Table 2), the depth head alone gives better depth than the full method (0.076 vs. 0.088 on DINOv2; 0.092 vs. 0.119 on C-RADIOv4, Table 3), and adding the depth head to +anchor actually reduces correspondence on C-RADIOv4 (r@10: from 73.92 to 73.09, Table 2). Overall r@10 gains over the off-the-shelf encoders are only +0.27 and +0.64, with no gain at 10–90° rotation (47.81 vs. 47.81 on DINOv2; 45.76 to 45.10 on C-RADIOv4, Table 4) and no variance reported.
  2. The evaluation does not match the motivation. The method targets cross-camera distillation on surround-view rigs, but KITTI-pairs only tests same-camera temporal pairs, public baselines in Table 1 are trained on indoor/object data without in-domain retraining, and no surround-view dataset (e.g., nuScenes) or downstream 3D task (e.g., BEV detection) is evaluated.
  3. The segmentation protocol is unclear. Section 4.2 reports results on the 200-image KITTI test set, whose ground-truth labels are not public; please clarify whether the numbers come from the 20 held-out images or the benchmark server. Moreover, LoRA rank 8 is selected based on the same mIoU reported in the tables (Section 4.4.3), with no separate validation split.
  4. Reproducibility is severely limited. Training data, code, and checkpoints are not released, and the KITTI-pairs random seed and rotation-balancing rule are omitted from Appendix A.6, so the proposed benchmark cannot be exactly reconstructed from the paper.
Questions for authors

Considering the limitations above, a rejection is proposed.

Reviewer_UBT1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper uses lidar points to finetune an image encoder for better 3D representation in its features. The idea is to use lidar points to create surfels, and project the feature maps from multiple context views onto them from the frozen encoder. Then by rendering the surfels in a target view, supervise the same encoder adapted with LoRA with the target view 2d feature map and the rendered feature map. The target view features are also supervised on the depth prediction with a DPT head. For the pixels without reprojected features, the supervision is done on the target view features from the original frozen encoder to avoid drift that decreases performance. The model is trained on in-house driving data and evaluated on KITTI dataset on depth, correspondence and segmentation.

Critical strengths

None

Critical weaknesses
  1. the paper's motivation is large viewpoint change on surround-view rigs, yet the paper only evaluates on KITTI, which has no surround rig. The rotation between the KITTI-pair is low (median 1.5deg). On top of that, in the higher rotation bin (10-90deg) there is no gain. The paper's evaluation at the end did not prove the central claim of robustness to large viewpoint changes.
  2. Novelty is limited. There are papers related or similar (LiDAR-Anchored Collaborative Distillation RA-L 2026, ConDense ECCV 2024) that are not cited. No variance is reported, considering how low the gains are in some cases.
Questions for authors
  • Would it be possible to have results on a surround rig (nuScenes, Waymo, Pandaset) ?
  • Where are the KITTI semantic test labels from?
  • Why train on proprietary data when many opensource datasets are available, as it would improve reproducibility?
  • Seeds or confidence intervals for correspondence and mIoU?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.