acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Representation Learning from Lidar Images

Abstract

In real-world applications, such as autonomous driving, the inherent complexity of the environment poses high requirements on the quality of the representations learned by perception models. While semantic understanding is important, it is usually not present natively in LiDAR data. To solve this, for the first time, we investigate self-supervised learning for LiDAR range-view representation with the hypothesis that distillation from VFMs fed with LiDAR images can be beneficial. The motivation is three-fold: datasets with annotated LiDAR data are rather scarce since they are costly and time-consuming to obtain, the rapid development of LiDAR technology makes existing datasets obsolete quickly, and perception solutions face the challenges of representation learning for complex environments in real world applications of autonomous driving. We pose two research questions: Can the semantic knowledge learned by VFMs from natural RGB images be transferred to LiDAR representations even when no RGB/camera data are available during the LiDAR pretraining process? And if so, what knowledge is actually transferred across this large modality gap? After full fine-tuning, SSL initialization matches the downstream semantic segmentation score of an ImageNet-21k-initialized supervised baseline with the same architecture, and VFM distillation yields substantially higher linear-probe scores than MAE. Our results reveal potential in LiDAR image-based self-supervised representation learning for unlabeled long-range, high-resolution sensors, paving the way for scalable learning for modern LiDAR systems. Our code will be publicly released on GitHub after the paper is accepted for publication.

Primary area:
applications to robotics, autonomy, planning
Keywords:
self-supervised learning, lidar, visual foundation model, masked autoencoding, semantic segmentation, autonomous driving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jQsFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates LiDAR SSL that does not rely on camera images, camera-LiDAR calibration, or cross-modal correspondences, learning semantic representations of LiDAR range-view images by DINOv3 feature distillation.

Critical strengths

This paper proposes a LiDAR self-supervised representation learning scheme that does not rely on camera data.

The method directly feeds LiDAR range-view images into a pretrained VFM and supervises the LiDAR encoder through feature distillation, requiring neither camera images nor camera-LiDAR calibration or cross-modal correspondences. This design reduces dependence on multimodal data acquisition conditions and has practical value in application scenarios lacking annotations or other sensor data.

Critical weaknesses

There is an inconsistency between the method design and the experimental results. The authors present VFM feature distillation and MAE reconstruction as core components of a joint framework, yet table 1 show that distillation alone already achieves the best results and that adding MAE brings about negative effects. Therefore, the paper has not sufficiently demonstrated the necessity of the joint framework or the independent value of MAE, and the overall narrative lacks consistency with the experimental findings.

Questions for authors

The evaluated datasets primarily feature structured spinning-LiDAR range views, where scan lines provide a relatively regular spatial organization. However, solid-state and semi-solid-state LiDARs can exhibit substantially different and potentially irregular sampling patterns, making their conversion into suitable range-view representations more challenging. It remains unclear whether the proposed image-based distillation framework can learn effective representations under such conditions, limiting the evidence for its applicability to diverse modern LiDAR sensors.

7 comments

Dear reviewer, Thank you for your feedback. Although we thought that the MAE results would be valuable knowledge to publish; as the first work, to the best of our knowledge, that attempted MAE on LiDAR images; the authors agree with your reported critical weakness about MAE, which in fact has also been pointed out by the other reviewers. Therefore, instead of making MAE a central part of our contribution, in the next paper revision we will treat the MAE branch as an add-on, but deactivate it in the main experiments, which will become more focused mostly on stand-alone VFM distillation instead. We still find value in publishing the MAE and VFM+MAE scores, although as a side-experiment instead of centrally. While the spinning-LiDAR data in the Waymo and Ouster datasets are natively structured as images, data in SemanticKITTI are not, which already produces significant performance degradation when using our LiDAR image-based method. This is discussed in Section 4.1, in the paragraph where Table 5 is referenced. We agree with the reviewer that, for modern solid-state LiDARs, which typically sample the environment in rather irregular patterns, converting data to the range-view is expected to result in a substantially worse image structure, which is expected to degrade performance of our method significantly compared to the performance that one could expect for Waymo and Ouster datasets. We will add this clarification in the next revision of the paper.

AuthorsOpenReview ↗

Updated manuscript

Dear reviewer,

We have now updated the manuscript with the following key changes to address the raised weaknesses and questions. Line, table, figure, and section numbers refer to the new manuscript version.

WeaknessManuscript update
There is an inconsistency between the method design and the experimental results. The authors present VFM feature distillation and MAE reconstruction as core components of a joint framework, yet table 1 show that distillation alone already achieves the best results and that adding MAE brings about negative effects. Therefore, the paper has not sufficiently demonstrated the necessity of the joint framework or the independent value of MAE, and the overall narrative lacks consistency with the experimental findings.We agree, and we have modified the narrative, where we no longer discuss "geometric" learning from MAE, and where our main experiments on Waymo data are now VFM distillation only (see Tables 1, 2, and 3). However, we have kept in Table 4 the experiment where it is shown that MAE does not improve VFM distillation only for Waymo, and in Table 5 we have added a row where it is shown that, for SemanticKITTI, adding MAE has actually made the result 2.1 mIoU points better than with VFM distillation only. As discussed in lines 476-480, we interpret that this may be a result of SemanticKITTI's less ordered and clean intensity images, which degrades the quality of VFM features that we distill from, and thus MAE can help in improving learning where the VFM input is poorer.
QuestionManuscript update
The evaluated datasets primarily feature structured spinning-LiDAR range views, where scan lines provide a relatively regular spatial organization. However, solid-state and semi-solid-state LiDARs can exhibit substantially different and potentially irregular sampling patterns, making their conversion into suitable range-view representations more challenging. It remains unclear whether the proposed image-based distillation framework can learn effective representations under such conditions, limiting the evidence for its applicability to diverse modern LiDAR sensors.It is a good point that we had not considered before. While the LiDAR in SemanticKITTI is spinning, the actual data is provided as 3D coordinates after ego motion compensation, which as discussed extensively in prior range-view art, and as can be seen in our Figure 3, produces a distorted image when projecting it to the range-view. So, to some extent, in our work the effect that is referred to has already been present. Still, it is possible that solid-state LiDARs, which may scan the environment in an even less structured manner than what one gets from SemanticKITTI, do not see as much benefit from our method. We have added lines 522-525 in the conclusion to acknowledge this potential limitation.

We would like to thank you again for your feedback that has helped to improve the paper, and remain open to further feedback and suggestions.

Thank you for the clarifications.

I still suggest that the authors reconsider and restructure the narrative of the paper. If MAE is intended as a baseline for comparison, it should not appear in the "Framework Overview" or "Method" sections. Conversely, if MAE is presented as one of the contributions of this work, the current results fail to demonstrate its effectiveness.

Therefore, I regret that I cannot increase the score.

Reviewer_EyuQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose a self-supervised method of Lidar data representation learning, which combines both vision foundation model feature distillation branch and MAE branch to learn the representation and enhance the capability of their proposed encoder. And they adopt the pre-trained range ViT encoder to downstream tasks. Their method can work without any camera calibration which is meaningful. They validate the performance on semantic segmentation for SemanticKITTI and Waymo datasets, which shows their results are comparable with the previous state-of-the-art models.

Critical strengths
  1. The framework is very clear and their design is also very reasonable. The first stage of pre-training based on two branches of representation alignment and second stage of fine-tuning on downstream task is easy to understand and follow. And this paper provide a way which can remove the camera calibration for the distillation. This camera-free design is meaningful for knowledge transfer.
Critical weaknesses
  1. The results from Table 3 shows their method (FT-100%) only improves 0.07 mIoU over RangeViT (IN21k), under only one random seed this result can not prove the effectiveness of their proposed method. The results from Table 1 shows adding the MAE branch can make the performance even worse, which means there is no need to add the MAE loss.

  2. Their methodology design is not novel enough cause the feature distillation and masked prediction are already well established. The simple combination of these two methods and the ablation results can not prove the real advantage of their method. At least this paper should provide some evidence and analysis to show the real effect of adding both parts.

Questions for authors
  1. From Table 1, the results shows that distillation alone can perform better than combining MAE and distillation. Thus it shows there is no need to add the branch of MAE cause it makes the result even worse. So could the authors explain this results? Or could the authors show more evidence that adding MAE branch can improve the performance?

  2. This paper only includes semantic segmentation as their downstream tasks. Could the authors provide more downstream tasks to show the effectiveness of their methods such as object detection?

4 comments

Dear reviewer, Thank you for your feedback. The authors hope to address your concern regarding "the effectiveness of their proposed method", as the effectiveness will be seen for low-label training budgets in the next paper revision, as explained in our reply to Reviwer EyuQ. While we agree that "there is no need to add the MAE loss" if one looks at the final semantic segmentation downstream task score, we thought that this result might be valuable knowledge to publish; as the first work, to the best of our knowledge, that has attempted MAE on LiDAR images. Publishing this result might prevent future researchers from unnecessarily adopting MAE for LiDAR image-based semantic segmentation downstream tasks. Therefore, instead of making MAE a central part of our contribution, in the next paper revision we will treat the MAE branch as an add-on, but deactivate it in the main experiments, which will become more focused mostly on stand-alone VFM distillation instead. We still find value in publishing the MAE and VFM+MAE scores, although as a side-experiment instead of centrally. While we see your point that our "methodology design is not novel enough cause the feature distillation and masked prediction are already well established", the novelty of our work rather lies in LiDAR-only, low label budget learning rather than in the learning techniques themselves. Having this said, we must highlight that the necessary adaptations were made to the adopted techniques, such as the LiDAR-tailored input normalization explained in Section 3.1, which shows that adapting VFM distillation and MAE to LiDAR images was not trivial. Finally, to the question about additional downstream tasks, we share your view, as we mentioned in Section 5 that "Future work should evaluate broader geometric and temporal downstream tasks", which most likely should include object detection, but it might be challenging, time and compute-resource-wise, to include these in this publication.

AuthorsOpenReview ↗

Updated manuscript

Dear reviewer,

We have now updated the manuscript with the following key changes to address the raised weaknesses and questions. Line, table, figure, and section numbers refer to the new manuscript version.

WeaknessManuscript update
The results from Table 3 shows their method (FT-100%) only improves 0.07 mIoU over RangeViT (IN21k), under only one random seed this result can not prove the effectiveness of their proposed method. The results from Table 1 shows adding the MAE branch can make the performance even worse, which means there is no need to add the MAE loss.We agree. To prove the effectiveness of our proposed method, we have added Table 1, which reports experimental results on diverse training data fractions for fine-tuning. The results show clear superior performance by our SSL compared to random and ImageNet21k initializations at low label fractions, while still keeping moderately better performance than those at higher fractions. Additionally, as the MAE did not contribute and even worsened performance in the Waymo data, we have removed it from the experiments in Tables 1, 2, and 3, which now report results of VFM distillation only; and now only Table 4 evaluates MAE against VFM distillation and the combination of both on Waymo data, and one new entry in Table 5 actually shows that for the SemanticKITTI dataset the combination of VFM distillation and MAE provides 2.1 points better mIoU performance than VFM distillation alone; which indicates that MAE might still be needed for certain experimental settings and, in particular, datasets.
Their methodology design is not novel enough cause the feature distillation and masked prediction are already well established. The simple combination of these two methods and the ablation results can not prove the real advantage of their method. At least this paper should provide some evidence and analysis to show the real effect of adding both parts.We would argue that the novelty of our work rather lies in LiDAR-only, self-supervised learning aimed for low label budget downstream tasks rather than in the learning techniques themselves. Having this said, we must highlight that the necessary adaptations were made to the adopted techniques, such as the LiDAR-tailored input normalization explained in Section 3.1, which shows that adapting VFM distillation and MAE to LiDAR images was not a trivial contribution. Additionally, we hope to have addressed the concern about missing evidence and analysis with the answer above.
QuestionManuscript update
From Table 1, the results shows that distillation alone can perform better than combining MAE and distillation. Thus it shows there is no need to add the branch of MAE cause it makes the result even worse. So could the authors explain this results? Or could the authors show more evidence that adding MAE branch can improve the performance?We believe that we have answered in the "Weaknesses" table above, but to sum up, we have removed MAE from the main experiments on Waymo, where it made the result even slightly worse, but we have kept it on one experiment in SemanticKITTI, where it has made the result 2.1 mIoU points better. As discussed in lines 476-480, we interpret that this may be a result of SemanticKITTI's less ordered and clean intensity images, which degrades the quality of VFM features that we distill from, and thus MAE can help in improving learning where the VFM input is poorer.
This paper only includes semantic segmentation as their downstream tasks. Could the authors provide more downstream tasks to show the effectiveness of their methods such as object detection?As the paper has a strong focus on representation learning on a RangeViT encoder where a simple linear decoder is attached, and no focus on decoders of any type, adding an object detection task would be non-trivial, as typically this task requires some sort of complex decoding mechanism compared to a dense task like semantic segmentation where a simpler per-point prediction is expected. While we agree that it would be interesting to perform such experiments, given the conceptual complexity for them to fit in our current concept, we are afraid that they must be left for future work.

We would like to thank you again for your feedback that has helped to improve the paper, and remain open to further feedback and suggestions.

My concerns are resolved and I will raise my score.

Thank you, Reviewer EyuQ, for your insightful comments and positive reassessment of our work.

Reviewer_xeE5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates LiDAR-only self-supervised pretraining by feeding a processed LiDAR intensity image directly to a frozen DINOv3 teacher. A RangeViT student receives return, range, and intensity channels from the same scan. Resizing aligns teacher and student token grids, enabling feature distillation without paired camera images or camera-LiDAR calibration. An optional reconstruction objective is evaluated alongside distillation, followed by semantic-segmentation linear probing and fine-tuning.

Critical strengths

The data requirement is a meaningful distinction: both teacher and student inputs originate from the LiDAR scan, rather than using a separate camera during pretraining and discarding it only at inference. This offers a clear route for datasets without paired camera observations. The preprocessing and token-alignment procedure are sufficiently concrete to understand the central idea.

The feasibility result is substantial in the reported setting. On Waymo, distillation reaches 37.65 mIoU with a frozen encoder, compared with 6.92 for random features and 27.28 for the best reported MAE variant. Full fine-tuning reaches 55.29 versus 55.22 for the ImageNet-21k-initialized baseline under the stated common architecture. The conclusion appropriately acknowledges that combining MAE with distillation does not improve the segmentation result. These findings support a useful alternative pretraining route, rather than a claim of segmentation state of the art.

Critical weaknesses

First, the label-efficiency claim lacks the decisive comparison. Table 3 contains low-label results only for the proposed method; the ImageNet-21k baseline uses all labels. The 50% result being close to a full-label baseline does not show an advantage over ImageNet initialization or training from scratch at the same 50% budget. Linear probing also uses all labels and is not a low-label control. The most informative experiment is a matched-subset, matched-update comparison of random initialization, ImageNet initialization, and the proposed SSL at several label fractions.

Second, the motivating question of what knowledge is transferred is not identified by the current experiments. Table 2 compares two initializations that both undergo the proposed SSL; it is not a direct ImageNet-initialized student baseline without this pretraining. A linear probe on the teacher's intensity features, direct ImageNet-student probing, and focused preprocessing or teacher-capacity controls would distinguish teacher capability, input design, and student distillation. The existing numbers establish useful transferred information, but not which information or why this particular route is preferable.

Third, transferability and scalability are not cleanly established. The source checkpoint and target-unlabeled-data access are not clearly tabulated for SemanticKITTI and Ouster. Running or pretraining on several datasets is different from fixed-source cross-sensor transfer. Ouster evidence is mainly qualitative, and some visualizations use an RGB-teacher-input variant rather than the main intensity-only setup. This should be separated from the central claim, without inferring that the main setup improperly uses cameras.

Fourth, Table 8 requires verification: the unmasked and masked distillation rows have identical reported values for all 22 classes, as well as the same mIoU. This is an observable reporting issue whose cause is unknown. Full-precision metrics and run provenance would clarify whether the rows represent independent evaluations. It is not evidence of misconduct.

Finally, some interpretations exceed their controls. The combined objective scores 37.52, below distillation alone, so additional geometric benefits remain unmeasured. The 55.29 versus 55.22 difference is only 0.07 with a single reported seed and should be treated as matching rather than a reliable improvement. The projection-loss analysis in A.6 also includes the choice to leave some colliding points without inference labels; its 2D/3D score gap is not a controlled estimate of an unavoidable sensor penalty. Table 4 and Table 9 use test and validation splits respectively, so their different values are not themselves a contradiction.

Questions for authors
  • Can you provide same-label-budget random/ImageNet/SSL curves, and teacher versus direct-ImageNet linear probes?
  • Which source checkpoints and target data produce each transfer result?
  • Please verify the repeated Table 8 rows and specify collision-point prediction rules.
  • Teacher feature-extraction/cache cost and exact low-label subsets would also clarify reproducibility and scalability.
3 comments
AuthorsOpenReview ↗

Thank you and ongoing experiments towards next paper revision

Dear reviewer, The authors truly appreciate your sound feedback. We agree that the label-efficiency claim lacks the decisive comparison; therefore, we have been running additional experiments to address this concern. In the few preliminary results that we got, it became clear that our approach helps with label efficiency. For example, fine-tuning for semantic segmentation with 10% training data budget yields approximately 45% mIoU with our pre-training, compared to 33% mIoU and 35% for the random encoder and ImageNet21k weight initialization, respectively; showing that our pre-training helps in low-label regimes. These and more experiments are being conducted, including 1%, 5%, 10%, 25%, and 50% training data budgets, with 5 different random subsets for each of them; and the final results as well as the exact subsets will be included in the paper's next revision. In parallel, we are also running the suggested DINOv3 linear probing experiments to understand how much of its knowledge transfers to our student through distillation, as well as ImageNet21k-student linear probing; these will be included in the next revision, too. Additionally, we are re-running the 2 mentioned experiments from Table 8 with additional cautionary measures to ensure preventing any overwrites in the final results as well as collecting more detailed evidence that may help to determine why the results were identical if they turned out to be so again. Regarding SemanticKITTI and Ouster experiments, in both cases the model was pre-trained and fine-tuned on their own data; to be clear, no cross-dataset transfer was attempted in our work at all. We will clarify this in the paper and will mention which exact checkpoints and data were used for each experiment. Additionally, we will report teacher feature-extraction/cache cost in the next revision of the paper. Finally, we agree that additional geometric benefits of MAE remain unmeasured. Due to this reason, which has been raised as an even higher concern by the other reviewers as well, we will treat the MAE branch as an add-on, but deactivate it in the main experiments, which will become more focused mostly on stand-alone VFM distillation instead.

AuthorsOpenReview ↗

Updated manuscript

Dear reviewer,

We have now updated the manuscript with the following key changes to address the raised weaknesses and questions. Line, table, figure, and section numbers refer to the new manuscript version.

WeaknessManuscript update
First, the label-efficiency claim lacks the decisive comparison. [...] The most informative experiment is a matched-subset, matched-update comparison of random initialization, ImageNet initialization, and the proposed SSL at several label fractions.Lines 392-413, including Table 1, discuss a new experiment which, as suggested, is a matched-subset, matched-update comparison of random initialization, ImageNet initialization, and the proposed SSL at several label fractions. The results show clear superior performance by our SSL at low label fractions; for example for 1% training budget, our SSL produces 34.4% mIoU, more than twice as much the 16.0 mIoU and 14.4 mIoU from random and ImageNet21k initialized weights, respectively, while still keeping moderately better performance at higher fractions. See Table 1 for full results, including more training data budgets.
Second, the motivating question of what knowledge is transferred is not identified by the current experiments. [...] A linear probe on the teacher's intensity features [...]A linear probe on the teacher's intensity features is added at Table 3 and the experiment is explained and results interpreted in lines 424-438. The results show that not only is the knowledge from DINOv3 fed with intensity transferred to the RangeViT student, but that the RangeViT student fed with all channels slightly exceeds its performance.
Third, transferability and scalability are not cleanly established.By transferability, we referred to transfer of VFM knowledge, which was acquired training on RGB images, into the LiDAR domain. Regarding SemanticKITTI and Ouster experiments, in both cases the model was pre-trained and fine-tuned on their own data; to be clear, no cross-dataset transfer was attempted in our work at all. We hope that transferability is now better addressed with the evidence from Table 3.
Fourth, Table 8 requires verification: the unmasked and masked distillation rows have identical reported values for all 22 classes, as well as the same mIoU.After investigating, the most likely cause was that a checkpoint file with identical name to the other one was overwritten at the same time that another process exported it for evaluation. We fixed this vulnerability in our code and repeated the experiment, which now shows similar, but slightly different, results; see Appendix A.9 including Table 15.
The combined objective scores 37.52, below distillation alone, so additional geometric benefits remain unmeasured.We agree and therefore we have removed the phrasing "additional geometric benefits" throughout the text, as these were never measured in our work. Additionally, we have switched the main focus towards VFM distillation and we have left MAE a less central role, as can be seen mainly in Section 4 but also throughout all other sections.
Table 4 and Table 9 use test and validation splits respectively, so their different values are not themselves a contradiction.We agree and we have removed the mentioned cross-table comparison from the text.
QuestionManuscript update
Can you provide same-label-budget random/ImageNet/SSL curves, and teacher versus direct-ImageNet linear probes?Yes, the requested data can now be found in Table 1 and Table 3.
Which source checkpoints and target data produce each transfer result?We hope it is now clearer in the improved Section 4 that all experiments are single-dataset, i.e. no dataset transfer has been attempted, and that our SSL weights are produced starting from randomly initialized student weights and distilling from DINOv3 and/or applying MAE depending on the experiment, using the input data as specified in Section 3.
Please verify the repeated Table 8 rows and specify collision-point prediction rules.Done. As explained above, the repeated values were likely due to a bug in exporting checkpoints, and new experiments show slight differences and can be found in Appendix A.9 and Table 15.
Teacher feature-extraction/cache cost and exact low-label subsets would also clarify reproducibility and scalability.We fully agree, we have now reported the requested teacher costs in Appendix A.5 and we have thoroughly reported exact low-label subsets and results for each subset and seed in Appendices A.6 and A.7, including Tables 9 to 13 and Figures 7 to 11.

We would like to thank you again for your high quality feedback that has greatly helped to improve the paper, and remain open to further feedback and suggestions.

Thanks for your rebuttal. However, I do not think so many changes on the manuscript will make this paper qualified paper for acceptance. In addition, I also agree with reviewer jQsF's comments. I would like to keep my original weak reject ratings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.