- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Designing and presenting an open source sensor platform (with multiple different sensors) and releasing collected data with that platform mounted on a car. Also discusses design decisions, engineering efforts, and lessons learned for the sensor platform. The paper further propose a MAE-based self-supervised pre-training setup for different (e.g. depth, flow, and semantic segmentation) downstream robotics perception tasks. The paper also compares the proposed pre-training against general vision foundation models such as DINO-family, SigLIP2, Perception Encoder, and V-JEPA.
Critical strengths
I really like the idea of having a “crowd-sourced” dataset with a shared sensor platform that the community can re-build, develop, and collect (and share) additional data with (potentially for multiple different robotic embodiments). (Which I interpret is the idea, but I think it needs to be clarified in the manuscript). The detailed engineering efforts and lessons learned are useful for the research field and is something that most (industry) datasets take for granted and don't discuss in depth. I think this part of the work is thoroughly done.
--
The paper focuses on an important aspect of pre-training for robotic perception where we have privileged data (in terms of offline processing, benefit of hindsight, and multiple sensors) to learn from. It is important with methods for efficiently fusing information from different sensors with different strengths, and pre-training these types of systems is an important topic. The paper does a good job of showcasing the strengths of multiple modalities while also discussing the difficulties with calibration, synchronization etc.
Critical weaknesses
Unclear usefulness and over-claimed generality of the dataset:
While I understand the complications, claiming the sensor setup is for “robot perception” in general when it is “only” equipped on a car feels a bit of a stretch. It would be nice (and make the dataset more unique?) with at least some proof-of-concept that this setup can be equipped on another robot embodiment. The choice of sensors is also not so clearly motivated, and the usefulness of e.g. the main selling point(?) sensor event-camera seems of limited value in the paper’s own evaluation (“The fact that we do not see event cameras and IMU contributing to the performance much…”). I think the choice of sensors needs to be further motivated (or the sensor setup simplified) and since it is mostly autonomous driving (AD) data in its current form, further comparisons and discussions with existing AD datasets would strengthen the paper. (What can we do with this dataset that existing AD datasets can’t already provide?). I think the same goes for the chosen downstream tasks: why are these selected? To me, for instance, more interesting/useful tasks are e.g. 3D object detection, 3D semantic occupancy prediction, or trajectory planning. Furthermore are the provided “ground truth” more of pseudo-labels from other methods and their accuracy is not discussed, limiting the understanding on what they actually provide for the user.
--
Limited related work and benchmarking:
The related work and benchmarks focus on “vision foundation models” (VFM), but there are also work focusing pre-training on autonomous driving/robotics data, including supervising cross sensor-modalities that would make the positioning and comparison of this paper’s method more clear (e.g. VIDAR, OccFeat, GASP,...). There are also e.g. lidar foundation models (e.g. SAL, Utonia) one could compare/position the method against. There are also other VFMs such as Depth Anything, Metric3D, VGGT with a heavier focus on geometry-training that would be interesting to compare the chosen tasks against. There is also in my opinion missing experiments comparing how well the model performs on the chosen tasks without any pre-training. Together the baselines in this method don't feel fair, and it is not clear why the proposed MAE-style of pre-training is better than other approaches. [Note that I don't mean that all the mentioned need to be compared against] I furthermore don’t follow the reasoning on R403: one could rather argue it is a drawback of the pre-training method that it doesn’t work better than a general model trained on very different data. It further raises the question which model works best on a more different dataset, e.g. nuscenes?
Questions for authors
Is it possible to provide further baseline benchmarks and comparison to non-pretrained methods in order to investigate the effects of the proposed pre-training method?
Can you further motivate the choice of sensors and downstream evaluation tasks?
How accurate are the provided pseudo-labels? (Could be on a smaller subset of the data)
“This first release of OctoSense data “ alludes to more releases, what will that include? Other embodiments?
Are you planning on hosting a server on which other people can upload their data?