acceptodds
Under review as a conference paper at ICLR 2027

OctoSense: Self-Supervised Learning for Multimodal Robot Perception

Abstract

We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a “late-fusion” masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. Links to the dataset, code, and supplementary videos will be provided with the camera-ready version.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multimodal perception, self-supervised learning, sensor fusion, and robot vision

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_8ZZ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Designing and presenting an open source sensor platform (with multiple different sensors) and releasing collected data with that platform mounted on a car. Also discusses design decisions, engineering efforts, and lessons learned for the sensor platform. The paper further propose a MAE-based self-supervised pre-training setup for different (e.g. depth, flow, and semantic segmentation) downstream robotics perception tasks. The paper also compares the proposed pre-training against general vision foundation models such as DINO-family, SigLIP2, Perception Encoder, and V-JEPA.

Critical strengths

I really like the idea of having a “crowd-sourced” dataset with a shared sensor platform that the community can re-build, develop, and collect (and share) additional data with (potentially for multiple different robotic embodiments). (Which I interpret is the idea, but I think it needs to be clarified in the manuscript). The detailed engineering efforts and lessons learned are useful for the research field and is something that most (industry) datasets take for granted and don't discuss in depth. I think this part of the work is thoroughly done.

--

The paper focuses on an important aspect of pre-training for robotic perception where we have privileged data (in terms of offline processing, benefit of hindsight, and multiple sensors) to learn from. It is important with methods for efficiently fusing information from different sensors with different strengths, and pre-training these types of systems is an important topic. The paper does a good job of showcasing the strengths of multiple modalities while also discussing the difficulties with calibration, synchronization etc.

Critical weaknesses

Unclear usefulness and over-claimed generality of the dataset:

While I understand the complications, claiming the sensor setup is for “robot perception” in general when it is “only” equipped on a car feels a bit of a stretch. It would be nice (and make the dataset more unique?) with at least some proof-of-concept that this setup can be equipped on another robot embodiment. The choice of sensors is also not so clearly motivated, and the usefulness of e.g. the main selling point(?) sensor event-camera seems of limited value in the paper’s own evaluation (“The fact that we do not see event cameras and IMU contributing to the performance much…”). I think the choice of sensors needs to be further motivated (or the sensor setup simplified) and since it is mostly autonomous driving (AD) data in its current form, further comparisons and discussions with existing AD datasets would strengthen the paper. (What can we do with this dataset that existing AD datasets can’t already provide?). I think the same goes for the chosen downstream tasks: why are these selected? To me, for instance, more interesting/useful tasks are e.g. 3D object detection, 3D semantic occupancy prediction, or trajectory planning. Furthermore are the provided “ground truth” more of pseudo-labels from other methods and their accuracy is not discussed, limiting the understanding on what they actually provide for the user.

--

Limited related work and benchmarking:

The related work and benchmarks focus on “vision foundation models” (VFM), but there are also work focusing pre-training on autonomous driving/robotics data, including supervising cross sensor-modalities that would make the positioning and comparison of this paper’s method more clear (e.g. VIDAR, OccFeat, GASP,...). There are also e.g. lidar foundation models (e.g. SAL, Utonia) one could compare/position the method against. There are also other VFMs such as Depth Anything, Metric3D, VGGT with a heavier focus on geometry-training that would be interesting to compare the chosen tasks against. There is also in my opinion missing experiments comparing how well the model performs on the chosen tasks without any pre-training. Together the baselines in this method don't feel fair, and it is not clear why the proposed MAE-style of pre-training is better than other approaches. [Note that I don't mean that all the mentioned need to be compared against] I furthermore don’t follow the reasoning on R403: one could rather argue it is a drawback of the pre-training method that it doesn’t work better than a general model trained on very different data. It further raises the question which model works best on a more different dataset, e.g. nuscenes?

Questions for authors

Is it possible to provide further baseline benchmarks and comparison to non-pretrained methods in order to investigate the effects of the proposed pre-training method?

Can you further motivate the choice of sensors and downstream evaluation tasks?

How accurate are the provided pseudo-labels? (Could be on a smaller subset of the data)

“This first release of OctoSense data “ alludes to more releases, what will that include? Other embodiments?

Are you planning on hosting a server on which other people can upload their data?

Reviewer_enWmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OctoSense, a large-scale multimodal dataset comprising 59 hours of synchronized data collected from various sensors, including stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit (IMU), GPS, and vehicle proprioception (CAN bus). The dataset covers diverse driving environments, including urban, suburban, rural, and highway settings, under varying lighting and environmental conditions. Building on this dataset, the authors propose a multimodal Masked Autoencoder (MAE) architecture combining modality-specific tokenizers with a late-fusion strategy, supported by extensive ablation studies. Pretraining on OctoSense followed by downstream task probing demonstrates substantial improvements over state-of-the-art self-supervised learning baselines, such as DINOv2 and V-JEPA, across several perception tasks. The authors also demonstrate promising transfer performance on the M3ED dataset, highlighting the potential of the learned multimodal representations.

Critical strengths

This paper presents a timely and valuable contribution. The availability of a large-scale multimodal dataset covering diverse sensors and environmental conditions is an important step toward developing more generalizable world models. I believe the OctoSense dataset will be a valuable resource for the research community. The results in Table 1, demonstrating improvements over state-of-the-art self-supervised learning methods on several downstream tasks, are promising, as are the transfer results reported in Table 4. Furthermore, the extensive ablation studies provide convincing empirical support for the proposed MAE architectural design choices.

Critical weaknesses

1. Limited generalization across sensor configurations:

The proposed architecture relies on modality-specific tokenizers and a particular sensor configuration, which may limit its applicability to other robotic platforms. Although the authors evaluate transfer to M3ED, it is unclear how well the pretrained representations would generalize to substantially different sensor specifications, resolutions, or placements.

2. Potential bias in semantic segmentation evaluation:

The semantic segmentation evaluation relies on pseudo-labels generated by EoMT trained on Cityscapes. This introduces potential biases. An evaluation on a small manually annotated test set would provide stronger evidence of the reported performance.

3. Limited evaluation of optical flow on dynamic objects:

The optical-flow ground truth is derived from ego-motion and LiDAR measurements, with independently moving objects excluded from the point-cloud accumulation process. This limits the conclusions that can be drawn about performance in dynamic scenarios involving moving vehicles or pedestrians.

4. Limited cross-dataset generalization:

While the proposed method substantially outperforms V-JEPA 2.1 on OctoSense, its performance on ego-motion estimation is noticeably worse when transferring to M3ED (Table 4). This raises concerns about the generalizability of the learned multimodal representations, particularly across different sensor configurations, calibrations, and driving environments.

5. Potential unfairness in baseline comparisons:

The proposed model is pretrained directly on OctoSense, whereas existing foundation-model baselines are pretrained on external datasets. Although the authors include RGB-only baselines trained on OctoSense, it remains difficult to disentangle the benefits of multimodal fusion from those of in-domain pretraining. Adapting existing foundation models to OctoSense through additional self-supervised pretraining would provide a more controlled comparison.

Since there is no borderline rating available, I am starting with a Weak Reject. However, I am willing to upgrade my recommendation if the authors can address the questions raised below during the rebuttal, especially question 1 and 5.

Questions for authors
  1. Given that the proposed architecture relies on modality-specific tokenizers and a particular sensor configuration, how easily can a pretrained OctoSense model be transferred to robotic platforms with different sensor specifications, resolutions, or placements?

  2. How sensitive are the semantic segmentation results to the choice of pseudo-labeler, particularly since the evaluation uses labels generated by EoMT trained on Cityscapes? ? Have the authors considered using a small manually annotated test set to validate the segmentation results independently?

  3. How would the proposed method perform on independently moving objects, such as pedestrians and vehicles? An evaluation on a dataset with full optical-flow annotations would help establish whether the reported improvements generalize to more challenging dynamic scenarios.

  4. In Table 4, the proposed method performs worse than V-JEPA 2.1 on ego-motion estimation when transferring to M3ED, despite substantially outperforming it on OctoSense. How do the authors explain this performance gap? Could it be related to differences in sensor configurations, calibration, or the distribution of driving scenarios? This seems particularly relevant given the goal of learning generalizable multimodal representations.

  5. Could the authors clarify the training protocol for the downstream probes in Table 1, particularly whether all baselines use the same training data and comparable prediction heads? Furthermore, since the proposed model is pretrained directly on OctoSense while existing foundation models are pretrained on external datasets, to what extent could the observed improvements be attributed to in-domain pretraining rather than multimodal fusion? Would adapting existing foundation models to OctoSense through additional self-supervised pretraining provide a more controlled comparison?

Reviewer_C9SpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript presents OctoSense, a multisensor platform and 59-hour driving dataset, together with a self-supervised representation-learning framework for heterogeneous robotic sensors.

The approach combines modality-specific tokenizers with a masked autoencoder that factorizes spatial, temporal, and cross-modal attention. The default model uses RGB, events, LiDAR, and IMU data, with token caching intended to support efficient streaming inference.

Experiments cover depth, optical flow, semantic segmentation, ego-motion, and steering prediction. The manuscript reports substantial in-domain gains over RGB-only representations, lower latency than early fusion, and transfer results on M3ED.

Critical strengths

The integrated platform and dataset could be valuable resources. Recording diverse synchronized sensors under realistic degradation involves substantial engineering, and the manuscript explains several practical challenges clearly.

The evaluation covers multiple tasks, nighttime conditions, degraded inputs, and missing modalities. The early-fusion control and attention/tokenizer ablations are useful, while Table 3 demonstrates a meaningful reduction in encoder latency on embedded hardware.

Critical weaknesses

The comparisons establish sensor complementarity more clearly than the benefit of the SSL design. Table 1 shows that LiDAR drives most depth and motion gains, whereas events and IMU contribute little. Beyond the early-fusion control, include an existing multimodal pretraining baseline and a supervised-from-scratch control with matched inputs. An RGB-plus-LiDAR training variant would help establish the broader sensor suite's value.

The evaluation targets need qualification. Section 3 derives depth and odometry from LiDAR processing, computes ego-motion-induced flow after removing vehicles and pedestrians, and uses EoMT segmentation pseudo-labels. These proxies do not establish accuracy on independently moving objects or human-annotated semantics. Quantify valid-pixel coverage and label reliability, and validate a representative subset against independent references where feasible. Besides, How does your SSL design differ from LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving (TPAMI 2025), which transfers image knowledge to LiDAR across datasets and sensor configurations, and LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes (CVPR 2025), which combines range, voxel, and point representations? Can matched-input comparisons, including supervised-from-scratch and RGB-plus-LiDAR controls, isolate gains from the learning method rather than additional sensors?

Generalization evidence is mixed. In Table 4, depth improves modestly, but translation and rotation errors exceed V-JEPA 2.1's (0.36 versus 0.22 m; 0.82 versus 0.64 degrees), while flow is tied. Analyze this reduced transfer advantage and qualify claims about broadly reusable representations. So, how reliable are the generated evaluation targets, and what fraction of pixels remains valid? Can you provide independent validation and clarify how moving-object exclusion and segmentation pseudo-labels limit the conclusions?

For streaming efficiency, specify what is cached, what is recomputed when the window shifts, and whether cached and uncached inference agree. Report timing boundaries, including sensor preprocessing and task heads, alongside the encoder-only numbers.

[Minor] There are similar literature on M3ED, including Perspective-Invariant 3D Object Detection (ICCV 2025) and 3EED: Ground Everything Everywhere in 3D (NeurIPS 2025), which study RGB-LiDAR grounding across vehicle, drone, and quadruped platforms. Although its task differs, its cross-platform setting provides relevant context: which sensor, calibration, or scene differences explain your weaker transfer advantage on M3ED, and how do they limit the generalization claims?

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_MCQTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OctoSense, a multimodal sensing platform and a dataset comprising 59 hours of synchronized driving data from RGB and event cameras, thermal imaging, LiDAR, IMU, GNSS, and vehicle proprioception. It also proposes a multimodal masked autoencoder combining modality-specific tokenizers, factorized spatial–temporal–cross-modal attention, and inference-time feature caching. The approach is evaluated on depth, optical flow, semantic segmentation, and ego-motion estimation, including nighttime and sensor-degraded conditions.

Critical strengths
  1. OctoSense provides an uncommon combination of sensing modalities at substantial scale, supported by extensive synchronization, calibration, and data-processing infrastructure. It could enable useful research on multimodal representation learning and sensor robustness.
  2. The late-fusion design substantially reduces encoder latency relative to early fusion while maintaining or improving downstream performance, making the approach relevant to resource-constrained robotic platforms.
  3. The ablations reveal a clear division of contributions: LiDAR dominates geometric tasks, RGB supports semantics, and additional modalities provide limited gains under the evaluated setup. These findings offer useful evidence about when sensor diversity translates into practical benefits.
Critical weaknesses
  1. Sensor benefits are insufficiently separated from methodological advances. Most headline comparisons pit the proposed multisensor model against RGB-only models. Although the early-fusion baseline provides a useful architectural control, matched-input comparisons with existing multimodal SSL methods, dedicated LiDAR-only/RGB–LiDAR models, and controls isolating the contribution of pretraining are needed.
  2. Limited demonstrated methodological novelty. The approach largely adapts established masked modeling, tokenization, and attention factorization techniques. Its engineering contributions are credible, but the experiments do not yet establish a distinctive, broadly applicable learning principle or the necessity of several proposed components.
  3. Transfer results provide limited evidence of general superiority. On M3ED, the method improves depth but approximately matches V-JEPA 2.1 on flow and underperforms it on translation and rotation. Thus, the strong in-domain gains do not establish a consistently more transferable robotic representation.
Questions for authors

Please see the critical weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.