- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SpikeOcc, a spiking neural network framework for resilient multimodal 3D occupancy prediction under missing-sensor conditions. The method combines a Spiking Mixture-of-Experts (SMoE) architecture with Sensor Availability Learning (SAL), which progressively introduces sensor failures during training. Experiments on OpenOccupancy and SemanticKITTI demonstrate competitive performance under complete sensing and substantial improvements under sensor outages. The paper also investigates expert specialization, curriculum learning, and computational efficiency.
Critical strengths
-
The paper addresses a practically important problem in autonomous driving, namely maintaining reliable 3D occupancy prediction under missing-sensor conditions.
-
The experiments cover two standard benchmarks and demonstrate substantial robustness improvements. The authors also provide relatively comprehensive ablations, including parameter-matched controls, alternative training schedules, and SAL-equipped ANN baselines.
Critical weaknesses
W1. Unclear necessity and methodological novelty of spiking computation.
The motivation for adopting SNNs is not sufficiently justified. The proposed robustness largely relies on availability-aware routing, expert specialization, and sensor-availability training, which can also be implemented using conventional ANNs. In fact, SAL alone substantially improves ANN baselines, while the theoretical robustness argument based on spike thresholds is also applicable to hard-quantized representations. Although the authors provide some ANN-based comparisons, stronger controls using lightweight gated fusion and quantized ANN architectures are missing. It remains unclear what unique advantages SNNs offer beyond existing robust multimodal learning techniques.
W2. Limited practical applicability and insufficient experimental validation.
Despite targeting reliable and energy-efficient autonomous driving, SpikeOcc reports approximately 1.8 seconds of inference latency, while the claimed energy savings are theoretical estimates limited to the fusion module, excluding conventional camera and LiDAR encoders. Moreover, the evaluation assumes that sensor failures are explicitly known, whereas real-world failures often involve corrupted or unreliable observations rather than complete sensor outages. The ablations also do not sufficiently isolate several key architectural choices, including routing supervision, bounded residuals, and expert width. These limitations make it difficult to assess whether the proposed framework offers meaningful practical advantages over simpler alternatives.
Questions for authors
-
What are the fundamental advantages of spiking computation over conventional ANN-based gated fusion or mixture-of-experts architectures? Can the authors provide comparisons with architecture-matched quantized ANN baselines under identical training settings?
-
Given that SAL alone substantially improves conventional ANN baselines, how much of the robustness improvement can be specifically attributed to spiking dynamics rather than expert specialization and sensor-availability training?
-
Can the authors provide more fine-grained ablations isolating the contributions of routing supervision, bounded residual connections, and full-width unimodal experts? Would a simpler availability-aware gating mechanism achieve comparable performance?
-
Given the reported inference latency of approximately 1.8 seconds, how practical is the proposed framework for real-time autonomous driving? Can the authors provide end-to-end latency and energy measurements, including camera and LiDAR encoders, rather than theoretical fusion-only estimates?
-
How well does SpikeOcc handle realistic sensor failures, including corrupted observations, calibration errors, and uncertain sensor availability? Does the reported robustness persist when sensor failures are not explicitly provided to the model?
5 comments
Response to Q2 & Q3
Q2: How much of the robustness improvement can be specifically attributed to spiking dynamics rather than expert specialization and sensor-availability training?
Response to Q2
We thank the reviewer for this attribution question. The main-text ablations and Appendix H examine the contributions of SAL, the expert architecture, and the expert operators through complementary comparisons.
1. General contribution of SAL in Table 14.
| Model | Full | All cameras missing | LiDAR missing |
|---|---|---|---|
| OccMamba + SAL | 26.4 (+0.2) | 16.1 (+6.7) | 8.5 (+6.9) |
| SPSC + SAL | 26.2 (+0.1) | 13.9 (+5.2) | 6.0 (+5.9) |
| SpikeOcc | 25.9 (+1.2) | 22.7 (+7.1) | 9.2 (+1.1) |
2. Architectural gains with SAL fixed in Table 3.
| Variant | SAL | Full | All cameras missing | LiDAR missing |
|---|---|---|---|---|
| Naive SNN | ✗ | 16.7 | 8.7 | 0.0 |
| Naive SNN | ✓ | 22.4 | 16.1 | 8.9 |
| SMoE | ✗ | 24.7 | 15.6 | 8.1 |
| SMoE | ✓ | 25.9 | 22.7 | 9.2 |
3. Spiking-operator gains with architecture and SAL fixed in revised Table 8.
Our additional Q2/Q4 ANN-MoE controls use the same full-training protocol.
| Expert operator | Full | All cameras missing | LiDAR missing |
|---|---|---|---|
| GELU-MoE | 23.4 | 13.7 | 6.8 |
| Q2 ANN-MoE | 20.3 | 12.9 | 4.6 |
| Q4 ANN-MoE | 24.4 | 14.1 | 7.1 |
| PLIF-MoE | 25.9 | 22.7 | 9.2 |
These results clarify the respective contributions.
Q3: Provide the contributions of routing supervision, bounded residual connections, and full-width unimodal experts? A simpler availability-aware gating mechanism?
Response to Q3
We thank the reviewer for suggesting more detailed component attribution. We provide separate ablations for routing supervision, the residual-coefficient bound, unimodal output width, and a simpler availability-aware gate. All performance values below are mIoU.
1. Auxiliary routing supervision.
We remove auxiliary routing CE during training while retaining hard availability masks, the same learnable router, expert architecture, residual design, and SAL.
| Variant | Hard masks | Routing CE | Full | All cameras missing | LiDAR missing |
|---|---|---|---|---|---|
| Mask + routing supervision | ✓ | ✓ | 25.9 | 22.7 | 9.2 |
| Mask only | ✓ | ✗ | 25.7 | 22.4 | 9.0 |
Retaining routing CE improves mIoU by 0.2/0.3/0.2 points across the three settings, supporting a modest, consistent benefit beyond hard availability masking.
2. Bound on the residual coefficient.
We remove the upper bound on the joint residual coefficient while retaining the continuous base, experts, router, hard masks, routing CE, and SAL.
| Variant | Continuous base | Full | All cameras missing | LiDAR missing |
|---|---|---|---|---|
| Bounded residual | ✓ | 25.9 | 22.7 | 9.2 |
| Uncapped residual | ✓ | 25.7 | 22.6 | 9.2 |
The bounded variant improves Full and Camera-off mIoU by 0.2 and 0.1 points, respectively, with unchanged LiDAR-off performance.
3. Full-width unimodal experts.
We separately halve the RGB output width, LiDAR output width, and both widths. Let D denote the shared fusion interface. Half-width outputs have D/2 channels and use fixed zero padding to maintain the D-channel interface. Bottleneck hidden widths and the remaining architecture and training settings are retained.
| Variant | RGB output width | LiDAR output width | Full | All cameras missing | LiDAR missing |
|---|---|---|---|---|---|
| Full-width experts | D | D | 25.9 | 22.7 | 9.2 |
| Half-width RGB | D/2 | D | 25.8 | 22.6 | 8.5 |
| Half-width LiDAR | D | D/2 | 25.5 | 21.2 | 9.1 |
| Half-width RGB and LiDAR | D/2 | D/2 | 25.3 | 21.0 | 8.4 |
These results support full-width outputs for preserving prediction capacity when an individual modality must provide the fallback representation.
4. Simpler availability-aware gating.
We replace the scene-feature-conditioned router with an availability-only linear gate. It receives only the sensor-state vector z=[m_c,m_l,r_c], where r_c is the available-camera fraction, and predicts three expert logits using a linear mapping. This gate has 12 learnable parameters and retains hard masks and routing CE. The protocol are retained.
| Fusion / gating variant | Scene-feature-conditioned gate | Routing CE | Full | All cameras missing | LiDAR missing |
|---|---|---|---|---|---|
| SMoE + original router | ✓ | ✓ | 25.9 | 22.7 | 9.2 |
| SMoE + original router, no CE | ✓ | ✗ | 25.7 | 22.4 | 9.0 |
| SMoE + availability-only linear gate | ✗ | ✓ | 25.8 | 22.6 | 9.2 |
The availability-only linear gate closely matches the original router, with differences of 0.1/0.1/0.0 mIoU points across the three conditions.
We have included these Tables in the revised manuscript.
Response to W2 & Q4
Q4: How practical is the proposed framework for real-time autonomous driving? Can the authors provide end-to-end latency and energy measurements, including camera and LiDAR encoders, rather than theoretical fusion-only estimates?
We thank the reviewer for the questions about practical driving deployment and full-pipeline efficiency.
1. Current implementation and real-time practicality.
The current GPU implementation does not yet meet real-time closed-loop driving requirements. Our objective is to establish the feasibility and sensor-outage resilience of spiking multimodal fusion for dense 3D occupancy prediction, providing an algorithmic foundation for subsequent deployment. To our knowledge, this work is the first to investigate spiking camera–LiDAR expert fusion for 3D semantic occupancy prediction under sensor outages. Real-time deployment requires further pipeline optimization and hardware–algorithm co-design across the encoders, fusion, and occupancy head.
2. Automotive neuromorphic computing developments.
Existing automotive applications and research initiatives provide concrete motivation for this direction:
- Mercedes-Benz VISION EQXX: Mercedes-Benz and BrainChip implemented “Hey Mercedes” hot-word detection using Akida neuromorphic technology [1].
- BMW–SynSense: In 2022, SynSense announced an exploration with BMW of smart-cockpit applications using its Speck neuromorphic vision chip [2].
Mercedes-Benz subsequently announced a research collaboration with the University of Waterloo in 2024 targeting neuromorphic computing for automated driving and driver assistance [3]. These developments motivate extending neuromorphic research from specialized automotive functions to more complex driving-perception tasks. Intel's Loihi 2 and Lava also provide a programmable neuromorphic research platform [4]. A major opportunity lies in combining spiking algorithms with dedicated neuromorphic hardware, which has demonstrated tens-to-hundreds-fold energy-efficiency gains on selected workloads. For example, a published Loihi 2 study reports approximately 75-fold lower total energy for audio denoising relative to its GPU baseline. For PilotNet steering-angle prediction, the I/O-unconstrained implementation uses over 150-fold less energy than the non-batched, 8-bit Jetson Orin Nano GPU baseline, at the evaluated accuracy operating points [5]. These task-specific results make hardware–algorithm co-design a promising route to energy-efficient, continuously operating perception in electric and autonomous vehicles. SpikeOcc provides an algorithmic foundation for exploring this opportunity in resilient multimodal occupancy.
3. Full-model comparison including camera and LiDAR encoders.
We provide full-model computation and latency results covering the camera encoder, LiDAR encoder, fusion, and occupancy prediction modules:
| Method | End-to-end latency (ms) ↓ | Derived serial FPS ↑ | Full-model MACs (G) ↓ | Full-model ACs (G) | Estimated operation energy (mJ) ↓ |
|---|---|---|---|---|---|
| M-CONet | 1918.8 | 0.521 | 4375.8 | 0 | 20128.7 |
| OccMamba+SAL | 1930.0 | 0.518 | 4576.2 | 0 | 21050.5 |
| SpikeOcc-LIF | 1862.0 | 0.537 | 4288.8 | 1.63 | 19729.8 |
| SpikeOcc-PLIF | 1797.3 | 0.556 | 4288.8 | 1.86 | 19730.0 |
SpikeOcc-PLIF has 6.88% lower reported latency and 6.27% lower estimated full-model operation energy than OccMamba+SAL; the corresponding reductions relative to M-CONet are 6.33% and 1.98%. We have added this table to the revision Table 15.
References
[1] Mercedes-Benz. VISION EQXX — taking electric range and efficiency to an entirely new level. Official release.
[2] SynSense. Neuromorphic chip developer SynSense steps into the field of smart cockpits by initiating a technological exploration with BMW. April 14, 2022. Company-issued release.
[3] Mercedes-Benz. Mercedes-Benz expands open innovation activities in Canada alongside research in neuromorphic computing. October 8, 2024. Official announcement.
[4] Intel. Taking Neuromorphic Computing to the Next Level with Loihi 2. Official technology brief.
[5] Sumit Bam Shrestha, Jonathan Timcheck, Paxon Frady, Leobardo Campos-Macias, and Mike Davies. Efficient Video and Audio Processing with Loihi 2. ICASSP 2024. Paper.
Response to W1 & Q1
We thank the reviewer for suggesting gated-fusion and architecture-matched quantized ANN comparisons. We address these two points below.
1. Existing ANN Gated-Fusion Comparison
Regarding the suggestion to compare against lightweight gated fusion, we clarify that Table 1 already includes an established learned ANN gated-fusion baseline, M-CONet [1]. Its adaptive fusion module predicts sigmoid gates to weight camera and LiDAR features before aggregation. This provides an existing comparison with conventional learned gated fusion.
To further examine the representation choice while retaining our routing and expert architecture, we provide the matched quantized controls below.
2. Architecture-Matched Q2/Q4 ANN-MoE Controls
We additionally provide full-training results for Q2 and Q4 ANN-MoE. They replace the six PLIF operators inside the three experts with stateless learned-step activation quantizers, using step-size learning and gradient scaling inspired by LSQ [2]:
\operatorname{clip}\left(\frac{\operatorname{ReLU}(x)}{s},0,2^b-1\right) \right],\qquad s>0.$$ Each quantization site learns its own step size. Q2 uses $b=2$ with four levels, while Q4 uses $b=4$ with sixteen levels. Quantization applies to expert activations; convolution weights remain floating point. | Variant | Expert activation | Membrane state across internal steps | Convolution weights | | --- | --- | --- | --- | | PLIF-MoE | Binary spikes per step | Integration, leak, and reset | Floating point | | Q2 ANN-MoE | 2-bit, four levels | None | Floating point | | Q4 ANN-MoE | 4-bit, sixteen levels | None | Floating point | The encoders, expert topology and widths, continuous paths, residuals, availability masks, router, auxiliary routing supervision, and prediction head are retained. All three use the same full-training protocol and SAL, with T=4 internal unrolling and averaged readout; the quantized ANN variants have no membrane state across steps. | Model | Full | All cameras missing | LiDAR missing | | --- | --- | --- | --- | | PLIF-MoE + SAL | **25.9** | **22.7** | **9.2** | | Q2 ANN-MoE + SAL | 20.3 | 12.9 | 4.6 | | Q4 ANN-MoE + SAL | 24.4 | 14.1 | 7.1 | These results provide direct task-specific evidence for PLIF experts while sharing the gating, expert organization, and training design. We will include the additional experiments and implementation details **in the revised manuscript Table 8**. ## References [1] Wang et al. OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception. ICCV 2023. [Paper](https://arxiv.org/pdf/2303.03991) [2] Esser et al. Learned Step Size Quantization. ICLR 2020. [Paper](https://arxiv.org/pdf/1902.08153).