acceptodds
Under review as a conference paper at ICLR 2027

SpikeOcc: Towards Resilient Multimodal 3D Occupancy Perception via Spiking Experts

13 views

Abstract

Multimodal 3D occupancy prediction underpins safe autonomous driving through comprehensive scene understanding. However, existing methods have largely pursued higher accuracy through sophisticated fusion, leaving resilience to missing sensors underexplored. In this paper, we investigate spiking neural networks (SNNs) as a means of building more resilient multimodal perception. Still, the discrete nature of spiking neurons makes optimization challenging. To overcome this difficulty, SpikeOcc is introduced with two complementary components, a Spiking Mixture-of-Experts (SMoE) and Sensor Availability Learning (SAL). Specifically, SMoE organizes RGB, LiDAR, and joint fusion into specialized pathways, enabling full-width unimodal representations while retaining the original multimodal features via a bounded residual. Beyond sensor dropout, SAL employs a staged curriculum with explicit routing supervision, progressively improving robustness under severe sensing conditions. Remarkably, SAL also yields notable gains for conventional models, highlighting its broader potential for resilient multimodal learning. Extensive experiments demonstrate that SpikeOcc maintains competitive full-sensor accuracy while markedly raising the performance floor under sensor outages. SpikeOcc illustrates the promise of neuromorphic computing, pointing toward more reliable and energy-efficient autonomous driving.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spiking Neural Networks, Neuromorphic Computing, 3D Occupancy Prediction, Spiking Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W5C3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpikeOcc, a spiking neural network framework for resilient multimodal 3D occupancy prediction under missing-sensor conditions. The method combines a Spiking Mixture-of-Experts (SMoE) architecture with Sensor Availability Learning (SAL), which progressively introduces sensor failures during training. Experiments on OpenOccupancy and SemanticKITTI demonstrate competitive performance under complete sensing and substantial improvements under sensor outages. The paper also investigates expert specialization, curriculum learning, and computational efficiency.

Critical strengths
  1. The paper addresses a practically important problem in autonomous driving, namely maintaining reliable 3D occupancy prediction under missing-sensor conditions.

  2. The experiments cover two standard benchmarks and demonstrate substantial robustness improvements. The authors also provide relatively comprehensive ablations, including parameter-matched controls, alternative training schedules, and SAL-equipped ANN baselines.

Critical weaknesses

W1. Unclear necessity and methodological novelty of spiking computation.

The motivation for adopting SNNs is not sufficiently justified. The proposed robustness largely relies on availability-aware routing, expert specialization, and sensor-availability training, which can also be implemented using conventional ANNs. In fact, SAL alone substantially improves ANN baselines, while the theoretical robustness argument based on spike thresholds is also applicable to hard-quantized representations. Although the authors provide some ANN-based comparisons, stronger controls using lightweight gated fusion and quantized ANN architectures are missing. It remains unclear what unique advantages SNNs offer beyond existing robust multimodal learning techniques.

W2. Limited practical applicability and insufficient experimental validation.

Despite targeting reliable and energy-efficient autonomous driving, SpikeOcc reports approximately 1.8 seconds of inference latency, while the claimed energy savings are theoretical estimates limited to the fusion module, excluding conventional camera and LiDAR encoders. Moreover, the evaluation assumes that sensor failures are explicitly known, whereas real-world failures often involve corrupted or unreliable observations rather than complete sensor outages. The ablations also do not sufficiently isolate several key architectural choices, including routing supervision, bounded residuals, and expert width. These limitations make it difficult to assess whether the proposed framework offers meaningful practical advantages over simpler alternatives.

Questions for authors
  1. What are the fundamental advantages of spiking computation over conventional ANN-based gated fusion or mixture-of-experts architectures? Can the authors provide comparisons with architecture-matched quantized ANN baselines under identical training settings?

  2. Given that SAL alone substantially improves conventional ANN baselines, how much of the robustness improvement can be specifically attributed to spiking dynamics rather than expert specialization and sensor-availability training?

  3. Can the authors provide more fine-grained ablations isolating the contributions of routing supervision, bounded residual connections, and full-width unimodal experts? Would a simpler availability-aware gating mechanism achieve comparable performance?

  4. Given the reported inference latency of approximately 1.8 seconds, how practical is the proposed framework for real-time autonomous driving? Can the authors provide end-to-end latency and energy measurements, including camera and LiDAR encoders, rather than theoretical fusion-only estimates?

  5. How well does SpikeOcc handle realistic sensor failures, including corrupted observations, calibration errors, and uncertain sensor availability? Does the reported robustness persist when sensor failures are not explicitly provided to the model?

5 comments
AuthorsOpenReview ↗

Response to W1 & Q1

W1: Although the authors provide some ANN-based comparisons, stronger controls using lightweight gated fusion and quantized ANN architectures are missing.

Q1: What are the fundamental advantages of spiking computation over conventional ANN-based gated fusion or mixture-of-experts architectures? Can the authors provide comparisons with architecture-matched quantized ANN baselines under identical training settings?

We thank the reviewer for suggesting gated-fusion and architecture-matched quantized ANN comparisons. We address these two points below.

1. Existing ANN Gated-Fusion Comparison

Regarding the suggestion to compare against lightweight gated fusion, we clarify that Table 1 already includes an established learned ANN gated-fusion baseline, M-CONet [1]. Its adaptive fusion module predicts sigmoid gates to weight camera and LiDAR features before aggregation. This provides an existing comparison with conventional learned gated fusion.

MethodFusion designIoUmIoU
M-CONet [1]Learned sigmoid-gated fusion29.520.1
SpikeOccSMoE35.825.9
Improvement+6.3+5.8

To further examine the representation choice while retaining our routing and expert architecture, we provide the matched quantized controls below.

2. Architecture-Matched Q2/Q4 ANN-MoE Controls

We additionally provide full-training results for Q2 and Q4 ANN-MoE. They replace the six PLIF operators inside the three experts with stateless learned-step activation quantizers, using step-size learning and gradient scaling inspired by LSQ [2]:

\operatorname{clip}\left(\frac{\operatorname{ReLU}(x)}{s},0,2^b-1\right) \right],\qquad s>0.$$ Each quantization site learns its own step size. Q2 uses $b=2$ with four levels, while Q4 uses $b=4$ with sixteen levels. Quantization applies to expert activations; convolution weights remain floating point. | Variant | Expert activation | Membrane state across internal steps | Convolution weights | | --- | --- | --- | --- | | PLIF-MoE | Binary spikes per step | Integration, leak, and reset | Floating point | | Q2 ANN-MoE | 2-bit, four levels | None | Floating point | | Q4 ANN-MoE | 4-bit, sixteen levels | None | Floating point | The encoders, expert topology and widths, continuous paths, residuals, availability masks, router, auxiliary routing supervision, and prediction head are retained. All three use the same full-training protocol and SAL, with T=4 internal unrolling and averaged readout; the quantized ANN variants have no membrane state across steps. | Model | Full | All cameras missing | LiDAR missing | | --- | --- | --- | --- | | PLIF-MoE + SAL | **25.9** | **22.7** | **9.2** | | Q2 ANN-MoE + SAL | 20.3 | 12.9 | 4.6 | | Q4 ANN-MoE + SAL | 24.4 | 14.1 | 7.1 | These results provide direct task-specific evidence for PLIF experts while sharing the gating, expert organization, and training design. We will include the additional experiments and implementation details **in the revised manuscript Table 8**. ## References [1] Wang et al. OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception. ICCV 2023. [Paper](https://arxiv.org/pdf/2303.03991) [2] Esser et al. Learned Step Size Quantization. ICLR 2020. [Paper](https://arxiv.org/pdf/1902.08153).
AuthorsOpenReview ↗

Response to Q2 & Q3

Q2: How much of the robustness improvement can be specifically attributed to spiking dynamics rather than expert specialization and sensor-availability training?

Response to Q2

We thank the reviewer for this attribution question. The main-text ablations and Appendix H examine the contributions of SAL, the expert architecture, and the expert operators through complementary comparisons.

1. General contribution of SAL in Table 14.

ModelFullAll cameras missingLiDAR missing
OccMamba + SAL26.4 (+0.2)16.1 (+6.7)8.5 (+6.9)
SPSC + SAL26.2 (+0.1)13.9 (+5.2)6.0 (+5.9)
SpikeOcc25.9 (+1.2)22.7 (+7.1)9.2 (+1.1)

2. Architectural gains with SAL fixed in Table 3.

VariantSALFullAll cameras missingLiDAR missing
Naive SNN✗16.78.70.0
Naive SNN✓22.416.18.9
SMoE✗24.715.68.1
SMoE✓25.922.79.2

3. Spiking-operator gains with architecture and SAL fixed in revised Table 8.

Our additional Q2/Q4 ANN-MoE controls use the same full-training protocol.

Expert operatorFullAll cameras missingLiDAR missing
GELU-MoE23.413.76.8
Q2 ANN-MoE20.312.94.6
Q4 ANN-MoE24.414.17.1
PLIF-MoE25.922.79.2

These results clarify the respective contributions.

Q3: Provide the contributions of routing supervision, bounded residual connections, and full-width unimodal experts? A simpler availability-aware gating mechanism?

Response to Q3

We thank the reviewer for suggesting more detailed component attribution. We provide separate ablations for routing supervision, the residual-coefficient bound, unimodal output width, and a simpler availability-aware gate. All performance values below are mIoU.

1. Auxiliary routing supervision.

We remove auxiliary routing CE during training while retaining hard availability masks, the same learnable router, expert architecture, residual design, and SAL.

VariantHard masksRouting CEFullAll cameras missingLiDAR missing
Mask + routing supervision✓✓25.922.79.2
Mask only✓✗25.722.49.0

Retaining routing CE improves mIoU by 0.2/0.3/0.2 points across the three settings, supporting a modest, consistent benefit beyond hard availability masking.

2. Bound on the residual coefficient.

We remove the upper bound on the joint residual coefficient while retaining the continuous base, experts, router, hard masks, routing CE, and SAL.

VariantContinuous baseFullAll cameras missingLiDAR missing
Bounded residual✓25.922.79.2
Uncapped residual✓25.722.69.2

The bounded variant improves Full and Camera-off mIoU by 0.2 and 0.1 points, respectively, with unchanged LiDAR-off performance.

3. Full-width unimodal experts.

We separately halve the RGB output width, LiDAR output width, and both widths. Let D denote the shared fusion interface. Half-width outputs have D/2 channels and use fixed zero padding to maintain the D-channel interface. Bottleneck hidden widths and the remaining architecture and training settings are retained.

VariantRGB output widthLiDAR output widthFullAll cameras missingLiDAR missing
Full-width expertsDD25.922.79.2
Half-width RGBD/2D25.822.68.5
Half-width LiDARDD/225.521.29.1
Half-width RGB and LiDARD/2D/225.321.08.4

These results support full-width outputs for preserving prediction capacity when an individual modality must provide the fallback representation.

4. Simpler availability-aware gating.

We replace the scene-feature-conditioned router with an availability-only linear gate. It receives only the sensor-state vector z=[m_c,m_l,r_c], where r_c is the available-camera fraction, and predicts three expert logits using a linear mapping. This gate has 12 learnable parameters and retains hard masks and routing CE. The protocol are retained.

Fusion / gating variantScene-feature-conditioned gateRouting CEFullAll cameras missingLiDAR missing
SMoE + original router✓✓25.922.79.2
SMoE + original router, no CE✓✗25.722.49.0
SMoE + availability-only linear gate✗✓25.822.69.2

The availability-only linear gate closely matches the original router, with differences of 0.1/0.1/0.0 mIoU points across the three conditions.

We have included these Tables in the revised manuscript.

AuthorsOpenReview ↗

Response to W2 & Q4

Q4: How practical is the proposed framework for real-time autonomous driving? Can the authors provide end-to-end latency and energy measurements, including camera and LiDAR encoders, rather than theoretical fusion-only estimates?

We thank the reviewer for the questions about practical driving deployment and full-pipeline efficiency.

1. Current implementation and real-time practicality.

The current GPU implementation does not yet meet real-time closed-loop driving requirements. Our objective is to establish the feasibility and sensor-outage resilience of spiking multimodal fusion for dense 3D occupancy prediction, providing an algorithmic foundation for subsequent deployment. To our knowledge, this work is the first to investigate spiking camera–LiDAR expert fusion for 3D semantic occupancy prediction under sensor outages. Real-time deployment requires further pipeline optimization and hardware–algorithm co-design across the encoders, fusion, and occupancy head.

2. Automotive neuromorphic computing developments.

Existing automotive applications and research initiatives provide concrete motivation for this direction:

  • Mercedes-Benz VISION EQXX: Mercedes-Benz and BrainChip implemented “Hey Mercedes” hot-word detection using Akida neuromorphic technology [1].
  • BMW–SynSense: In 2022, SynSense announced an exploration with BMW of smart-cockpit applications using its Speck neuromorphic vision chip [2].

Mercedes-Benz subsequently announced a research collaboration with the University of Waterloo in 2024 targeting neuromorphic computing for automated driving and driver assistance [3]. These developments motivate extending neuromorphic research from specialized automotive functions to more complex driving-perception tasks. Intel's Loihi 2 and Lava also provide a programmable neuromorphic research platform [4]. A major opportunity lies in combining spiking algorithms with dedicated neuromorphic hardware, which has demonstrated tens-to-hundreds-fold energy-efficiency gains on selected workloads. For example, a published Loihi 2 study reports approximately 75-fold lower total energy for audio denoising relative to its GPU baseline. For PilotNet steering-angle prediction, the I/O-unconstrained implementation uses over 150-fold less energy than the non-batched, 8-bit Jetson Orin Nano GPU baseline, at the evaluated accuracy operating points [5]. These task-specific results make hardware–algorithm co-design a promising route to energy-efficient, continuously operating perception in electric and autonomous vehicles. SpikeOcc provides an algorithmic foundation for exploring this opportunity in resilient multimodal occupancy.

3. Full-model comparison including camera and LiDAR encoders.

We provide full-model computation and latency results covering the camera encoder, LiDAR encoder, fusion, and occupancy prediction modules:

MethodEnd-to-end latency (ms) ↓Derived serial FPS ↑Full-model MACs (G) ↓Full-model ACs (G)Estimated operation energy (mJ) ↓
M-CONet1918.80.5214375.8020128.7
OccMamba+SAL1930.00.5184576.2021050.5
SpikeOcc-LIF1862.00.5374288.81.6319729.8
SpikeOcc-PLIF1797.30.5564288.81.8619730.0

SpikeOcc-PLIF has 6.88% lower reported latency and 6.27% lower estimated full-model operation energy than OccMamba+SAL; the corresponding reductions relative to M-CONet are 6.33% and 1.98%. We have added this table to the revision Table 15.

References

[1] Mercedes-Benz. VISION EQXX — taking electric range and efficiency to an entirely new level. Official release.

[2] SynSense. Neuromorphic chip developer SynSense steps into the field of smart cockpits by initiating a technological exploration with BMW. April 14, 2022. Company-issued release.

[3] Mercedes-Benz. Mercedes-Benz expands open innovation activities in Canada alongside research in neuromorphic computing. October 8, 2024. Official announcement.

[4] Intel. Taking Neuromorphic Computing to the Next Level with Loihi 2. Official technology brief.

[5] Sumit Bam Shrestha, Jonathan Timcheck, Paxon Frady, Leobardo Campos-Macias, and Mike Davies. Efficient Video and Audio Processing with Loihi 2. ICASSP 2024. Paper.

Reviewer_zwS6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpikeOcc for robust multimodal 3D occupancy prediction under sensor outages. It combines a Spiking Mixture-of-Experts (SMoE) with Sensor Availability Learning (SAL), which progressively introduces missing-sensor conditions. Experiments on OpenOccupancy and SemanticKITTI demonstrate competitive full-sensor accuracy and improved robustness under sensor failures.

Critical strengths
  1. Practical relevance. The paper addresses sensor failures in multimodal 3D occupancy prediction, an important challenge for autonomous driving.

  2. Empirical evidence under sensor outages. Evaluations cover multiple sensor outages and unseen camera combinations. With matched SAL training, SpikeOcc achieves 22.7 mIoU under complete camera failure, compared with 16.1 for OccMamba+SAL (Appendix Table 9).

  3. Useful ablations. Comparisons of GELU/PLIF experts, model capacities, and curriculum schedules support the design choices. SAL also improves conventional ANN models, suggesting broader applicability.

Critical weaknesses
  1. The contribution of spiking mechanisms remains insufficiently explained. Appendix A's threshold-margin proposition is valid but establishes only local spike stability, not whole-network robustness. Although Table 3 favors PLIF over GELU, the weaker GELU baseline raises questions about optimization adequacy. The gains may also arise from expert specialization, residual design, and routing. Stronger evidence linking spiking dynamics to robustness would help clarify the spiking-specific novelty.

  2. The efficiency evaluation is limited. The reported 9.4 mJ is a theoretical estimate for the fusion module, not measured system-level energy consumption. ANN encoders and continuous-valued operations further limit end-to-end efficiency claims. Table 7 compares against M-CONet rather than OccMamba+SAL. Although latency (1797.3 ms) and throughput (4.45 FPS) are consistent with eight-GPU aggregate throughput, their measurement protocols need clarification. Matched full-model profiling or narrower efficiency claims would address this concern.

  3. Statistical evidence and component attribution remain incomplete. Variance is not reported despite the use of three random seeds, particularly for the 0.7 mIoU gain under LiDAR failure with matched SAL training. Although curriculum ablations are provided, the additional contribution of auxiliary routing supervision beyond hard availability masks remains unclear. Seed-level statistics and a routing-loss ablation would strengthen the evidence.

Questions for authors
  1. Was the GELU MoE optimized comparably to PLIF MoE? Can the authors provide evidence linking spike stability to prediction robustness?
  2. Can the authors clarify the latency/throughput protocols and compare full-model efficiency with OccMamba+SAL?
  3. Can the authors report seed-level variance and ablate auxiliary routing supervision while retaining availability masks?
5 comments
AuthorsOpenReview ↗

Response to W1 & Q1

W1: Appendix A's threshold-margin proposition? GELU baseline optimization adequacy? Expert specialization, residual design, and routing?

Q1: Was the GELU MoE optimized comparably to PLIF MoE? Can the authors provide evidence linking spike stability to prediction robustness?

1. Evidence Connecting Spike Stability to Robustness

We thank the reviewer for recognizing the validity of the threshold-margin proposition. We provide further support through prior research and additional experiments in our task.

Prior research provides relevant motivation: Ding et al. investigate stability for particular membrane-perturbation dynamics and associated training methods, while their later work studies adversarial robustness through specialized temporal encoding and decoding. ICML 2024, Nature Communications 2025.

We additionally provide task-specific evidence through architecture-matched Q2/Q4 ANN-MoE controls. They replace the six PLIF operators inside the three experts with stateless 2-bit/4-bit learned-step activation quantizers, yielding 4/16 discrete levels. Convolution weights remain floating point. The expert architecture, continuous paths, residuals, routing, SAL, full-training protocol, four-step execution, and averaged readout are retained.

ModelExpert activationMembrane stateFullAll cameras missingLiDAR missing
PLIF-MoEBinary spikes per internal stepIntegration, leak, reset25.922.79.2
Q2 ANN-MoE2-bit, 4 levelsNone20.312.94.6
Q4 ANN-MoE4-bit, 16 levelsNone24.414.17.1

PLIF maintains its advantage over Q2/Q4 under full sensing and both modality-outage conditions. Prior research and our revised empirical results in Table 8 jointly support the motivation of this work and the value of spiking experts.

2. Implementation and Optimization of GELU-MoE

GELU-MoE retains the expert organization, convolutional structure and channel widths, continuous readouts, joint residual, availability masks, router, auxiliary routing supervision, occupancy head, and SAL curriculum, replacing the expert PLIF nonlinearities with GELU. We additionally provide validation curves to document the optimization trajectories, as shown in Figure 12.

3. Independent Contributions of Expert Architecture, Residual Bound, and Routing Supervision

3(a). Expert Architecture

The expert-architecture versus capacity-scaling comparison is already reported in Table 4, “Parameter-matched comparison,” under Capacity Scaling. All models below are trained without SAL.

ModelParameters (M)FullAll cameras missingLiDAR missing
Naive SNN109.116.78.70.0
Dense SNN-S111.116.18.10.1
Dense SNN-M141.818.08.51.6
Dense SNN-L158.118.58.93.7
SMoE112.324.715.68.1

SMoE outperforms the larger Dense SNN-L by 6.2/6.7/4.4 mIoU points, showing that increasing dense-SNN capacity does not reproduce the gains of the tested expert architecture.

3(b). Residual-Coefficient Bound

This control retains the continuous base, expert architecture, router, hard availability masks, auxiliary routing CE, and SAL, while removing the upper bound on the joint residual coefficient to evaluate the effect of this constraint.

VariantContinuous baseFullAll cameras missingLiDAR missing
Bounded residual✓25.922.79.2
Uncapped residual✓25.722.69.2

Retaining the upper bound improves Full and Camera-off mIoU by 0.2 and 0.1 points, respectively, while preserving LiDAR-off performance. These results are revised in Table 10.

3(c). Auxiliary Routing CE

This control removes auxiliary routing CE during training while retaining the same learnable router, hard availability masks, expert architecture, residual design, and SAL. It examines the additional effect of routing supervision beyond availability masking.

VariantHard masksAuxiliary routing CEFullAll cameras missingLiDAR missing
Mask + routing supervision✓✓25.922.79.2
Mask only✓✗25.722.49.0

These results are revised in Table 11.

References

  1. Jianhao Ding, Zhaofei Yu, Jian K. Liu, and Tiejun Huang. Neuromorphic computing paradigms enhance robustness through spiking neural networks. Nature Communications 16, 10175 (2025). Paper.
  2. Jianhao Ding, Zhiyu Pan, Yujia Liu, Zhaofei Yu, and Tiejun Huang. Robust Stable Spiking Neural Networks. ICML 2024. Paper.
AuthorsOpenReview ↗

Response to W2 & Q2

W2: System-level energy consumption? Table 7 compares with OccMamba+SAL? Measurement protocols clarification?

Q2: Can the authors clarify the latency/throughput protocols and compare full-model efficiency with OccMamba+SAL?

We thank the reviewer for the suggestions on our efficiency evaluation. We address the energy methodology, full-model comparison, updated Table 7, and measurement protocol below.

1. Energy estimation methodology.

The Computational Efficiency paragraph explicitly identifies 9.4 mJ as the theoretical energy of the fusion module. We estimate energy from MAC/AC counts, following an approach commonly used in recent SNN research, including SAD (NeurIPS 2024) [1] and Spiking Point Transformer (AAAI 2025) [2].

2. Full-model efficiency including the encoders.

We supplement the fusion analysis with full-model accounting for the camera encoder, LiDAR encoder, fusion, and occupancy prediction modules. We will include Table below in Appendix J of the revised manuscript.

VariantFull-model MACs (G) ↓Full-model ACs (G)Estimated operation energy (mJ) ↓
M-CONet4,375.8020,128.7
OccMamba+SAL4,576.2021,050.5
SpikeOcc-LIF4,288.81.6319,729.8
SpikeOcc-PLIF4,288.81.8619,730.0

SpikeOcc-PLIF requires approximately 4,288.8 G MACs, reducing the full-model MAC count by 1.99% and 6.28% relative to M-CONet and OccMamba+SAL, respectively. These estimates retain an operation-cost advantage when the continuous-valued encoders are included. 

3. Updated Table 7.

We include OccMamba+SAL in the updated comparison below and will incorporate the table and corresponding analysis into the revised manuscript.

VariantLatency (ms) ↓FPS ↑Fusion MACs (G)Fusion ACs (G)Fusion energy (mJ) ↓
M-CONet1918.84.1714.7067.6
OccMamba+SAL1930.04.15000
SpikeOcc-LIF1862.04.301.681.639.2
SpikeOcc-PLIF1797.34.451.681.869.4

4. Latency and throughput protocol.

The manuscript reports measurements on eight NVIDIA A800 GPUs, batch size 1, matched input settings, and warm-up. Table below distinguishes single-sample latency and derived serial throughput from the eight-GPU-scale FPS reported.

VariantReported end-to-end latency (ms) ↓Serial FPS (1000/latency) ↑Latency reduction vs. OccMamba+SAL ↑
M-CONet1918.80.5210.58%
OccMamba+SAL1930.00.5180.00%
SpikeOcc-LIF1862.00.5373.52%
SpikeOcc-PLIF1797.30.5566.88%

References

[1] Zhu et al. Autonomous Driving with Spiking Neural Networks. NeurIPS 2024. Appendix F describes MAC/AC-based theoretical energy estimation. Paper.

[2] Wu et al. Spiking Point Transformer for Point Cloud Classification. AAAI 2025. The energy analysis and Table 5 use MAC/AC operation costs. Paper.

AuthorsOpenReview ↗

Response to W3 & Q3

W3: Variance is not reported. Additional contribution of auxiliary routing supervision beyond hard availability masks?

Q3: Can the authors report seed-level variance and ablate auxiliary routing supervision while retaining availability masks?

We thank the reviewer for the suggestions on seed-level statistics and component attribution.

1. Seed-Level Statistics

We now report the individual results from three independently trained seeds, together with their mean, sample standard deviation, and sample variance, for full sensing, complete camera outage, and LiDAR outage.

ModelSensor conditionIndividual seed resultsMean ± SDSample variance
OccMamba+SALFull26.2 / 26.4 / 26.626.40 ± 0.200.040
OccMamba+SALAll cameras missing16.1 / 16.2 / 16.016.10 ± 0.100.010
OccMamba+SALLiDAR missing8.5 / 8.2 / 8.88.50 ± 0.300.090
SpikeOccFull25.9 / 25.9 / 25.925.90 ± 0.000.000
SpikeOccAll cameras missing22.4 / 22.7 / 23.022.70 ± 0.300.090
SpikeOccLiDAR missing9.2 / 9.2 / 9.29.20 ± 0.000.000

Given the space constraints, we present the complete seed-level statistics here for the reviewer’s reference, keeping the manuscript focused on the main results and analysis.

2. Additional Contribution of Auxiliary Routing Supervision

We provide an ablation that retains hard availability masks. This control removes auxiliary routing CE during training while retaining the same learnable router, hard availability masks, expert architecture, residual design, and SAL.

VariantHard masksAuxiliary routing CEFullAll cameras missingLiDAR missing
Mask + routing supervision✓✓25.922.79.2
Mask only✓✗25.722.49.0
Gain——+0.2+0.3+0.2

Retaining routing CE improves over mask-only training by 0.2/0.3/0.2 mIoU points across the three conditions, supporting an additional performance benefit from auxiliary routing supervision beyond hard availability masking. We will include this comparison and analysis in the main paper Table 11.

Reviewer_rx5EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes a method of Spiking Mixture of Experts and Sensor Availability Learning, aiming to maintain a certain performance for multimodal models even in the absence of sensors. The experimental results currently appear to be reliable

Critical strengths
  1. A good applaction of spiking neural networks in Occupancy tasks
  2. The experiment is complete and can demonstrate that the proposed methodcan bring certain performance stability in the absence of sensors
Critical weaknesses
  1. Insufficient motivation and interpretability for the usage of Spiking Mixture of Experts because there are so many MoE structure network such as vision-moe, switch transformer etc...
  2. Although experiments show the robustness in missing sensors, the whole sensors results are weakness, specially in mIoU
  3. The article did not explain the structure and principle of the Spkiing Fusion Module
Questions for authors
  1. What is the usage insight of Spiking Mixture of Expert
  2. What is the design of Spiking Fusion Module and how to explain only one head that achieves coarse to fine effects
4 comments
AuthorsOpenReview ↗

Response to W1 & Q1

W1: Insufficient motivation and interpretability for the usage of Spiking Mixture of Experts because there are so many MoE structure network such as vision-moe, switch transformer etc...

Q1: What is the usage insight of Spiking Mixture of Expert

We thank the reviewer for recommending V-MoE [1] and Switch Transformer [2], influential contributions from Google to sparse expert modeling. We will discuss their relationship to our method in Related Work and clarify our motivation and supporting evidence below.

1. Motivation

Directly transferring multimodal occupancy fusion to the spiking regime introduces a specific optimization difficulty. Figure 2 shows pronounced gradient amplification and oscillation in the Naive SNN; Appendix B, Figure 11 further localizes this behavior mainly to the fusion module, while the camera and LiDAR branches remain comparatively stable. SpikeOcc exhibits substantially smoother fusion gradients.

These observations motivate separate RGB, LiDAR, and joint expert pathways, preserving modality-specific processing alongside cross-modal interaction. Continuous information paths and spiking corrections further ease optimization.

2. Design evidence

(a) Capacity controls. Table 4 compares models trained without SAL. Increasing dense-SNN capacity does not reproduce the SMoE gains:

ModelParameters (M)Full sensingAll cameras missingLiDAR missing
Naive SNN109.116.78.70.0
Dense SNN-L158.118.58.93.7
SMoE112.324.715.68.1

All performance columns report mIoU. SMoE outperforms the larger Dense SNN-L across all three conditions, supporting expert specialization beyond capacity scaling. To further examine the choice of spiking computation, we additionally compare against architecture-matched quantized ANN-MoE variants.

(b) Q2 and Q4 ANN-MoE controls. We replace the six PLIF operators within the three experts with stateless learned-step activation quantizers inspired by LSQ [3]:

\operatorname{clip}\left( \frac{\operatorname{ReLU}(x)}{s},\,0,\,2^b-1 \right)\right],$$ where each quantization site has a learnable step size $s>0$, initialized from training activations and optimized using a straight-through estimator with LSQ-style step-size gradient scaling. Setting $b=2$ or $b=4$ produces Q2 or Q4, respectively: | Variant | Activation precision | Output levels at each site | Membrane dynamics | Convolution weights | | --- | --- | --- | --- | --- | | PLIF MoE | Binary spikes per internal step | $\{0,1\}$ per step, followed by temporal readout | Integration, leak, and reset | Floating point | | Q2 ANN-MoE | 2-bit activations | 4 levels: $\{0,s,2s,3s\}$ | None | Floating point | | Q4 ANN-MoE | 4-bit activations | 16 levels: $\{0,s,2s,\ldots,15s\}$ | None | Floating point | Q2/Q4 denote expert activation precision. All other architectural components and the full-training/evaluation protocol, including SAL and $T=4$ are kept unchanged. | Model | Full sensing mIoU | All cameras missing mIoU | LiDAR missing mIoU | | --- | --- | --- | --- | | PLIF MoE + SAL | **25.9** | **22.7** | **9.2** | | Q2 ANN-MoE + SAL | 20.3 | 12.9 | 4.6 | | Q4 ANN-MoE + SAL | 24.4 | 14.1 | 7.1 | Together, these comparisons support both the expert architecture and the empirical value of PLIF experts: neither dense capacity scaling nor the tested stateless quantized activations reproduce SMoE's performance. **We will include the additional controls and implementation details in the Ablation Study**, and strengthen the motivation and design rationale in the revision. ## References [1] Carlos Riquelme et al. Scaling Vision with Sparse Mixture of Experts. NeurIPS, 2021. [Paper](https://papers.nips.cc/paper_files/paper/2021/file/48237d9f2dea8c74c2a72126cf63d933-Paper.pdf). [2] William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR, 2022. [Paper](https://jmlr.org/papers/volume23/21-0998/21-0998.pdf). [3] Steven K. Esser et al. Learned Step Size Quantization. ICLR, 2020. [Paper](https://arxiv.org/pdf/1902.08153).
AuthorsOpenReview ↗

Response to W2

W2: Although experiments show the robustness in missing sensors, the whole sensors results are weakness, specially in mIoU

We thank the reviewer for highlighting this limitation. We acknowledge that SpikeOcc does not achieve the best full-sensor semantic accuracy in our comparison. Our objective is to improve resilience to sensor outages while maintaining competitive full-sensor accuracy.

Failure-case analysis. We performed a paired analysis of SpikeOcc and OccMamba+SAL on all 6,019 OpenOccupancy validation samples, with all six cameras and LiDAR available. Both models use their existing checkpoints and original inference settings, with the same image resolution, evaluation grid, annotations, and evaluation mask.

We have added the following failure-case figure to the ablation study, comparing Ground Truth, OccMamba+SAL, and SpikeOcc with identical crops and viewpoints. The examples highlight weaknesses in construction vehicle, traffic cone, and bicycle: SpikeOcc misses more occupied voxels for construction vehicles and traffic cones. These findings identify object recovery and control of false positives as priorities for improving full-sensor accuracy.

AuthorsOpenReview ↗

Response to W3 & Q2

W3: The article did not explain the structure and principle of the Spkiing Fusion Module

Q2: What is the design of Spiking Fusion Module and how to explain only one head that achieves coarse to fine effects

We thank the reviewer for highlighting the insufficient explanation of the fusion module and prediction pathway. We have revised Fig. 3 to clarify the module boundaries and distinguish spiking fusion from coarse-to-fine prediction.

1. Clarification of the spiking fusion module

SMoE itself constitutes the spiking fusion module: the RGB, LiDAR, and joint experts produce features that are aggregated by an availability-aware router,

F=∑k∈{c,l,j}wkFk,\mathbf F=\sum_{k\in\{c,l,j\}}w_k\mathbf F_k,

and passed directly to the occupancy head. There is no additional standalone spiking network after expert aggregation. The current layout of Fig. 3 can suggest otherwise, and we will correct it accordingly.

2. Coarse-to-fine prediction within one occupancy head

For a fair comparison, we retain the prediction-head architecture and coarse-to-fine procedure of OccMamba [1]. The single head box in Fig. 3 denotes the complete prediction subsystem, rather than a single classification layer.

In Sec. 3.2, the “Occupancy head” paragraph of OccMamba describes a coarse-to-fine module “which interpolates the dimensions to match the size of the ground truth,” followed by an MLP for voxel classification. Its Eq. (6) gives

=\operatorname{MLP}\left( F_{c\rightarrow f}(\mathbf V_P) \right),$$ where $\mathbf V_P$ denotes voxel features after OccMamba feature processing. We will make this division explicit in **Appendix F**. ## Reference [1] Heng Li et al. OccMamba: Semantic Occupancy Prediction with State Space Models. CVPR, 2025. See Sec. 3.2, “Occupancy head,” Eq. (6). [Paper](https://arxiv.org/pdf/2408.09859).
AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer rx5E,

Thank you again for your time and constructive feedback. Your questions about the motivation and empirical support for the spiking expert design overlap with concerns raised by reviewer zwS6, who has indicated that most concerns have been addressed following our additional experiments.

We have provided quantized ANN comparisons and further design evidence, added failure-case analysis, and clarified the fusion module and prediction head.

We have not yet received your feedback on these additions. As the discussion period is nearing its end, we would greatly appreciate hearing your thoughts. Please let us know if any concerns remain. We are ready to respond promptly and provide further clarification.

Thank you again for your consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Experimental Evidence

Dear Reviewers,

We sincerely thank all reviewers for your thoughtful and constructive feedback. We have carefully considered the shared concerns regarding the unique contributions of spiking computation, the attribution of robustness improvements, and the practical efficiency and reliability of our framework.

To address these concerns, we have provided architecture-matched ANN comparisons (Table 8), extensive component ablations (Table 9-12), multi-seed statistics, and full-model efficiency analyses (Table 15), along with further methodological clarifications and a discussion of practical limitations. We believe these additional experiments and explanations substantially strengthen the evidence supporting our contributions and address the central concerns raised during the review.

We would be truly grateful if you could revisit our detailed responses and reconsider your evaluation in light of the new evidence. We warmly welcome any further questions or suggestions and would be happy to provide additional clarification.

Thank you again for your valuable time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.