acceptodds
Withdrawn from ICLR 2027

Adapting 3D Foundations to Spike Cameras at Test Time

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D foundations have substantially advanced novel-view synthesis (NVS) by pretraining on large-scale RGB datasets. However, RGB cameras commonly operate at 30–60 Hz, providing limited temporal resolution for rapid scene motion. This limited temporal resolution restricts the direct application of 3D foundations to high-speed scenes. Spike cameras provide an attractive alternative that captures high-speed scenes as binary streams at up to 40 kHz. To exploit these high-rate spike streams for NVS, we propose SpikeTTT, a framework that adapts 3D foundations to spike cameras at test time. During test-time training, SpikeTTT freezes the pretrained weights and optimizes only lightweight adapters through a tri-signal self-supervised objective. In this way, SpikeTTT adapts 3D foundations to the spike-camera distribution while retaining the benefits of large-scale RGB pretraining. We implement and evaluate SpikeTTT with diverse 3D foundations and NVS tasks. Our evaluation covers posed and pose-free settings on both static and dynamic datasets. With AnySplat, SpikeTTT improves rendering PSNR relative to the frozen cascade by  dB on Tanks and Temples and  dB on DL3DV. It also outperforms spike-specific baselines. SpikeTTT thus provides a practical route to high-quality novel-view synthesis in high-speed scenes by bringing large-scale RGB pretraining to spike-based 3D reconstruction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel view synthesis, Spike Camera, High-Speed Scenes

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w8VaOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines a spike-to-RGB reconstructor with a pretrained NVS model. It adapts small parts of both models per scene, using a spike-derived photometric target and agreement between reconstructed and rendered images.

Critical strengths

This is a practical way to use RGB-pretrained 3D models with spike cameras. The gains over the frozen SJDDNet cascade are clear, and the method works with three different foundations.

Critical weaknesses
  1. The specific contribution is unclear. USP-Gaussian already uses spike-derived targets and reconstruction-rendering consistency. already combines spike-derived supervision with reconstruction–rendering consistency. A comparison using its objective with the same pretrained models, adapters, and compute budget is needed to distinguish SpikeTTT’s contribution from the benefits of scene-specific adaptation.

  2. The main experiments do not validate the high-speed claim. All quantitative spike data is simulated from interpolated RGB videos; real-camera results are qualitative only. This does not establish recovery of fast-motion details that conventional RGB cameras miss. Controlled high-speed measurements and a matched RGB comparison are needed, or the claim should be narrowed.

  3. The practical gain is less convincing than the headline suggests. On Tanks and Temples, the reported 6.98 dB gain is relative to frozen SJDDNet–AnySplat. Against the stronger frozen TFP–AnySplat pipeline, the gain is 1.56 dB and requires about 22 minutes of adaptation per scene. An equally budgeted, simpler adaptation baseline is needed to judge whether this tradeoff is worthwhile.

Questions for authors

see the weakness

Reviewer_QupgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SpikeTTT, which adapts pretrained RGB-based 3D models to spike-camera inputs at test time. It connects a spike-to-image reconstruction network with a 3D foundation model and updates lightweight adapters using reconstruction–rendering consistency and a spike-derived TFP photometric target. However, the evaluations are all based on existing RGB benchmarks, which is quite confusing.

Critical strengths

It is interesting to reuse pretrained 3D models for spike-camera domain. This method works across several architecture architectures and the results show improvements over frozen counterparts.

Critical weaknesses
    1. The high-speed imaging motivation is not adequately supported by the evaluation. All quantitative results use spikes simulated from interpolated RGB videos. Real spike camera results are qualitative only.
    1. It is not surprisingly if the proposed methods tries to match the distributions of Spike camera to existing model, which leads to better performance. The performance gain is quite limited.
Questions for authors
  • Can the authors please clarify their motivation on doing 3D reconstruction on spike cameras?
  • Can the authors please provide a benchmark results on real spike camera inputs?
Reviewer_tvngOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the distribution gap between spike camera data and 3D foundation models learned from RGB cameras. The authors propose SpikeTTT, a test-time training framework for novel view synthesis using spike cameras. Experimental results validate the effectiveness of the proposed approach and demonstrate state-of-the-art (SoTA) performance on both the Tanks and Temples and DL3DV benchmarks.

Critical strengths
  • Framework Design: Proposes the SpikeTTT framework to adapt 3D foundation models to spike cameras using a test-time training protocol.
  • Objective Function: Introduces a tri-signal self-supervised objective to optimize the LoRA adapter.
  • Performance: Demonstrates clear performance improvements over existing task-specific spike baselines.
Critical weaknesses
  • Technical Novelty: The technical novelty is modest, as the proposed method is built primarily on top of existing 3D foundation models using a LoRA adapter.
  • Real-World Evaluation: Quantitative comparisons on real spike data are missing, which raises concerns about the method's performance in real-world scenarios.
Questions for authors
  • Ablation Study: An ablation study or discussion is missing regarding the performance impact of using only a single mutual loss while omitting the remaining loss components.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.