acceptodds
Withdrawn from ICLR 2027

Learning to Trust Generative Features for Spike Image Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spike cameras encode dynamic scenes as binary firing streams with ultra-high temporal resolution and high dynamic range. Recovering fine details from spike sequences remains challenging for reconstruction models based on direct regression. Generative restoration can provide complementary information, but its reliability varies across image regions. We propose LTGF, a framework that learns to trust generative features to enhance a base regression network. Conditioned on base spike features, a shared flow matching network starts from coarse intensity estimates based on spike firing rates (TFP) and inter-spike intervals (TFI) to generate complementary candidates. To select useful candidate information, multi-level uncertainty estimates in log space are combined with a learned content-adaptive tolerance that accounts for reconstruction benefit. The resulting weights control residual feature corrections while retaining the base representation. Experiments demonstrate the best performance among the compared methods on synthetic spike data and real spike data.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
spike camera, image reconstruction, flow matching, uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tiXBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the image reconstruction of spike cameras and proposes LTGF, which utilizes a feature enhanced basic regression network with generative restoration branches. The model constructs TFP images based on the frequency of issuance and TFI images based on the spike interval from the same spike sequence, serving as two starting points for the shared and spike conditional flow matching network. Each source infers candidate images and multi-scale features through a single step calculation. The experiment covers synthetic spike data derived from REDS and six real scenes, reporting reconstruction quality, computational complexity, ablation, as well as uncertainty and candidate error relationships on the test set.

Critical strengths
  1. Obtained a valuable compromise between reconstruction quality and computational cost. In Table 1, the PSNR of LTGF is 43.4765 dB, which is 0.2891 dB higher than the strongest listed baseline Spike ODE. SSIM and LPIPS also improve simultaneously. According to the report in Table 5, the number of parameters and FLOPs decreased by approximately 46.4% and 83.7%, respectively. This result of improving reconstruction quality while reducing computational costs has practical significance for high temporal resolution imaging tasks.

  2. The verification of 'uncertainty assisted selection' is relatively complete. The attention fusion comparison in Table 4 retains the same base extractor, two candidate sources, residual correction, and decoder, making it more explanatory than simply removing the entire branch. The complete method improved the control by approximately 0.2914 dB and reduced the LPIPS from 0.0384 to 0.0299. Appendix F further validated the relationship between prediction uncertainty and candidate error on all 80 synthetic test samples and 20 scenes, and provided scenes clustering bootstrap intervals. These pieces of evidence collectively support using the prediction error scale as a fusion cue.

Critical weaknesses

W1. The contribution of flow matching training itself has not been isolated yet. According to equations (6) and (8), at the t=0 endpoint used for fusion in practice, velocity supervision is equivalent to mean square error supervision of candidate images. The existing results cannot clearly distinguish the benefits brought by random flow time training from those brought by additional auxiliary recovery networks, TFP/TFI inputs, and joint supervision. Table 3 validates the role of candidate sources, while Table 4 validates the fusion strategy, but lacks a control that maintains consistency between the network and fusion structure and only uses endpoint residual regression training. This affects the assessment of the core contribution of 'generative features'.

W2. The relative PSNR gain of SpikeODE and attention fusion is about 0.29 dB. The bootstrap analysis in Appendix F focuses on the relationship between uncertainty and candidate errors, but has not yet covered the differences in final reconstruction performance. The average advantage of real data is also relatively small and dependent on the scenes: for example, the average gain of MUSIQ is 0.1000, where the gain of Car is 0.7350, and the average difference of the other five scenes is about -0.0270. The existing results support the average improvement on the evaluated data, but more caution is still needed in assessing the stability of the advantages across different scenes.

Questions for authors

Q1. Can you retain the same shared U-Net, TFP/TFI starting point, uncertainty header, and fusion module, remove only random training with t > 0, and use residual regression supervision with t = 0?

Q2. Does the feature attention fusion in Table 4 retain the uncertainty header and the same LuncL_ {\mathrm{unc}} auxiliary supervision, only removing the −U/2-U / 2 term from the fusion logit? If these supervisions are also removed, the current differences will include both the effects of auxiliary training and fusion rules.

Q3. Does the 800/80/80 data partitioning ensure that the original REDS video or scene is mutually exclusive? Should each learning baseline use public weights or retrain on the same partition?

Q4. Can you provide a scene-based paired bootstrap interval for the PSNR and LPIPS differences between the complete model and SpikeODE/attention fusion?

Reviewer_NBWvOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LTGF, a spike image reconstruction framework that incorporates generative features from a shared flow matching model initialized with TFP and TFI observations. A multi-level uncertainty-guided fusion mechanism selectively integrates generative corrections into a regression-based reconstruction network. Experiments demonstrate improved reconstruction quality on synthetic and real spike datasets.

Critical strengths
  1. The paper is well-motivated, addressing the limitations of regression-based reconstruction in recovering fine details.
  2. The proposed framework achieves competitive performance on both synthetic and real datasets.
  3. The ablation studies demonstrate the effectiveness of combining complementary generative sources and adaptive feature fusion.
Critical weaknesses
  1. The proposed method mainly combines established techniques, including flow matching, uncertainty estimation, and adaptive residual fusion. The technical novelty beyond existing uncertainty-aware feature fusion approaches is not sufficiently clear.
  2. The learnable tolerance may dominate the uncertainty-based weighting, making the proposed mechanism conceptually similar to conventional adaptive gating. The current ablations do not sufficiently isolate the contribution of explicit uncertainty modeling.
  3. Despite achieving the best reported results, the improvement over the strongest baseline is relatively small (e.g., 0.29 dB PSNR over SpikeODE), raising questions about the significance of the proposed methodological complexity.
  4. The effectiveness of the proposed approach under different noise levels, illumination conditions, and motion patterns remains insufficiently explored.
Questions for authors
  1. What is the key methodological novelty compared with existing uncertainty-aware fusion and adaptive gating methods?
  2. Could the authors provide separate ablations for uncertainty-only and tolerance-only fusion to clarify their individual contributions?
  3. How does the predicted uncertainty correlate with the actual reconstruction improvement achieved through feature fusion?
Reviewer_uAnGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LTGF, a framework for spike image reconstruction that uses generative features to improve a regression-based model. A base network first extracts multi-scale features from the spike stream. A shared flow matching model then starts from TFP and TFI intensity estimates to produce complementary restoration features. The proposed fusion module uses multi-level uncertainty estimates and a learned content-adaptive tolerance to weight residual corrections to the base features. The method reports good results on synthetic data, along with strong average no-reference quality scores on real scenes and relatively low computational cost.

Critical strengths

Using generative restoration as an auxiliary source, rather than directly relying on a generated image, is a sensible approach. Keeping the regression pathway as a reference allows the model to recover additional details without giving up the more stable base reconstruction.

The motivation for reliability-aware fusion is clear. Generative models may recover missing textures, but they may also introduce incorrect details. Adjusting the contribution of each candidate across spatial regions is therefore worth exploring.

The performance-efficiency trade-off is encouraging. Compared with SpikeODE, LTGF improves PSNR by 0.2891 dB on the synthetic benchmark while reducing parameters by about 46.4%, FLOPs by 83.7%, and inference time by 63.3%. The source and fusion ablations, together with the uncertainty analysis in the appendix, also provide some support for the full design.

Critical weaknesses
  1. Section 3 contains many equations describing fairly standard feature operations, but gives less space to explaining the key design choices intuitively. Figures 2 and 3 are also quite crowded. It takes some effort to identify the main pipeline and understand which components are essential.

  2. Both standalone FM branches are considerably weaker than the base model in Table 3. Their fusion with the base improves performance, but this does not show that the gain specifically comes from flow-based generative modeling. At inference time, each branch uses only one FM evaluation at t = 0, which looks quite similar to predicting an image correction with a conventional residual restoration network. There is no comparison with such a network with comparable model capacity and training budget, and no analysis of whether more FM steps improve the result. I therefore find it difficult to judge what FM adds over a simpler alternative, or whether the one-step choice is a well-supported quality-efficiency trade-off.

  3. This is my main concern. The fusion involves an uncertainty predictor, a learned content-adaptive tolerance, and a feature correction network. Uncertainty has explicit error supervision, while the tolerance and correction networks are mainly learned through the reconstruction loss. Since the tolerance network already sees the base and candidate features, it may be able to learn when a correction is useful on its own. How much does the explicit uncertainty term actually contribute? Table 4 shows that the full model outperforms feature-attention fusion, but it does not separate the effect of uncertainty from auxiliary supervision and model capacity. An uncertainty-only variant is missing, as is a control that retains the uncertainty head, its training loss, and comparable capacity but removes uncertainty from the fusion logits. The correction network also lacks an independent ablation. The current results show that the full fusion design works, but do not yet establish that the proposed mechanism is genuinely learning which generative features should be trusted, rather than simply performing effective learned feature fusion.

  4. The margin over the strongest baseline is small, and its consistency is unclear. The PSNR gain over SpikeODE is only 0.2891 dB, with no results across multiple random seeds. The real-data evaluation includes just six scenes, and the no-reference metrics cannot directly measure reconstruction fidelity. In fact, based on Appendix Table 7, if the Car scene is excluded, LTGF has a slightly lower average MUSIQ than SpikeODE on the remaining five scenes. More evidence is needed to tell whether the improvement is stable across runs and scenes.

  5. A few implementation details need clarification. Section 3.6 says the uncertainty loss is applied at both the sampled time t and t = 0, but Eq. (19) includes only t for the uncertainty term, whereas Appendix B.2 describes supervision at both times. This matters because inference uses uncertainty predicted at t = 0. It is also unclear how TFI is handled when valid neighboring spikes are unavailable, and how the initial TFP/TFI intensities are normalized in practice.

Questions for authors
  1. Can the authors compare one-step FM with a standard residual restoration network under a similar architecture, parameter count, and training budget? Does increasing the number of FM inference steps improve the candidate images or the final reconstruction?

  2. Can the authors provide a more controlled ablation of the trust mechanism, comparing uncertainty-only fusion, content-adaptive gating alone, and their combination? In particular, what happens if the uncertainty head and its supervision are retained, but the uncertainty prediction is removed from the fusion weights?

  3. How necessary is the feature correction network? Also, can the authors examine whether predicted uncertainty is related to the actual improvement or degradation after a candidate correction is fused, rather than only to the candidate's own reconstruction error?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.