acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Reuse of Residual Potentials in Low-Timestep Spiking Diffusion Models

Abstract

Diffusion models generate high-quality samples through many denoising steps, requiring repeated forward passes of large denoising networks and leading to high inference cost. Spiking neural networks (SNNs) offer a promising way to reduce this cost through sparse spike-based computation, but few neuronal timesteps provide only a coarse approximation of activations, causing errors that can propagate to later denoising steps. We observe that residual potentials retain activation information not yet expressed through spikes. Because consecutive denoising steps operate on successive sample states, this information can remain useful across neighboring steps; however, changes in the sample state and diffusion-time conditioning make full inheritance unreliable. Based on this insight, we propose Residual-Adaptive Spiking Diffusion (RASD), which combines selective residual-potential inheritance with residual-informed threshold adaptation. The former uses a calibrated schedule to control how much residual potential is inherited across denoising steps, while the latter forms historical range observations from residual potentials and spike outputs and uses them to predict the current activation range for threshold adaptation.Experiments on DDPM, Improved-DDPM, and DiT show consistent improvements in low-timestep generation quality with only T=4 or T=8 neuronal timesteps, with approximately 65% energy savings.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Diffusion Models, ANN-to-SNN Conversion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kqgKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Residual-Adaptive Spiking Diffusion (RASD) to improve the generation quality of low-timestep spiking diffusion models. The method uses selective residual-potential inheritance to reuse information across consecutive denoising steps, while adjusting the inheritance ratio according to the denoising stage and network location. It also introduces residual-informed threshold adaptation, which uses historical activation information to estimate the activation range and adjust firing thresholds. Experiments on DDPM, Improved-DDPM, and DiT demonstrate improvements in generation quality with low timesteps.

Critical strengths
  1. The motivation of reusing residual potentials across denoising steps is reasonable.

  2. The method is evaluated on different diffusion architectures, including both U-Net-based models and DiT.

Critical weaknesses
  1. TA-SpikingDiT has already explored reusing residual membrane potentials across denoising steps. RASD mainly extends this idea through selective inheritance and threshold adaptation. Therefore, the novelty of this work appears somewhat limited. Moreover, the improvements over TA-SpikingDiT on DiT are relatively small.

  2. The contribution of threshold adaptation is not sufficiently demonstrated, as the paper lacks a clear component-wise ablation study.

  3. The energy analysis only compares RASD with ANNs. Comparisons of inference latency and energy consumption with other spiking diffusion methods are missing.

Questions for authors
  1. What are the actual inference latency and energy consumption of RASD compared with existing spiking diffusion methods?

  2. How could RASD be implemented on neuromorphic hardware.

Reviewer_LuXqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies ANN-to-SNN conversion of diffusion models at very few neuronal timesteps (T=4T = 4 or 88 per denoising step), where spike outputs only coarsely approximate activations and errors propagate along the sampling trajectory. The source of this error lies in the residual potentials, and the current ways of treating residual potentials are essentially either "full inheritance" or "reset to zero". The authors argue that neither is optimal, and propose RASD, which contains two components.

(1) Selective residual-potential inheritance (Sec. 4): by the potential-conservation identity a=y+ra = y + r (Eq. 6 in Sec. 3.2), the residual potential rr retains activation information not expressed through spikes. The paper adopts neither full inheritance, i.e., vk[0]=rk+1v^k[0] = r^{k+1} (Eq. 7), for which the second term on the right-hand side of Eq. 10 becomes unreliable when local responses drift, nor reset, i.e., directly discarding this information; instead, a per-channel, per-location inheritance coefficient γ(λk;Γ)\gamma(\lambda_k; \Gamma) is calibrated offline against the ANN teacher at M=15M = 15 log-SNR control points (Eqs. 12 and 13) and applied as vk[0]=γ(λk;Γ) rk+1v^k[0] = \gamma(\lambda_k; \Gamma)\, r^{k+1} (Eq. 11).

(2) Residual-informed threshold adaptation (Sec. 5): the accumulated input of each completed denoising step is recovered exactly from quantities available after the neuronal timesteps, i.e., a^k=yk+rk−vk[0]\hat{a}^k = y^k + r^k - v^k[0] (Eq. 14); it is then summarized into channel-wise quantile range observations hk=[qk−,qk+]h^k = [q_k^-, q_k^+] (Eq. 17), and a damped two-point linear extrapolation in log-SNR (Eq. 18) predicts the current activation range and sets the threshold as Vk=Rk/TV^k = R^k / T (Eq. 20).

The theory includes the cross-step residual compensation proposition for full inheritance (Prop. 1, Sec. 4.1), the extension of this proposition to selective inheritance (App. A.2), an extrapolation error bound (App. D.2), and a stochastic-branching analysis of inheritance gain (App. B).

Experiments on DDPM and Improved-DDPM (CIFAR-10, 1000-step sampling) and DiT-XL/2-256 (ImageNet 256, 25 steps) show consistent FID/sFID/IS/Precision improvements over no-inheritance, full-inheritance, and adapted TA-SpikingDiT baselines (Table 1), at roughly 35% of ANN arithmetic energy at T=4T = 4 (Table 2).

Critical strengths
  1. Clear motivation, with the implementation aligned to the motivation. The exact recovery a^k=yk+rk−vk[0]\hat{a}^k = y^k + r^k - v^k[0] via potential conservation (Eq. 14; verified numerically to about 10−610^{-6} in App. F.2) and the telescoping-error analysis (Prop. 1 and its selective-inheritance extension in App. A.2) give a principled account of why residual reuse helps and why full inheritance becomes unreliable when the operating point drifts. The fixed-γ\gamma ablation (Table 3) directly supports the design of the calibrated schedule: fixed coefficients γ=0.25,0.5,0.75\gamma = 0.25, 0.5, 0.75 yield no consistent improvement over no inheritance (FID 7.85/7.96/7.94 vs. 7.83), whereas selective inheritance reaches 7.52. The stochastic-branching analysis (App. B) is an honest and solid treatment of an effect that most conversion papers ignore.
  2. Architecture-agnostic empirical gains with credible baselines. Improvements hold for both convolutional (DDPM, Improved-DDPM) and Transformer (DiT) denoisers; the closest prior method, TA-SpikingDiT, is carefully adapted to each architecture; the two components are ablated against fixed-coefficient variants (Table 3); and trajectory-level MSE analyses (Fig. 4) connect the mechanism to the outcome. The authors are also transparent about cases where the baselines are strong (e.g., TA-SpikingDiT on DiT, FID 5.94 vs. RASD 5.91 at T=4T = 4) rather than overselling.
Critical weaknesses
  1. The absolute performance gains are limited. First, I am not familiar with the performance landscape of diffusion-oriented ANN-to-SNN research, but my intuition is that the absolute improvement brought by RASD is limited, which may make the motivation of this work look less "essential". For example, in Table 1, on Improved-DDPM at T=4T = 4, RASD improves FID from 7.83 (no inheritance) to 7.52 while the ANN achieves 5.81, i.e., the converted SNN still trails the teacher by a wide margin; on DiT at T=4T = 4, the FID gap between RASD (5.91) and the adapted TA-SpikingDiT (5.94) is within 0.03, i.e., essentially on par with prior work on the strongest baseline. Finally, in Table 3, compared with the fixed-γ\gamma strategies, the advantage of RASD does not seem very significant either; the authors may need to comment on this.
  2. The key parameter settings required for reproducibility are not documented. RASD relies on many hyperparameter settings: the behavior of threshold adaptation is governed by the quantile levels p−p^- and p+p^+ (Eq. 17), the damping factor ρk\rho^k (Eq. 18), and the clip bounds [Rmin⁡,Rmax⁡][R_{\min}, R_{\max}] (Eq. 55 in App. D.3), while the inheritance schedule is governed by the candidate set G\mathcal{G}, the number of control points MM, and the calibration-set size ∣Ci∣|\mathcal{C}_i| (Eq. 37 in App. C.1). These values do not appear in the main text and are almost impossible to find even in the appendix; for reproduction this is extremely unfriendly. It would be best to provide a dedicated list of the RASD hyperparameter settings.
  3. Counting only computational efficiency is not enough. The headline energy saving of roughly 65% (Table 2) is an arithmetic-operation estimate under the Horowitz 45 nm model (4.6 pJ per MAC, 0.9 pJ per AC, Sec. 6.2); for neuromorphic computing hardware without real large-scale deployment, this metric does not carry much engineering significance at present. Instead, peak memory usage and actual computation latency are crucial for the community, yet the paper does not mention them.
Questions for authors
  1. RASD involves a large number of hyperparameters: Eq. 17 in Sec. 5.1 defines p−p^- and p+p^+ for adjusting the adaptive threshold; Sec. 5.2 predicts the current range by damped extrapolation (Eq. 18, factor ρk\rho^k); App. D.3 clips the predicted range to [Rmin⁡,Rmax⁡][R_{\min}, R_{\max}] (Eq. 55); and App. C.1 states that the inheritance schedule is calibrated over M=15M = 15 log-SNR control points, 6 network groups, and 2 branches, i.e., 180 control values, each selected from a candidate set G⊆[0,1]\mathcal{G} \subseteq [0,1] by minimizing normalized MSE against the ANN teacher (Eq. 37). Are these hyperparameters identical across all experiments? If not, what values of p−p^-, p+p^+, ρk\rho^k, Rmin⁡R_{\min}, Rmax⁡R_{\max}, G\mathcal{G}, and ∣Ci∣|\mathcal{C}_i| were used in the reported experiments, and how sensitive are the results to each? In particular, does a single shared (p−,p+,ρ)(p^-, p^+, \rho) transfer directly to all three models, or are they re-tuned per model?
  2. The inheritance coefficients are calibrated by minimizing the teacher-consistency loss, i.e., normalized MSE between SNN and ANN responses at each control point (Eq. 12 and App. C.1), and the threshold adaptation relies on the exact recovery of Eq. 14 along the actual sampling trajectory (App. D.1 explicitly notes that discrepancies propagated from preceding steps remain in the recovered observation). Does minimizing MSE to the ANN teacher correlate with generation quality (FID)? Specifically, can over-calibrating to the teacher be counterproductive, e.g., inheriting the teacher's own errors, or amplifying the SNN-ANN gap at low log-SNR where the teacher itself is hardest to approximate? How was ∣Ci∣|C_i| chosen, and did you observe calibration-set overfitting, e.g., by varying ∣Ci∣|\mathcal{C}_i|?
  3. Table 2 estimates energy under the Horowitz 45 nm model and reports 35.18% (DDPM) and 35.63% (Improved-DDPM) of ANN arithmetic energy at T=4T = 4, counting only MACs and ACs. RASD additionally requires (i) storing the residual potentials rkr^k for inheritance (Eq. 11) and the range observations hkh^k for prediction (Eq. 18) across all 1000 denoising steps, and (ii) per-step extrapolation and threshold updates. What is the measured end-to-end overhead relative to no-inheritance at T=4T = 4, i.e., the actual computation latency and peak GPU memory? Also, how costly is the offline calibration of the 180-value schedule (in GPU-hours or calibration samples), and is it a one-time computational cost per model?
  4. In Table 1, on DiT-XL/2-256 at T=4T = 4, the performance of adapted TA-SpikingDiT (FID 5.94) and RASD (5.91) is almost identical; while on Improved-DDPM at T=4T = 4, the improvement of RASD over no-inheritance is 7.52 vs. 7.83, with the ANN at 5.81. On DiT, which of the two components (selective inheritance of Sec. 4 vs. threshold adaptation of Sec. 5) is the key factor that actually affects performance? Do the two mechanisms interact, e.g., does threshold adaptation change the calibrated γ\gamma schedule? In other words, would the simpler method of threshold adaptation alone, i.e., without inheritance, already capture most of the gain?
  5. The abstract motivates the method with the sentence: "this information can remain useful across neighboring steps; however, changes in the sample state..." As written, the logic of this sentence is difficult to follow for two reasons. First, "this information" refers to the residual potential rr, i.e., the portion of the accumulated input not expressed through spikes (Sec. 3.2, a=y+ra = y + r), but in its current form it can easily be misread as information about sample-level changes. Second, the "however" contrasts two sides of a single premise (neighboring steps are similar but not identical): reuse is worthwhile because steps are similar, whereas full inheritance fails because steps are not completely identical; spelling out this two-sided nature of the operating-point drift explicitly would make the turn read more naturally. Could you rewrite the abstract (and the corresponding passage in Sec. 1) accordingly?
  6. A few additional suggestions: (i) Could you provide visualizations related to "reset" and "full inheritance" to illustrate the motivation more clearly, in particular the phenomenon that the second term of Eq. 10 in Prop. 1 becomes unreliable? (ii) Some terms are ambiguous and cause reading difficulty, e.g., network location, calibration samples, control point, etc.; could you describe them with more accessible vocabulary, or simply explain them? Note that readers read sequentially and will not cross-check back and forth as a reviewer does. (iii) Could the ℓTC\ell_{\mathrm{TC}} in Eq. 12 be written out explicitly? (iv) γic\gamma_i^{c} and γ(λk;Γ)\gamma(\lambda_k; \Gamma) do not appear to be the same computational construct: the former is selected via the consistency loss, while the latter is an interpolation of the former; in my view the current presentation is not very easy to follow. Could you consider organizing this part of the writing in a better way? Also, does Eq. 13 require the λic\lambda_i^c to be monotonic?
  7. Extension question (not mandatory): can RASD be applied to flow-matching-based methods, e.g., Drifting, MeanFlow, etc.?

Rating justification. Overall, RASD is sound in its motivation analysis and, at the macro level, in its methodology. However, given no small number of hyperparameters, the resulting gains are not evident, which leads me to question, at least to some extent, whether the method has identified the essence of the problem. To be frank, my own research is not on ANN-to-SNN conversion of diffusion models, so my assessment of the performance gains may be biased. That said, I believe other colleagues may hold the same concern. Given ICLR's acceptance bar, I still hope the authors can address this point head-on: for example, stepping outside the authors' perspective, would peers really adopt RASD in their own research? Would they build further improvements on top of RASD? (These questions do not require explicit answers, but I hope the authors will take them into account in the rebuttal.)

Reviewer_B6wGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Residual-Adaptive Spiking Diffusion (RASD) as a method of implementing a spiking neural network (SNN) diffusion model. The RASD model preserves historical information through selective membrane potential inheritance and modulates the firing thresholds of spiking neurons via residual-informed threshold adaptation. Generative experiments on CIFAR-10 and ImageNet indicate that the proposed method strikes a better balance between accuracy and energy consumption.

Critical strengths
  1. This paper points out that inheriting the total membrane potential can easily lead to significant cumulative errors.
  2. The firing threshold of spiking neurons is adjusted dynamically based on residual information.
Critical weaknesses
  1. The core novelty is limited. The reuse of residual membrane potential information and dynamic threshold adjustment have both been previously proposed in SNN work. Therefore, this paper's main contribution lies in modifying existing paradigms rather than introducing significant innovations.
  2. The proposed method is highly complex due to its reliance on offline calibration parameters for selective inheritance, which undermines its generality and effectiveness.
  3. Threshold-adaptive linear extrapolation relies on the assumption that the activation statistics of adjacent denoising steps are locally smooth. However, extrapolation fails when the activation distribution along the denoising trajectory suddenly changes—a common occurrence in real-world task environments.
  4. Ablation experiments are missing. Table 3 only compares selective and full membrane potential reuse. However, it lacks ablation experiments for threshold modulation. Therefore, it is impossible to evaluate the effectiveness and degree of complementarity of each component of the proposed method independently.
  5. This paper does not provide a comprehensive comparison of the proposed method with existing SNN diffusion models. Therefore, it is not possible to assess its true advantages. Furthermore, as shown in Table 1, the proposed method does not significantly outperform the comparison methods, and in some cases, it performs worse.
  6. The proposed method requires storing historical upper and lower percentile statistics for each channel in every layer in order to extrapolate thresholds. This can result in significant overhead, especially for large-scale models. Consequently, deploying this method on neuromorphic chips may not be feasible. Additionally, theoretical power consumption assessments should account for the overhead introduced by the proposed method. Ideally, the authors would provide a comparison of actual hardware overhead.
  7. The figure captions in this paper should be revised to fully explain the phenomena, conclusions, or methodological procedures depicted.
Questions for authors

Please see Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.