acceptodds
Under review as a conference paper at ICLR 2027

Spike-by-Spike Diffusion: Trajectory-Unrolled ANN-to-SNN Conversion for Anytime Text-to-Image Generation

Abstract

ANN-to-SNN conversion provides a weight-preserving route to spike-based diffusion inference, but existing methods execute each denoising stage in a stage-major manner: all SNN time steps are completed before advancing to the next stage. This execution order largely removes the temporal flexibility of SNNs, since no terminal image is available until the final denoising stage. We introduce Spike-by-Spike Diffusion, a trajectory-level conversion framework that makes the complete sampling trajectory the unit of temporal refinement. Each global SNN step traverses all denoising stages and produces a terminal latent prefix that can be decoded immediately and further refined within the same run. To support this execution, we reformulate affine diffusion schedulers into incremental updates with one-shot stochastic injection, and introduce a step-sensitive multi-threshold neuron with layer–stage base calibration and stage-specific temporal threshold schedules for accurate early readout. Experiments on Stable Diffusion v1.5 with LCM-LoRA and SANA-Sprint show that our method produces high-quality terminal outputs substantially earlier than stage-major conversion and progressively approaches the ANN reference as computation increases.

Primary area:
generative models
Keywords:
Image Generation, Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_peuhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Spike-by-Spike Diffusion, which replaces stage-level execution with trajectory-level execution in SNN for Diffusion model. It makes each SNN step pass through all denoising stages and refines the output in the same inference process. It also designs an incremental scheduler to ensure the consistency in the accumulation of increments across stages. It further proposes a step-sensitive multi-threshold neuron and a two-stage calibration method to improve generation accuracy without modifying pretrained weights.Experimental results shows that this method offers advantages in both final generation quality and energy consumption.

Critical strengths
  1. The trajectory-level SNN execution proposed by Spike-by-Spike Diffusion solves the problem of SNN diffusion models being unable to perform terminal image readout before the last timestep by making the complete diffusion trajectory the unit of temporal accumulation.

  2. Based on the property that, in standard diffusion sampling, a single-stage latent variable update can be expressed in affine form, he designed an SNN-compatible incremental scheduler that allows latent increments to propagate during the denoising stage to support trajectory-level SNN execution. He also designed step-sensitive multi-threshold neurons and, through Layer–stage-based and temporal sensitivity calibration, improved the model’s generative accuracy during early SNN time steps.

Critical weaknesses
  1. Layer–Stage Base Calibration requires running the SNN diffusion inference multiple times and then comparing the reconstruction errors to determine the coefficient kℓ,dbasek_{\ell,d}^{\mathrm{base}}; Temporal Sensitivity Calibration uses CMA-ES search to find the thresholds α\alpha for different diffusion stages. Since there are many parameters to calibrate, these two calibration methods may incur a high computational cost. Furthermore, the calibration datasets and the number of calibration samples are not not fully described.

  2. As mentioned in Appendix B.5, during each global SNN step, all denoising phases must advance and retain a separate SNN state SdS_d for each stage, while the parameters introduced by the two calibration methods must also be retained during the inference process. However, the paper does not quantify the size and precision of these retained states.

Questions for authors
  1. How much additional overhead do layer–stage base calibration and temporal sensitivity calibration introduce?

  2. What is the peak inference memory for the D=2D=2 and D=4D=4 configurations compared with stage-level execution? Could the authors clarify how they scale with DD and TT?

  3. In Table 1, why did the converted SNN perform slightly better than the ANN reference in multiple sets of FID/sFID tests? Is this statistically significant?

Reviewer_HnF1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces trajectory-interleaved ANN-to-SNN conversion for text-to-image diffusion. Each global SNN step traverses all denoising stages, allowing intermediate images to be decoded and refined without restarting. Incremental affine schedulers and stage-dependent threshold calibration support this execution. Experiments on Stable Diffusion v1.5 and SANA-Sprint evaluate generation quality, progressive readouts, and estimated computational costs.

Critical strengths
  1. Interleaving computation across the full trajectory enables reusable intermediate outputs within one run. This is a meaningful capability beyond improving final-output conversion accuracy.
  2. Results across two model families consistently improve on the adapted SpikingDiT baseline. On SANA-Sprint with two denoising stages, the final FID of 6.82 closely matches the ANN reference of 6.79.
Critical weaknesses
  1. The incremental treatment of attention and normalization is insufficiently specified. Appendix B.1 establishes scheduler consistency, but full trajectory equivalence remains conditional on denoiser-output correspondence.
  2. Important calibration details are missing, including sample counts, separation from evaluation data, the base-threshold search grid, and numerical CMA-ES settings. These omissions hinder reproduction of a central component.
  3. All A0–A5 ablations use interleaved execution. Without a stage-major baseline receiving comparable calibration, the quality gains cannot be cleanly attributed to the execution order.
  4. Efficiency evidence relies on operation counts and fixed per-operation energy estimates. End-to-end latency, the memory cost of retaining separate stage states, and preview decoding costs are not established.
Questions for authors
  1. How are attention and normalization handled under incremental inputs, and which operations remain dense?
  2. What calibration data and numerical search settings were used, and were calibration samples disjoint from evaluation samples?
  3. Does the quality advantage persist when the stage-major baseline receives the same applicable range and threshold calibration?
1 comment

As no responses are found, I still tend to reject the submission.

Reviewer_CK8tOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper replaces stage-major ANN-to-SNN diffusion execution with trajectory-level execution. Each SNN step traverses all denoising stages, so an image can be read out after every step and refined within a single run. It also adds an incremental affine scheduler and a multi-threshold neuron with stage-specific and time-varying thresholds. On SD v1.5 and SANA-Sprint, it produces usable images earlier than SpikingDiT and reaches final FID close to the ANN.

Critical strengths
  1. Identifies a real limitation of stage-major spiking diffusion and fixes it with a simple, weight-preserving reordering that enables single-run anytime generation.

2 Consistent gains over SpikingDiT across two model families, two datasets, and several (D, n) settings, with budget-matched comparisons and informative calibration ablations.

Critical weaknesses
  1. The value of ANN-to-SNN conversion is not shown. Why do we need such A2S method? There are no firing rates, no absolute energy comparison with the ANN or quantized baselines. Energy appears only as within-SNN fractions. Reaching ANN-level quality takes eight full traversals and still trails single-pass W8A8, and the graded multi-level events look closer to residual quantization than to sparse spiking. The work therefore shows only that one SNN beats another, not that conversion beats the ANN. The anytime property is also available from fewer-stage or early-exit (quantized) ANNs, since consistency samplers output at every stage, and this baseline is not evaluated on energy.

  2. Accuracy gains do not come from the core contribution. The ablation (Table 4) indicates that the improvement over SpikingDiT comes mainly from stage-aware base calibration. Trajectory-level execution, the paper's central contribution, is worse than SpikingDiT on its own: A0 reaches a final FID of 44.16 versus 34.30. A0→A3 accounts for most of the gain (FID 44.16→27.01). This calibration seems equally applicable to stage-major execution, yet no stage-major + proposed-calibration control is provided. The core contribution therefore adds an anytime-readout capability rather than better accuracy, and that capability is itself also available from fewer-stage or early-exit baselines, since consistency samplers output x0x_0 at every stage.

Questions for authors

See weaknesses

Reviewer_o6QiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies ANN-to-SNN conversion of few-step text-to-image diffusion models. Prior work (SpikingDiT) finishes all TT SNN steps of one denoising stage before moving to the next. The authors instead advance all DD stages once per SNN step and pass the increments through the affine LCM/sCM scheduler (Eqs. 5–7), so a terminal image can be decoded after every SNN step and refined within the same run. They also calibrate a multi-threshold neuron whose threshold depends on the stage, layer, channel and SNN time step. On SD v1.5 + LCM-LoRA and SANA-Sprint-0.6B, the method beats SpikingDiT at all budgets, e.g., FID 21.84 vs. 66.11 after one SNN step and 6.96 vs. 7.87 after eight (SANA-Sprint, D=4D=4).

Critical strengths
  • Swapping the stage loop and the SNN-time loop is a simple observation, and the incremental scheduler is exact for affine schedulers, with derivations for both LCM and sCM (App. B.1).
  • Against SpikingDiT, the method wins in all 24 budget–configuration pairs (Fig. 11).
Critical weaknesses
  • The motivation does not hold, and the anytime comparison is against a strawman.
    • The paper claims that stage-major execution gives no terminal image until the final stage (Abstract; Sec. 3.1). This is not true for the samplers used here. LCM and sCM compute a clean estimate z~0,d\tilde z_{0,d} at every stage (Eqs. 33 and 42), which a stage-major SNN can decode after each stage, or even after each SNN step with the prefix readout of App. A.1.
    • SANA-Sprint is a one-step model (FID 7.04 at one step vs. 6.48 at four [1]). Decoding z~0,1\tilde z_{0,1} should already be close to one-step ANN quality after 8 denoiser updates (D=4D=4, T=8T=8). At the same cost, the proposed method is still at FID 11.39 (t=2t=2).
    • The paper's own numbers do not help. D=4D=4, where all headline gains are reported, is dominated by D=2D=2 of the same method at every matched number of denoiser updates (8: 7.21 vs. 11.39; 16: 6.82 vs. 7.80). At D=1D=1, the two execution orders are identical.
    • The 96–99% number in Fig. 2(a) is measured at D∈{25,50,100}D\in\{25,50,100\}, which is never evaluated.
  • No evidence that the SNN is worth running, and the conversion is underspecified.
    • The ANN's energy is never reported under the same accounting. My rough estimate for SANA-Sprint-0.6B at 1024² (28 blocks, width 1152, 1024 tokens [2]) is ≈0.55 TMACs per forward, i.e., ≈2.5 J at 4.6 pJ/MAC, or ≈5 J for D=2D=2 and ≈10 J for D=4D=4.
    • Fig. 6(d) reports 4.95–6.18 J (D=2D=2) and 10.73–13.14 J (D=4D=4) for T=8T=8. Even under the 45 nm MAC/AC model, which already favors SNNs, the converted model is at best on par with the dense ANN.
    • Fig. 6(b) implies ≈0.5×10¹² ACs per denoiser update, i.e., almost no sparsity. Counting graded events with stage-, time- and channel-dependent magnitudes as plain ACs is also optimistic.
    • Sec. 4 only describes the neuron. How attention, normalization, GELU/SiLU/GLU, products of two dynamic tensors and cross-attention are computed on non-stationary increments is never stated. This is exactly where the correspondence assumed in Eq. 28 can break; App. B.1 only proves that the scheduler is linear.
  • Minor: Tables 1–2 are shrunk to a barely readable font; "Treshold" in Fig. 4. [1] Chen, Junsong, et al. "Sana-sprint: One-step diffusion with continuous-time consistency distillation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025. [2] Xie, Enze, et al. "Sana: Efficient high-resolution image synthesis with linear diffusion transformers." The Thirteenth International Conference on Learning Representations. 2025.
Questions for authors
  1. Please add stage-major SpikingDiT with z~0,d\tilde z_{0,d} readouts and a D=1D=1 SNN to Fig. 6(b), plotted against cumulative energy.
  2. Please report the energy of the ANN and the W8A8/W4A8 baselines under the same accounting. Which operations are spiking and which are dense?
  3. On SD v1.5, the SNN beats its own ANN (FID 26.75 vs. 27.78), and sFID gets worse from t=6t=6 to t=8t=8 (56.67 → 59.98; ANN 61.84). FID on 5K/10K samples cannot resolve conversion fidelity. Please report LPIPS/PSNR against ANN outputs with matched seeds, as SVDQuant [1] does.
  4. Which prompts, and how many, are used for calibration? Fig. 10 shows separate αd(t)\alpha_d(t) for COCO2014 and COCO2017, which suggests calibrating on each test set.
  5. Please give Kbase\mathcal{K}_{\text{base}}, ωt\omega_t, wℓ,dw_{\ell,d}, λsm\lambda_{\text{sm}}, the CMA-ES budget and the calibration time. [1] Li, Muyang, et al. "Svdquant: Absorbing outliers by low-rank component for 4-bit diffusion models." The Thirteenth International Conference on Learning Representations. 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.