Spike-by-Spike Diffusion: Trajectory-Unrolled ANN-to-SNN Conversion for Anytime Text-to-Image Generation

ANN-to-SNN conversion provides a weight-preserving route to spike-based diffusion inference, but existing methods execute each denoising stage in a stage-major manner: all SNN time steps are completed before advancing to the next stage. This execution order largely removes the temporal flexibility of SNNs, since no terminal image is available until the final denoising stage. We introduce Spike-by-Spike Diffusion, a trajectory-level conversion framework that makes the complete sampling trajectory the unit of temporal refinement. Each global SNN step traverses all denoising stages and produces a terminal latent prefix that can be decoded immediately and further refined within the same run. To support this execution, we reformulate affine diffusion schedulers into incremental updates with one-shot stochastic injection, and introduce a step-sensitive multi-threshold neuron with layer–stage base calibration and stage-specific temporal threshold schedules for accurate early readout. Experiments on Stable Diffusion v1.5 with LCM-LoRA and SANA-Sprint show that our method produces high-quality terminal outputs substantially earlier than stage-major conversion and progressively approaches the ANN reference as computation increases.