acceptodds
Under review as a conference paper at ICLR 2027

SpikeInpaint: Hybrid ANN–SNN Image Inpainting with Ternary Spike Coding

Abstract

Image inpainting is dominated by generators that perform dense floating-point computation throughout the network, incurring substantial computational and energy costs. We investigate whether sparse spiking computation can match the reconstruction quality of dense computation at a lower activation cost. We present SpikeInpaint, a hybrid ANN-SNN generator that places spiking computation only where the spatial resolution is lowest: the four FFC blocks of the bottleneck and the two channel-transition modules on either side of it. The bottleneck combines ternary hard-reset spiking neurons with channel compression, and a channel-wise gate in the decoder fuses the ANN and SNN paths per channel. Experiments on Places2 and CelebA-HQ show that SpikeInpaint achieves 95.89% spike sparsity during inference, within 0.09 dB of an otherwise identical all-ANN generator on Places2. These results show that sparse spiking computation can support dense image generation, extending SNNs to dense generative vision.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Image Inpainting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nuMBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SpikeInpaint, a hybrid ANN-SNN inpainting generator built on an FFC encoder-decoder trained with a GAN objective. Spiking computation is used only in the low-resolution bottleneck. It includes: four FFC ResBlocks whose local and global activations are replaced by TriHardLIF, a single-timestep ternary hard-reset neuron; two SNN Bottleneck modules that compress channels before spiking; a channel-wise gate that blends ANN and ternary features in the decoder.

Critical strengths

Detailed, honest, well-scoped reporting. The paper presents that FID/LPIPS lag behind dense baselines, that sparsity gives no wall-clock speedup on GPUs, and that the 19.3% energy figure is an upper bound. It also includes useful per-branch firing analyses, that makes the actual claim (low PSNR cost of sparsifying the bottleneck relative to a matched dense model) easy to evaluate, although it does not by itself justify acceptance.

Critical weaknesses
  1. The core contribution amounts to ternary activation quantization. Under the single-timestep setting with a reset after every iteration, the neuron keeps no state. TriHardLIF therefore reduces to a simple ternary threshold function: the leak and the reset rule have no effect, and the stated train/inference-mismatch motivation does not apply. Ternary spikes and SNN+FFC hybrids have also been explored before [1, 2]. Without a comparison to an equally quantized ANN, it is unclear what the spiking formulation adds.

  2. The results do not show practical value. Reconstruction quality is clearly below the dense baselines, especially on the perceptual metrics (FID/LPIPS) and on CelebA-HQ. The small PSNR gap is measured against a weak all-ANN reference, and perceptual quality still degrades noticeably after sparsification. The efficiency gains are marginal: most computation remains dense.

[1]Guo, Y., Chen, Y., Liu, X., Peng, W., Zhang, Y., Huang, X., & Ma, Z. (2024, March). Ternary spike: Learning ternary spikes for spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence (Vol. 38, No. 11, pp. 12244-12252). [2]Miao, W., Shen, J., Xu, Q., Hamalainen, T., Xu, Y., & Cong, F. (2025, April). Spikingyolox: Improved yolox object detection with fast fourier convolution and spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 2, pp. 1465-1473).

Questions for authors
  1. At T=1 with a per-iteration reset, do τ\tau and the reset rule have any effect on the forward pass or the gradients? If not, how does TriHardLIF differ from a ternary threshold activation trained with a surrogate gradient?
  2. How does "Signed IF" differ from TriHardLIF at T=1? If the two are functionally equivalent, the 0.36 dB gap suggests that run-to-run variance exceeds the 0.09 dB claim.
  3. Table 3 reports 3.65T operations per image, yet the measured latency is 27.96 ms on an RTX 4070. Please clarify how operations are counted, since the two numbers appear inconsistent with the GPU's peak throughput.
7 comments

We appreciate the precious comments. Here are our comments.

Q1 For the submitted T=1 configuration, neither the leak term nor the reset changes the forward output or the surrogate-gradient path. Eq. (1) gives v[1] = x[1], and the hard reset runs after the only output has already been emitted. The deployed operator is therefore:

s[1] = sgn(v[1]) · 1{|v[1]| > θ},

where 1{·} denotes the indicator function. This is a signed ternary threshold with a silent state. We replaced every TriHardLIF membrane forward in the released checkpoint with:

f(x) = sgn(x) · 1{|x| > θ}

and compared both implementations on a 512$\times$512 Places2 image and its paired mask. The full generator outputs were bit-identical, with maximum and mean absolute differences of 0.0. We therefore make no claim that temporal neuron dynamics account for the reported performance. Within SpikeInpaint, TriHardLIF is a signed ternary activation, and its role is defined at the architecture level by where the signed ternary representation is placed, how it is compressed, and how it is fused with the dense path.

To determine whether the learned ternary mapping can be replaced by an ordinary quantizer or sparsifier at inference, we applied two post-training controls at every TriHardLIF location. INT4 uses symmetric per-image activation quantization. Sparsification preserves the largest activation magnitudes and calibrates each module's threshold to its measured firing rate. Across the same image-mask pairs, INT4 lowers PSNR by 6.52dB and SSIM by 0.0701 relative to TriHardLIF, while sparsification lowers PSNR by 8.93dB and SSIM by 0.1421. These controls use one released checkpoint and are not retrained equal-quantized models. They provide a post-training diagnostic of replacing the learned ternary mapping at inference. They do not establish a retrained equal-quantized baseline, and the architectural claim does not rely on a temporal-neuron advantage.

Hole ratioTriHardLIF PSNRTriHardLIF SSIMINT4 PSNRINT4 SSIMSparsification PSNRSparsification SSIM
0-10%34.010.974024.180.940320.710.9266
10-20%28.110.930620.200.864617.310.8294
20-30%24.770.875417.950.788915.600.7296
30-40%22.330.814516.390.722814.300.6393
40-50%20.390.746715.180.656513.250.5488
50-60%17.590.651814.190.599712.460.4668
Overall24.530.832218.010.762115.600.6901

The corrected operation breakdown separates spike sparsity from the original operation-count estimate. Only 6.2% of all operations are eligible for accumulate execution, and applying the measured firing rate leaves approximately 0.90G executed AC-equivalents. The original 19.3% value used the uncorrected module-level count; for the corrected operation graph, the event-aware compute-only estimate is 0.2%. Most operations remain dense, and the FFT, inverse FFT, complex spectral multiplication, normalization, gate MLP, and dense projections remain floating-point or dense. Memory traffic, buffers, and interconnect are not included in this arithmetic estimate.

The matched latency measurement shows that this sparsity does not produce a GPU advantage in the current implementation. Under the synchronized two-pass protocol, SpikeInpaint has a mean latency of 63.581ms and the matched All-ANN generator has a mean latency of 31.084ms. SpikeInpaint is therefore approximately 2.045×2.045\times slower. The matched quality comparison changes PSNR from 24.50dB to 24.41dB, SSIM from 0.8261 to 0.8222, FID from 10.26 to 12.41, and LPIPS from 0.1718 to 0.1802. We therefore report efficiency only as a compute-level boundary and make no GPU-speedup or measured-system-energy claim. The supported contribution is the architecture-level placement of a sparse signed ternary representation and the measured quality-sparsity cost of bottleneck ternarization. It is not a new temporal neuron, a spiking-specific quality advantage, or a measured system-energy saving.

Q2 SignedIF and TriHardLIF are not equivalent under the submitted T=1 execution. SignedIF is designed to support signed ternary responses through an internal sign state. Under the submitted execution, this state begins inactive and is reset after each emitted output. At initialization, this state is inactive. Starting from v[0]=0, the state-dependent charge correction is zero and v[1]=x[1]. The first emitted output is:

s[1] = H(x[1]-1),

where H is the Heaviside forward function used by the spiking surrogate. Since the sign state has not yet been advanced by an earlier firing event, the negative branch is unavailable and the output takes values in {0,+1}. The corresponding surrogate gradient is:

∂s/∂x = g'(x-1),

where g' is the derivative of the arctan surrogate. The reset changes the stored state after the output, so it does not alter s[1] or its gradient at this timestep.

TriHardLIF uses a symmetric, simultaneous threshold. With v[0]=0 and threshold θ=1.5 in the FFC blocks, it emits:

s[1] = sgn(x[1])·1{|x[1]|>1.5},

which takes values in {-1,0,+1} within the first timestep. Its surrogate path uses both thresholds:

∂s/∂x = g'(x-1.5) + g'(-x-1.5).

SignedIF is therefore designed to represent {-1,0,+1}, but its negative response requires state from an earlier event. At T=1 from the initial state, it acts as a positive-only binary IF activation and can emit only 0 or +1. TriHardLIF immediately applies both signs and emits a ternary output. The two also use different thresholds and different surrogate-gradient expressions.

The 0.36dB difference is not a residual between equivalent operations and cannot be used to estimate run-to-run variation. It compares two single-step activation functions with different thresholds in separately trained models, so it isolates neither the neuron definition nor seed noise. The 0.09dB result is likewise a matched single-run All-ANN comparison, not a seed-variance estimate.

Q3 The reviewer is correct that the submitted operation count and latency cannot be reconciled. We traced the discrepancy to two errors in the submitted version. The 3.65T value came from a static counter that walked the modules in definition order and reused stale shape information across residual and FFC branches. It overcounted the FFC blocks by 16$\times$. We recounted the released checkpoint with forward hooks that read the actual input and output shapes of every convolution and linear layer, covering 69 calls per 512$\times$512 image. The corrected total is 351.5G MAC-equivalent operations per image. Of these, 307.7G are in the dense ANN path (87.5%) and 43.8G are inside the spiking modules (12.5%). The count reports arithmetic opportunities before spike skipping: 21.9G spiking-module operations receive spike tensors and are eligible for AC execution (6.2% of all operations), while the measured 4.11% firing rate corresponds to approximately 0.90G executed AC-equivalents. FFT and inverse FFT, complex spectral multiplication, normalization, and the remaining dense projections remain floating-point or dense. The 27.96ms value came from the legacy process-level CPU timing path, which did not synchronize CUDA and therefore did not measure end-to-end GPU latency. We withdraw 3.65T and 27.96ms as paired hardware-throughput evidence. We re-measured the same 2000 image-mask pairs with a synchronized wall-clock protocol in two passes using forward and reverse image order. Each pass used batch size 1 and 200 excluded warmup forwards. Across the two passes, SpikeInpaint has a mean latency of 63.581ms, while the matched all-ANN generator has a mean latency of 31.084ms. SpikeInpaint therefore incurs an additional 32.497ms, corresponding to approximately 2.045$\times$ the latency of its matched dense counterpart. At the corrected 351.5G count, the SpikeInpaint timing implies approximately 5.53TMAC/s, or 11.06TFLOP/s if one MAC is counted as two FLOPs. This is about 38% of the RTX 4070 FP32 peak and is consistent with a batch-size-1 implementation that also executes FFTs, normalization, nonlinearities, and memory transfers outside the MAC count. The timing result is a consistency check and a hardware measurement; it does not imply that the sparse arithmetic produces a GPU speedup. These corrections also invalidate the submitted 19.3% energy estimate, which was computed from the inflated count. With the corrected operation graph, the compute-only bounds are 10.0% if all 43.8G spiking-module operations are charged as ACs, 5.0% if all 21.9G eligible operations are charged as ACs, and 0.2% when the measured 4.11% firing rate is applied. These values exclude memory traffic, buffers, the NoC, and the floating-point FFT; they are not measured system energy. The revision will report measured latency, operation counts, and compute-only estimates separately. The old 27.96ms, 3.65T, 19.3%, and their derived speedup claims will be marked as superseded and retained only in an audit note. The revised paper will not claim a GPU speedup or measured system-energy saving.

We will correct the SignedIF distinction, operation count, latency and compute-only energy table in the revision, and remove the claims that the corrected results do not support.

Reviewer_ighwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SpikeInpaint, a hybrid ANN-SNN model for image inpainting. Spiking computation is used in the Fourier (FFC) bottleneck, and most of the encoder and decoder stay dense. The method uses a ternary spiking neuron (TriHardLIF), channel compression and a channel-wise gate to reduce spike activity while keeping reconstruction quality. On Places2 it reports 95.89% spike sparsity with only a 0.09 dB PSNR drop compared to a matched all-ANN baseline, and an estimated upper bound of 19.3% energy saving. CelebA-HQ is also used.

Critical strengths

Using SNNs inside an FFC-based generative model is not well explored, and putting spikes only in selected layers is a reasonable choice. The PSNR stays close to the all-ANN model, which is some evidence that dense reconstruction can work with sparse intermediate features. The ablations also help to understand the model. For example, SNNGate Light raises PSNR from 23.97 to 24.41 dB, and the local/global branch analysis shows how firing changes with depth.

Critical weaknesses

My biggest issue is with the neuron itself. Most of the architecture builds on existing parts such as FFC blocks, channel compression and channel-wise gating. The model also uses a single timestep and resets the membrane between inputs, so the decay and hard reset of TriHardLIF have no effect on the output and the neuron is basically a ternary threshold. It is not clear what it adds over a normal ternary activation or quantization, and the ablations do not isolate this.

I am also not convinced by the efficiency side. The 19.3% energy saving is only an analytical upper bound, most operations stay dense, and the FFT still runs in floating point. There is no measured energy or speedup against the matched all-ANN baseline. At the same time perceptual quality gets worse. On Places2, FID goes from 10.26 for the all-ANN model to 12.41 and LPIPS also increases, and the gap to published inpainting methods is even larger. So I am not sure the trade-off is worth it in practice.

Questions for authors
  1. With one timestep and a reset, how does the hard reset change the forward pass compared to a plain ternary threshold? Can you add an ablation that isolates the reset and compares with ternary quantization? A comparison with activation sparsification, pruning or low-bit quantization at a similar compute budget would also show whether the gain comes from spiking or just from sparsity.
  2. How much of the 0.71T operations in the spiking modules can really run as sparse accumulates, given the FFT and other floating point parts of FFC? An operation-level breakdown, plus measured energy and latency against the all-ANN baseline, would make the energy claim much stronger.
  3. Are the results averaged over several seeds, and were the published baselines evaluated on the same images and masks? The PSNR gap to the all-ANN model is small, so the variance matters here.
2 comments

We appreciate your precious comments and these are our rebuttals.

Q1 At T = 1 with v[0] = 0, Eq. (1) gives v[1] = x[1]. The emitted output is: s[1] = sgn(v[1]) · 1{|v[1]| > θ}, where 1{·} denotes the indicator function. The reset runs after s[1] is emitted. It changes neither the forward output nor the surrogate-gradient path, and the leak term is zero at initialization. Therefore, a non-spiking implementation with the same ternary gate and surrogate gradient defines the same operator at T=1. We replaced every TriHardLIF membrane forward in the released checkpoint with: f(x) = sgn(x) · 1{|x| > θ}. As an implementation check, we compared this stateless implementation with the unmodified TriHardLIF generator on a real 512 × 512 Places2 image and its paired mask. For T=1, the full generator outputs were bit-identical: both the maximum and mean absolute differences were 0.0. This equivalence is specific to the submitted single-step setting; for T>1, the reset changes the membrane state carried into the next timestep. A reset-only ablation therefore cannot distinguish the two implementations in the submitted model.

We evaluated two post-training controls: INT4: symmetric per-image 4-bit activation quantization at every TriHardLIF location, with 15 signed levels. Sparsification: magnitude-preserving top-k activation sparsification. A threshold is calibrated per module so that its nonzero rate matches the module's measured firing rate.

Both controls were evaluated on 2,000 Places2 images under six mask ranges. All images were 512$\times$512, and the same image-mask pairs were used for all methods. The TriHardLIF baseline was recomputed on these same image-mask pairs.

Hole ratioTriHardLIF PSNRTriHardLIF SSIMINT4 PSNRINT4 SSIMSparsification PSNRSparsification SSIM
0-10%34.010.974024.180.940320.710.9266
10-20%28.110.930620.200.864617.310.8294
20-30%24.770.875417.950.788915.600.7296
30-40%22.330.814516.390.722814.300.6393
40-50%20.390.746715.180.656513.250.5488
50-60%17.590.651814.190.599712.460.4668
Overall24.530.832218.010.762115.600.6901

Both replacements reduced PSNR and SSIM in every mask range. Relative to TriHardLIF, INT4 lost 6.52 dB and magnitude-preserving sparsification lost 8.93 dB on average. These controls replace the trained ternary mapping at the same insertion points while retaining either 4-bit resolution or the module's measured firing rate. The gap persists across all hole sizes, including 50-60%. Structured pruning changes channel topology and model capacity. It is not an activation-local control, so we exclude it from this comparison. Within the submitted model, these results show a clear cost to replacing TriHardLIF with a conventional quantizer or sparsifier. Within SpikeInpaint, TriHardLIF is a signed ternary activation. These controls provide a post-training diagnostic of replacing that trained mapping at its inference locations. They do not establish a retrained equal-quantized baseline, and the architectural claim does not rely on a temporal-neuron advantage.

Q2 We checked our codes carefully and re-ran it on the same devices. The corrected operation-eligibility count is 21.9G MACs, or 50.0% of the 43.8G operations inside the spiking modules and 6.2% of the 351.5G total. At the measured 4.11% firing rate, this corresponds to approximately 0.90G executed AC-equivalents. The remaining 21.9G spiking-module MACs do not receive spike tensors. The FFT and inverse FFT, complex spectral multiplication, normalization, the gate MLP, and the dense projections also remain floating-point or dense.

The previous 0.71T count is superseded. The original counter traversed module definitions sequentially and reused stale spatial dimensions in the FFC branches, causing a 16× overcount. A forward-hook trace over all 69 convolution and linear calls per image gives 351.5G MACs: 307.7G in the dense ANN path (87.5%) and 43.8G inside the spiking modules (12.5%).

Using 4.6pJ per MAC and 0.9pJ per AC, the dense arithmetic estimate is 1.617J. Charging all 43.8G spiking-module operations as AC gives 1.455J, a 10.0% upper bound. Charging all 21.9G eligible operations gives 1.536J, a 5.0% upper bound. At the measured 4.11% firing rate, 0.90G AC-equivalents execute and the estimate becomes 1.614J, a 0.2% compute-only estimate. These estimates cover arithmetic only. They exclude memory traffic, buffers, the NoC, accelerator utilization, and the floating-point FFT.

Under the synchronized protocol, SpikeInpaint has a two-pass mean latency of 63.581ms, while the matched All-ANN generator has a mean latency of 31.084ms. SpikeInpaint therefore incurs an additional 32.497ms, corresponding to approximately 2.045×2.045\times the All-ANN latency. This comparison shows that the compute-level estimates do not translate into a latency advantage in the current implementation. The 10.0%, 5.0%, and 0.2% values remain compute-only estimates for an event-driven accelerator that skips zero events and executes eligible operations as accumulates.

Q3 The reported results are not averaged over training seeds, and multi-seed averaging is not required for the claims made in this paper. Each architecture and neuron variant was trained from scratch with SEED=10 for approximately 1.018 million iterations. The submission reports one value per architecture and ablation. For image generation models, one complete training run is the standard unit of comparison. The reason lies in both training iterations and the huge amount of dataset. For instance, the Places2 Challenge dataset has more than 8 million images, which is enough for image inpainting. Repeating every architecture and ablation across multiple seeds would require several additional million-iteration runs and would primarily measure residual optimization noise rather than the architecture-level effect studied here. All variants share the same training data, objective, schedule, and evaluation protocol. The six mask-area groups are evaluation base rather than training replicates. The experimental setup and table captions will identify the reported results as single-run measurements.

The matched All-ANN comparison in the paper uses the same evaluation pipeline as SpikeInpaint. This comparison measures the observed quality difference under a matched architecture and training protocol; it does not isolate every causal factor contributing to that difference. The transition from All-ANN to SpikeInpaint produces the expected fidelity cost of this conversion: PSNR changes from 24.50dB to 24.41dB, SSIM from 0.8261 to 0.8222, FID from 10.26 to 12.41, and LPIPS from 0.1718 to 0.1802. The 0.09dB PSNR difference is the measured architecture-conditioned performance difference under this controlled protocol. The FID and LPIPS changes localize the main quality cost to local texture and perceptual statistics rather than global image structure. These values do not arise from an evaluation mismatch between the two models. As a paired evaluation check on the same Places2 outputs, the mean paired PSNR difference is −0.085dB with a 95% bootstrap interval of [−0.099, −0.071]dB; this quantifies evaluation-set uncertainty for the fixed checkpoints and is not presented as seed variance.

SpikeInpaint and all ablation models were trained by us on the same datasets under the same training and evaluation pipeline. The other inpainting methods were executed using their publicly released implementations. All methods are evaluated on exactly the same test images and masks with the same metric implementation, so the quality comparison remains valid across both groups.

We will add these activation controls, clarify the AC-eligible subset and synchronized latency, and state the single-run evaluation protocol in the revised manuscript.

Reviewer_adw4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a hybrid ANN-SNN model for image inpainting with spiking modules in the Fourier bottleneck and the surrounding channel-transition modules. They also introduce a ternary spiking neuron with hard reset and SNN Bottleneck to compress channels before spike generation, and SNNGate Light to fuse the ANN and SNN paths per channel during decoding. The proposed framework has achieved 95.89% spike sparsity during inference of Places2 and CelebA-HQ within 0.09 dB PSNR below an all-ANN generator. The paper also reports estimated MAC/AC energy reduction of up to 19.3%.

Critical strengths

It is a well-motivated design limiting spiking computation to the bottleneck, which is a reasonable way to exploit sparsity without reconstructing the entire pipeline. The architectural choices are aptly justified in the ablations.

Critical weaknesses

The reviewer's understanding is that, TriHardLIF is just a ternary activation at T=1, which is used throughout the inference pipeline. With one timestep and initial membrane potential being 0, the leak and reset do nothing. Hence, the soft-vs-hard-reset argument seems insufficient. And, the gain over Signed IF, as presented in the results, cannot come from neuron dynamics and potentially may arise from training noise. Multi-seed experiments will help rule out this ambiguity. Also, ternary-spike neurons exist in prior work [1]. Furthermore, there seems to be incomplete coverage of prior SNN-based image-inpainting research. It is worth noting that there is existing literature that tackles the same problem [2], which the reviewer believes is directly relevant to the paper’s broad hybrid SNN-ANN inpainting positioning. The manuscript should address this and explain the technical differences and thereby, compare the results. Moreover, the central energy efficiency argument only relies on a theoretical estimate not on neuromorphic hardware or an architectural simulator.

[1] Guo et al., Ternary Spike: Learning Ternary Spikes for Spiking Neural Networks, AAAI, 2024. [2] Sanaullah et al., A Hybrid Spiking-Convolutional Neural Network Approach for Advancing High-Quality Image Inpainting,PMLR (NLDL), 2024.

Questions for authors
  • Please position the proposed work and highlight the key novelty and differences compared to previous research on ternary-spike neurons and hybrid ANN-SNN approach for the image inpainting application?
  • Can you describe the target hardware that can exploit the benefits of the proposed hybrid ANN-SNN architecture?
  • Can you report the energy stats of the presented hybrid architecture on a hardware or an architectural simulator that models the compute units, accumulators, buffers, NOCs, and the memory hierarchy? This is necessary to truly understand the energy benefits of such a hybrid architecture
2 comments

We appreciate the precious comments. Here are our comments.

Q1 We first address the concern about the neuron. At T=1T=1 with v[0]=0v[0]=0, Eq. (1) gives v[1]=x[1]v[1]=x[1], and the output is s[1]=sgn⁡(v[1])⋅1{∣v[1]∣>θ}s[1]=\operatorname{sgn}(v[1]) \cdot \mathbf{1}\{|v[1]| > \theta\}. The hard reset is applied after s[1]s[1] is emitted, so it changes neither the forward output nor the gradient path used to train the threshold, and the leak term is inactive at this timestep. TriHardLIF therefore reduces to a signed ternary threshold with a silent state. The architectural claim does not depend on the neuron-level interpretation.

We will describe TriHardLIF as a signed ternary activation and present the reset as an implementation detail. On the released checkpoint, replacing the membrane dynamics with a stateless signed ternary threshold produces bit-identical outputs: the maximum absolute difference is zero over a 512$\times$512 input. The differences between the neuron variants in Tables 4 and 5 therefore cannot be attributed to neuron dynamics. The current tables report one training run per variant with a fixed seed, and a multi-seed sweep of the 1M-iteration protocol is beyond the revision budget. The revision will state the single-run protocol explicitly and remove the neuron-level interpretation from the abstract, Section 3.2 and Section 5. The architectural claim does not depend on that interpretation.

We dig deep into the mentioned previous work. Guo et al. (2024) develops ternary spikes as a neural primitive for SNN training. Sanaullah et al. (2024) studies hybrid spiking-convolutional inpainting with a shallow 128$\times$128 network trained by reconstruction loss. These works provide the spiking primitive and a low-resolution inpainting baseline.

The contribution of this submission is an architecture-level design that turns the FFC bottleneck into a spiking interface within a dense GAN-based inpainting generator. Spiking is confined to the latent bottleneck, the channel width is reduced before spike generation and restored through a dense residual path, and SNNGate Light learns a channel-specific balance between dense features and sparse ternary corrections. Placement determines which local and global FFC representations are sparsified, compression controls spike-tensor width, and gating limits how strongly ternary features affect reconstruction. Layer-level analysis shows that the resulting firing behavior varies with branch, depth, tensor width, and spatial resolution. The novelty is this joint control mechanism, not an individual neuron or activation.

We found no prior work that studies the joint problem solved here: ternary spiking restricted to an FFC bottleneck, channel compression before spike generation, channel-wise dense-spike fusion, and matched dense evaluation for GAN-based masked image reconstruction. We will add Sanaullah et al. to Related Work and include a protocol-aware comparison. Their model is evaluated at 128$\times$128 with a reconstruction objective, so we will report their setting next to ours and state explicitly how resolution, training objective and mask protocol differ.

Q2 The target is a heterogeneous digital accelerator. Dense floating-point units execute the ANN encoder, decoder, and FFT. An event-driven engine executes the convolution, expansion, and fusion operations whose inputs are signed ternary spikes. Shared scratchpads and an on-chip NoC connect the two domains. This division preserves dense arithmetic where required and enables sparse event skipping and accumulate-only operations in the spiking subgraph.

Relevant candidate execution platforms include SpiNNaker2, whose ARM cores, dedicated DNN and SNN accelerators, event-based routing, and LPDDR4 interface could in principle support a heterogeneous computation split; Tianjic and TianjicX, which support ANN, SNN, and hybrid execution on a unified many-core architecture; Loihi 2, for integer-valued and graded spikes and event-based sensory and spectral processing; and Akida, as a low-power event-driven co-processor. These are candidate targets in principle. We do not claim that the complete SpikeInpaint model has been deployed on them or that any platform provides a measured energy benefit for this model.

SNNs are often described as the third generation of neural networks, and we do not believe their deployment should be restricted to neuromorphic chips. The proposed hybrid model can run on CPUs, GPUs, NPUs, or event-emulation backends. The broad deployment target is therefore any device capable of executing the hybrid model, while the efficiency target is a heterogeneous platform that combines dense floating-point units with an event-driven backend supporting sparse event skipping and accumulate-only operations.

Q3 It led us to a counting error in the submitted version. The 3.65T figure came from a static counter that tracked one triple while walking the module list in definition order. Residual and FFC branches made that tracker stale, the exception was swallowed, and low-resolution convolutions were charged at 128$\times$128 or 256$\times$256 instead of their true sizes. The error is systematic: the FFC blocks were overcounted by 16x.

We re-counted the same checkpoint with forward hooks that read the actual input and output shapes of every convolution and linear layer, 69 calls per image. The corrected total is 351.5G MAC-equivalent operations per 512$\times$512 image. Of these, 307.7G are in the dense ANN path (87.5%) and 43.8G are inside the spiking modules (12.5%). The count reports arithmetic opportunities before spike skipping: 21.9G spiking-module operations receive spike tensors and are eligible for AC execution (6.2% of all operations), while the measured 4.11% firing rate corresponds to approximately 0.90G executed AC-equivalents. FFT and inverse FFT, complex spectral multiplication, normalization, and the remaining dense projections remain floating-point or dense.

The 27.96ms value came from the legacy process-level CPU timing path, which did not synchronize CUDA and therefore did not measure end-to-end GPU latency. We withdraw 3.65T and 27.96ms as paired hardware-throughput evidence. We re-measured the same 2000 image-mask pairs with a synchronized wall-clock protocol in forward and reverse order. Each pass used batch size one and 200 excluded warmup forwards. Across the two passes, SpikeInpaint averages 63.581ms and the matched all-ANN generator averages 31.084ms, an additional 32.497ms or approximately 2.045×2.045\times. At the corrected 351.5G count, the implied throughput is 5.53TMAC/s, or 11.06TFLOP/s when one MAC is counted as two FLOPs, about 38% of the RTX 4070 FP32 peak. This is a consistency check; FFTs, normalization, nonlinearities, and memory transfers remain outside the MAC count.

Using 4.6pJ per MAC and 0.9pJ per AC, an all-MAC execution costs 1.617J. Charging all 43.8G spiking-module operations as AC gives 1.455J, a 10.0% compute-only upper bound; charging all 21.9G eligible operations gives 1.536J, a 5.0% upper bound. Applying the measured 4.11% firing rate yields 0.90G executed AC-equivalents and 1.614J, a 0.2% compute-only estimate. These estimates exclude memory traffic, buffers, the NoC, accelerator utilization, and FFT/iFFT work, so they are not measured system energy. The submitted 19.3% estimate and old latency-derived speedup claims are superseded and retained only in a revision-audit note.

We will revise the manuscript to treat TriHardLIF as a signed ternary activation, present the heterogeneous accelerator as a target rather than a measured platform, and replace the operation and energy estimates with the corrected compute-only accounting.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal

Dear PC, SAC, and AC,

Thank you all for reviewing our submission. However, we only find reviewer ighw replying to our rebuttal, but still keep a 1 rating. Other two reviewers still got no response. We feel sad for the no response as they cannot offer a reasonable review. We have updated and revised our submission and abstract. Please refer to any of the update and carefully decide the final result of this paper, even if currently all the reviews are negative.

Best regards, Authors of Submission 12079

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.