acceptodds
Under review as a conference paper at ICLR 2027

Potential–Time Dual Representations for TTFS Transformer Conversion

Abstract

Time-to-first-spike (TTFS) coding represents each value by a single spike time, reducing the number of spikes required to represent an activation. Converting a pretrained Transformer into a TTFS-based continuous-time spiking neural network (SNN), however, requires composing Transformer operations from analog-compatible operators and evaluating the converted model under timing noise. We propose a conversion framework based on the dual relationship between membrane-potential space and spike-time space . The framework composes a small set of fixed primitive operators with known hardware realizations to implement affine transformations, attention, LayerNorm/RMSNorm, GELU, and SwiGLU. Across pretrained ViT, RoBERTa, GPT-2, and Llama 2 7B models, conversion reduces classification accuracy by at most pp (one fewer correct SST-2 prediction out of 872), while the absolute WikiText-2 perplexity change is below . We model spike-time noise and the resulting deadline misses, parameterizing the noise with BrainScaleS-2 measurements from the physical paths implementing the linear and logarithmic operators. Code is available at https://anonymous.4open.science/r/52AD.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Network, Time-to-first-spike, ANN-to-SNN, Neuromorphic, Analog

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_QwkmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper converts pretrained Transformers into TTFS-SNNs. It defines two encoders from potential to spike time and three decoders from spike time back to potential. Composing them gives multiplication, division, exponentiation, Softmax, normalization, GELU and SwiGLU. Without noise, the converted models (ViT up to Llama-2-7B) lose at most about 0.1% accuracy. The paper also injects timing noise scaled from BrainScaleS-2 measurements.

Critical strengths
  • This paper proposes a clear set of primitives.
  • Evaluate up to 7B.
  • The time noise is based on hardware measurements and negative results are reported openly.
Critical weaknesses
  • The declared ranges require a very large dynamic range. The GELU exponent input is [-80, 80], so E=e−zE=e^-z is in $
e−80,e80e^{-80}, e^80

$. Since windows and gains follow the declared range, every evaluation uses a 160-unit window and an e80e^{80} gain, which float64 handles but analog hardware cannot.

  • With the measured noise, all classifiers fall to chance. The paper does not explain why of propose any fix. Noise grows with window width, and the windows come from min–max calibration, so outliers are a likely cause. Clipping, per-channel ranges or noise-aware calibration should be tried.
  • The noise measurement is thin. Only 4 of 63 NP circuits passed the selection criteria, and those criteria were relaxed after looking at the data. The assumption that noise scales linearly with window width is not tested.
Questions for authors
  • Please report per-operator dynamic ranges and test a smaller clip.
  • Does outlier-robust or noise-aware calibration change the α\alpha needed to keep accuracy?
  • Is the linear noise scaling supported by measurements at several window widths?
5 comments

We appreciate your close look at the dynamic range required by the GELU exponential. Your suggestions led us to test a narrower clip for its input and examine whether percentile calibration could improve robustness to timing noise.

Question 1

Please report per-operator dynamic ranges and test a smaller clip.

Response

Table 1 compares the observed ranges under the original [−80,80][-80,80] clip and the narrower [−10,5][-10,5] clip. We varied the lower and upper bounds of the exponential input within GELU independently. Table 2 reports accuracy across the sweep. The [−10,5][-10,5] setting reduced the GELU exponential time window from 160 to 15 and preserved all predictions on the 500 validation images. With the [−10,2.5][-10,2.5] clip, accuracy decreased from 86.4% to 85.8%, a drop of 0.6 percentage points.
 

Table 1. Observed operator ranges with the original and narrower GELU exponential input clips

Each interval gives the observed interval across the indicated operator calls on 500 validation images.

Operator and quantityObserved [−80,80][-80,80]Observed [−10,5][-10,5]
ϕNP\phi_{\mathrm{NP}} input potential[−200,126][-200,126][−200,126][-200,126]
ϕNP\phi_{\mathrm{NP}} output spike time[0,729][0,729][0,729][0,729]
ϕNL\phi_{\mathrm{NL}} input potential[10−5,5.54×1034][10^{-5},5.54\times10^{34}][10−5,2.20×104][10^{-5},2.20\times10^{4}]
ϕNL\phi_{\mathrm{NL}} output spike time[0,80][0,80][0,50][0,50]
Explicit ψInt\psi_{\mathrm{Int}} output potential[−206,1.03×104][-206,1.03\times10^{4}][−206,1.02×104][-206,1.02\times10^{4}]
ψED\psi_{\mathrm{ED}} output potential[1.80×10−35,26.0][1.80\times10^{-35},26.0][2.01×10−22,26.0][2.01\times10^{-22},26.0]
fMulf_{\mathrm{Mul}} output potential[−206,1.03×104][-206,1.03\times10^{4}][−206,1.02×104][-206,1.02\times10^{4}]
fDivf_{\mathrm{Div}} output potential[1.80×10−35,1][1.80\times10^{-35},1][4.54×10−5,0.993][4.54\times10^{-5},0.993]
GELU cubic output potential[−3.41×106,6.53×105][-3.41\times10^{6},6.53\times10^{5}][−3.50×106,6.56×105][-3.50\times10^{6},6.56\times10^{5}]
GELU fExpf_{\mathrm{Exp}} input potential after clipping[−80,80][-80,80][−10,5][-10,5]
GELU fExpf_{\mathrm{Exp}} output potential[1.80×10−35,5.54×1034][1.80\times10^{-35},5.54\times10^{34}][6.74×10−3,2.20×104][6.74\times10^{-3},2.20\times10^{4}]
ϕNL\phi_{\mathrm{NL}} output spike time within GELU division[0,80][0,80][0,10][0,10]
fGELUf_{\mathrm{GELU}} output potential[−0.170,86.8][-0.170,86.8][−0.170,86.3][-0.170,86.3]
LayerNorm output potential[−200,126][-200,126][−200,126][-200,126]
Linear projection output potential[−150,86.8][-150,86.8][−152,86.9][-152,86.9]
Patch projection output potential[−7.96×100,7.19×100][-7.96\times10^{0},7.19\times10^{0}][−7.96×100,7.19×100][-7.96\times10^{0},7.19\times10^{0}]

We report attention separately because a common shift of the scores in each row can rescale the exponential output range while preserving the Softmax weights in exact arithmetic. In both runs, the largest exponential value in attention was 1.84×10471.84\times10^{47}, against a declared upper bound of 9.69×10729.69\times10^{72}.

 

Table 2. Top-1 accuracy (%) across the GELU exponential input clips

Rows and columns give the lower and upper bounds, respectively.

Lower bound / Upper bound2.52.5551010202040408080
−2.5-2.50.20.20.20.20.20.20.20.20.20.20.20.2
−5-574.274.273.673.673.273.273.273.273.273.273.273.2
−10-1085.885.886.486.486.486.486.486.486.486.486.486.4
−20-2085.885.886.486.486.486.486.486.486.486.486.486.4
−40-4085.885.886.486.486.486.486.486.486.486.486.486.4
−80-8085.885.886.486.486.486.486.486.486.486.486.486.4

Question 2

Does outlier-robust or noise-aware calibration change the α needed to keep accuracy?

Response

On ViT-B/16, we compared mean top-1 accuracy across three matched noise seeds. Percentile calibration of the two MLP projection inputs increased the largest tested noise multiplier that met our 85.4% mean accuracy target from α=0.003\alpha=0.003 to 0.010.01. The target was one percentage point below the noise-free min–max accuracy. At α=0.01\alpha=0.01, the 0.01% cutoff per tail gave 85.87% accuracy versus 85.27% for min–max. The paired 95% confidence interval for the difference was [−0.39,1.59][-0.39, 1.59] percentage points.

We changed the calibration bounds for one group at a time using training percentiles. All four groups improved accuracy at α=0.10\alpha=0.10, while LayerNorm calibration caused the largest loss in noise-free accuracy.

 

Table 1(a). ViT-B/16 top-1 accuracy (%) with a 0.01% cutoff per tail

α\alphaMin–maxAttentionLayerNormMLP projection inputsGELU inputs
0086.486.486.286.274.074.085.885.886.486.4
0.010.0185.685.685.885.874.274.286.286.285.285.2
0.0750.07553.453.464.264.249.449.474.074.065.865.8
0.100.1013.413.439.239.225.425.453.053.034.634.6

Table 1(b). ViT-B/16 top-1 accuracy (%) with a 0.1% cutoff per tail

α\alphaMin–maxAttentionLayerNormMLP projection inputsGELU inputs
0086.486.485.285.264.264.284.284.286.486.4
0.010.0185.685.685.085.064.864.884.084.085.685.6
0.0750.07553.453.465.665.633.633.673.273.268.268.2
0.100.1013.413.439.839.814.814.849.849.839.639.6

Table 1 reports seed 0 on 500 ImageNet-1k validation images.

We then applied the 0.01% cutoff jointly to attention, GELU inputs, and MLP projection inputs. The gains at intermediate noise levels were most pronounced for ViT. CCT-7 and RoBERTa-B showed smaller accuracy gains. GPT-2’s perplexity decreased at some higher noise levels, where it remained high, and increased without noise.

 

Table 2. Performance with joint calibration and change relative to min–max at the same α\alpha

α\alphaViT-B/16ViT-S/16CCT-7RoBERTa-BGPT-2
0085.6085.60% (−0.80-0.80 pp)81.1081.10% (−1.18-1.18 pp)96.5096.50% (−0.03-0.03 pp)93.6993.69% (−0.69-0.69 pp)31.1331.13 (+41.60+41.60%)
0.0030.00385.5385.53% (−0.73-0.73 pp)81.1181.11% (−1.05-1.05 pp)96.4996.49% (−0.02-0.02 pp)93.6293.62% (−0.57-0.57 pp)31.2731.27 (+41.65+41.65%)
0.010.0185.3385.33% (+0.07+0.07 pp)80.9580.95% (−0.45-0.45 pp)96.4896.48% (−0.03-0.03 pp)93.0093.00% (−0.15-0.15 pp)32.6732.67 (+41.72+41.72%)
0.030.0384.8784.87% (+4.87+4.87 pp)80.4780.47% (+7.09+7.09 pp)96.4296.42% (0.000.00 pp)86.4786.47% (+1.61+1.61 pp)48.0948.09 (+24.67+24.67%)
0.050.0583.1383.13% (+10.93+10.93 pp)79.4179.41% (+18.05+18.05 pp)96.3896.38% (+0.04+0.04 pp)71.2571.25% (+5.66+5.66 pp)116.15116.15 (−51.38-51.38%)
0.0750.07581.9381.93% (+30.80+30.80 pp)76.4776.47% (+33.63+33.63 pp)96.2796.27% (+0.02+0.02 pp)57.0357.03% (+2.94+2.94 pp)941.20941.20 (−86.22-86.22%)
0.100.1078.3378.33% (+63.53+63.53 pp)71.1571.15% (+48.44+48.44 pp)96.0996.09% (+0.05+0.05 pp)52.9152.91% (+1.41+1.41 pp)20,942.0320{,}942.03 (−87.61-87.61%)
0.300.300.400.40% (+0.40+0.40 pp)1.121.12% (+1.01+1.01 pp)88.8088.80% (+2.19+2.19 pp)50.8450.84% (+0.69+0.69 pp)1.64×10171.64\times10^{17} (+63.63+63.63%)
11—0.110.11% (0.000.00 pp)11.7111.71% (+1.53+1.53 pp)50.3450.34% (+0.08+0.08 pp)2.07×10642.07\times10^{64} (+658.21+658.21%)

*Classifier columns report accuracy and percentage-point changes. GPT-2 reports token-weighted perplexity and its relative percentage change, with lower values indicating better performance. Nonzero α\alpha rows report three-seed means.

Question 3

Is the linear noise scaling supported by measurements at several window widths?

Response

On BrainScaleS-2, we measured the standard deviation of the first spike time at several time window widths, using three circuits for each of the ϕNP\phi_{\mathrm{NP}} and ϕNL\phi_{\mathrm{NL}} operators. For each circuit, we fit σt=aT^p\sigma_t=a\widehat{T}^p by minimizing the mean squared error between the measured standard deviations and the fitted values, where T^\widehat{T} is the fitted signal span. The mean fitted exponent across circuits was 1.28 (range 1.07–1.44) for ϕNP\phi_{\mathrm{NP}} and 1.51 (range 1.43–1.55) for ϕNL\phi_{\mathrm{NL}}. For one ϕNP\phi_{\mathrm{NP}} circuit, fitting the coefficient with p=1p=1 gave nearly the same mean squared error as fitting both parameters. All three ϕNL\phi_{\mathrm{NL}} circuits had fitted exponents above 1.

The linear scaling used in our original experiments underestimates spike-time noise at wider time windows, as observed in circuits for both ϕ\phi operators. In practice, shorter time windows can help mitigate this noise, although achieving them while preserving the operator response may incur additional circuit costs.

Reviewer_UNQEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a way to convert pretrained Transformers into continuous-time TTFS spiking networks without retraining. Its organizing idea is that a value in this computing style exists in two forms, stored as a membrane potential and transmitted as a spike time, and that alternating between the two forms with different time constants is itself a source of nonlinearity. The authors reduce the interface to a small fixed set of primitives and show that compositions of them implement multiplication, division, exponentiation, powers, GELU and scaled dot-product attention, from which attention, LayerNorm, RMSNorm and SwiGLU are built. Physical implementations have finite potential ranges and observation windows. The conversion calibrates a potential range per layer and sets a matching spike-time window, using analytic bounds where they exist and a positive floor for the logarithmic operator. Evaluated on ViT, CCT, RoBERTa, GPT-2 and Llama 2 7B, the converted models match their source networks in continuous-time double-precision simulation, with classification differences within a single test sample and perplexity differences in the fourth decimal place. A separate analysis injects Gaussian timing noise, with its scale fitted to measurements on two BrainScaleS-2 paths. It models deadline misses and reports how accuracy degrades as the noise scale and the number of perturbed blocks increase.

Critical strengths
  1. The composition principle is economical, and it is carried through to the architectures people actually use daily. Alternating between the two representations with mismatched time constants produces a cubic nonlinearity without a purpose-built neuron. From that starting point the authors build a fixed operator set that covers RMSNorm and SwiGLU as well. The same set covers ViT, RoBERTa and Llama 2 under one construction, whereas earlier TTFS work implemented the nonlinearities with purpose-built current kernels.
  2. The conversion-fidelity result is close to a consistency check, and the parts of the pipeline that can lose accuracy are not separated. Section 5.1 states that the operator identities hold exactly within their stated domains, and the operators were built to compute exactly those functions. A near-zero difference between the ANN and the converted model mainly confirms that the implementation matches the specification. The finite-domain step is where accuracy can go missing, and the same section notes that the chosen bounds set the clipping boundaries and the spike-time windows. The cubic tanh form of GELU in Table 2 adds a second approximation on that path.
Critical weaknesses
  1. The fidelity claim is measured in an ideal simulator, and the experiment grounded in hardware measurements reports failure. The abstract states that conversion reduces classification accuracy by at most 0.11 percentage points. In the timing-noise experiment whose scale is fitted to BrainScaleS-2 measurements, all four classifiers approach chance accuracy, two of them below 0.2%. Section 6.4 qualifies this as a sensitivity analysis under a scaling assumption, and the conclusion defers hardware execution to future work. Neither qualification appears in the abstract or the introduction.
  2. The conversion-fidelity result is close to a consistency check, and the parts of the pipeline that can lose accuracy are not separated. Section 5.1 states that the operator identities hold exactly within their stated domains. The operators were built to compute those functions, and a near-zero difference between the ANN and the converted model mainly confirms that the implementation matches the specification. Accuracy can go missing in the finite-domain step, where the chosen bounds set the clipping boundaries and the spike-time windows. Table 2 introduces a cubic tanh substitute for GELU on the same path. That substitution and the clamping appear in the same reported difference, and no measurement separates the cost of the approximation from the cost of the bounds.
Questions for authors
  1. Which condition does the paper intend a reader to take as its central claim? If the contribution is the operator construction and its exactness within declared domains, would you move the measured-noise result into the abstract and lead with the sensitivity finding?
  2. Could the reported difference be decomposed into three numbers: conversion with the exact GELU and no clamping, conversion with the cubic GELU, and the additional cost of clamping? That would separate what is exact by construction from what the finite ranges and the approximation cost.
  3. The noise model reduces to a single scalar that scales two measured fractions while preserving their ratio, with the standard deviation assumed proportional to the local time window. How sensitive is the near-chance result at the measured scale to that scaling law? A per-operator measurement, or a sweep over the exponent relating noise to window width, would tell a reader whether the collapse is a property of the converter or of the assumption.
4 comments

Thank you for the careful review. Your questions prompted us to separate the effects of GELU approximation and clamping and to test how sensitive our results are to the assumed relationship between timing noise and time window width.

Question 1

Which condition does the paper intend a reader to take as its central claim? If the contribution is the operator construction and its exactness within declared domains, would you move the measured-noise result into the abstract and lead with the sensitivity finding?

Response

Our main contribution is the construction of operators that are algebraically exact within their declared domains. We will clarify in the abstract and introduction that the conversion results were obtained without injected timing noise and that the sensitivity analysis assumes that the timing noise standard deviation is proportional to the local time window width.

We will add the simulation results to the end of the abstract. At the scale derived from BrainScaleS-2 measurements, all four classifiers approached chance accuracy and GPT-2 perplexity exceeded 106010^{60}. At 1% of that scale, classification accuracy fell by at most 1.22 percentage points and GPT-2 perplexity rose by 4.86% relative to the converted models without noise. Running the converted models on BrainScaleS-2 remains future work.

Question 2

Could the reported difference be decomposed into three numbers: conversion with the exact GELU and no clamping, conversion with the cubic GELU, and the additional cost of clamping? That would separate what is exact by construction from what the finite ranges and the approximation cost.

Response

We evaluated the same checkpoints and validation examples in double precision without timing noise. For the unclamped conditions, we evaluated the conversion operators directly in potential space, using either the exact GELU or the cubic tanh⁡\tanh GELU. We then restored the declared clamps with the cubic GELU. The first two rows report top-1 accuracy changes relative to the original ANN. The third row reports the additional change due to clamping.

ComparisonViT-B/16 on ImageNet-1k (5,000 images)RoBERTa-B on SST-2 (872 examples)
Exact GELU without clamping, relative to the ANN0.000.00 pp (4,303→4,3034{,}303 \to 4{,}303 correct)0.000.00 pp (824→824824 \to 824 correct)
Cubic tanh⁡\tanh GELU without clamping, relative to the ANN−0.06-0.06 pp (4,303→4,3004{,}303 \to 4{,}300 correct)−0.1147-0.1147 pp (824→823824 \to 823 correct)
Additional effect of clamping with cubic tanh⁡\tanh GELU0.000.00 pp (4,300→4,3004{,}300 \to 4{,}300 correct)0.000.00 pp (823→823823 \to 823 correct)

Restoring the clamps changed no predictions on either evaluation set. The clamps moved the logits by at most 9.5×10−39.5\times10^{-3} for ViT-B/16 and 8.2×10−68.2\times10^{-6} for RoBERTa-B. With the clamps in place, the numerical reference matched every prediction from the TTFS simulation, with a maximum absolute logit difference below 8×10−128\times10^{-12} across both models. The observed accuracy loss in these evaluations arose at the GELU approximation step.

We will add this decomposition as a table in the appendix and refer to it from Section 6.2, so that the conversion fidelity results separate what is exact by construction from the approximation and clamping effects.

Question 3

The noise model reduces to a single scalar that scales two measured fractions while preserving their ratio, with the standard deviation assumed proportional to the local time window. How sensitive is the near-chance result at the measured scale to that scaling law? A per-operator measurement, or a sweep over the exponent relating noise to window width, would tell a reader whether the collapse is a property of the converter or of the assumption.

Response

To test how the result in Fig. 3(a) depends on the linear scaling assumption, we varied the exponent pp relating local time-window width to the timing noise standard deviation in ViT-B/16. From an earlier ViT-B/16 run, we took the median declared window width for each ϕ\phi operator type. At each of these reference widths, all three exponents used the same noise standard deviation for a given α\alpha.

We compared the original linear scaling (p=1p=1) with scaling by the square root of the time window width (p=0.5p=0.5) and faster growth (p=1.5p=1.5). The p=1.5p=1.5 condition is consistent with the circuit measurements in our response to Qwkm Q3. Across the three BrainScaleS-2 circuits implementing ϕNL\phi_{\mathrm{NL}}, the fitted exponents ranged from 1.43 to 1.55 (mean 1.51). In the simulation, we applied each tested exponent to both ϕ\phi operators.

The table reports top-1 accuracy (%) on a fixed subset of 500 ImageNet-1k validation images. Each noisy condition is averaged over three seeds.

Noise multiplier α\alphap=0.5p=0.5p=1p=1 (linear)p=1.5p=1.5
0.00\mathbf{0.00} (noise-free)86.40\mathbf{86.40}86.40\mathbf{86.40}86.40\mathbf{86.40}
0.010.0185.8085.8085.2785.2781.2781.27
0.030.0382.8082.8080.0080.0031.5331.53
0.050.0578.2078.2072.2072.200.200.20
0.0750.07563.7363.7351.1351.130.070.07
0.100.1042.9342.9314.8014.800.000.00
1.001.000.070.070.000.000.000.00

At α=1\alpha=1, accuracy was near chance under all three scaling laws. The sharp drop occurred at a lower noise multiplier for p=1.5p=1.5 than for p=1p=1 or p=0.5p=0.5.

I thank the authors for three well-targeted experiments. They answer my questions. My recommendation now rests on what those experiments show. The decomposition establishes that clamping changes no predictions and that the remaining error comes entirely from the cubic GELU, so the constructed operators are effectively exact in simulation. The noise sweep establishes that this exactness does not survive contact with the substrate: at the measured scale every classifier is at chance, and the collapse holds under all three scaling exponents. At one percent of that scale, GPT-2 perplexity still rises by 4.86 percent. That leaves at least two orders of magnitude between the hardware's timing precision and the point at which the method becomes usable. A construction that is exact in a simulator and unusable on the hardware it targets is a boundary result. I would support the submission that reports that boundary as its subject. In its present framing the headline is still the converter, and the converter does not run. Therefore I will keep my current rating.

Thank you for your feedback. We will give greater emphasis to the contrast between ideal conversion fidelity and degradation under the noise model based on hardware measurements in the abstract, introduction, and conclusion. We will include the GELU/clamping decomposition and state the limitations of practical hardware implementation.

Reviewer_G3FoOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The overriding goal of this paper is to translate a pre-trained, off-the-shelf transformer LLM to a neuromorphic computer, in particular a spiking neural network like BrainScaleS-2. There are several strategies in the literature for accomplishing this translation, but the authors argue for time-to-first-spike coding. The arithmetic operations performed by the transformer must somehow be implemented in terms of the firing times and potentials of a network of spiking neurons.

They show how to do this using just five primitive operations. Each operation either controls the potential of one neuron in terms of the firing time of its inputs, or vice-versa (Table 1). By composing these primitives, they implement all the arithemtic operations used by a transformer (Table 2). A simulation of their proposed dynamics shows that it almost perfectly preserves accuracy of the original model.

The spiking neural network model uses continuous time and potentials. In the real world, these quantities would be corrupted by noise. The authors investigate whether their design is robust to this kind of noise. They calibrate a noise model from real hardware, and find that model accuracy would not survive unless there was one or two orders of magnitude less noise.

Critical strengths

This paper addresses a clear, well-scoped problem that is likely to be of significant interest to those in its field.

The paper introduces a clever and elegant method for solving the problem. It seems natural enough to inspire future work on related problems. In the noiseless setting, it seems pretty effective.

Barring some technical passages, the paper is well-presented on the whole and enjoyable to read, certainly so by ICLR standards.

Critical weaknesses

This paper doesn't give a non-expert reader the tools to evaluate whether this model of computation is reasonable or useful in practice. I understand that truly answering this question may be out of scope. It would at least be helpful to discuss the prognosis for getting this to work on physical hardware and compare it to the alternative approaches cited in section two, like Zhao and Wang.

If we accept the noise model as measured, with α=1\alpha = 1, the results of Fig 3a are not encouraging. They seem to suggest that our present hardware (at least BrainScaleS-2) is not precise enough for this idea to work well in practice.

I have a few questions about the methodology, listed below. My rating of "4: Clear Acceptance" is contingent on these being addressed in the rebuttal. It may be acceptable to simply acknowledge something as a limitation, especially if you can place it in the broader context of the field.

Questions for authors
  1. How realistic is a model of computation based on the five primitive operations in Table 1 (or four if you exclude ΨED\Psi_{ED} by the construction of Appendix B.2) is for analog chips? Ignoring the transformer-specific challenges for a minute, how realistic is the method of Stanojevic for FNNs or CNNs? Is this idea anywhere near usable with the hardware we have now?

  2. Can you explain the implications of this point on line 843? "Potentials must be held and supplied as constant currents during the next integration stage; this remains an operator-level assumption" Should I be worried about this?

  3. Likewise for line 862: "Every φ operation receives one synchronization event; a scalar reference is synchronized once per operator call and delivered to each destination." As I understand, this is necessary because some of the primitive operations must read the potential of one neuron at particuluar times, e.g. in equation 6. Is that correct? A motivation I have seen for spiking neural networks is precisely that they are asynchronous. Does this requirement limit the hardware so it is no longer as advantangeous in, e.g., energy usage?

  4. Why are synaptic operations computed for only some rows of Table 3, and none of Table 4? Isn't that supposed to be a selling point of the model of computation chosen by the present work?

  5. Line 1232 "Too few NP circuits met the same criteria to summarize their variation...the four selected NP circuits...provide an exploratory estimate" Should I be worried about this? Why is it so hard to find more than four relaiable NP circuits?

  6. How do the results in Fig 3a compare to Stanojevic's method for FNNs / CNNs? How about to the other methods in the literature proposed for transformers? I know it may difficult to compare them since they use different operations which may require different error models. But basically, I want to know if Fig 3a reflects poorly on this method or poorly on the hardware / the field as a whole. Alternatively, is there any reason to think that α=10−2\alpha = 10^{-2} could be realistic?

  7. Is "Φ\Phi/Ψ\Psi framework" an existing term of art? I understand that you are using Φ\Phi to denote mappings from potential space to spike-time space and Ψ\Psi to denote mappings from spike-time space to potential space, but if this terminology is meant to suggest some important connection to previous work I missed it.

  8. Line 303: Are the gaussianity and independence assumptions justified here? Is this a good statistical model of the noise? I don't see that justified in Appendix E

  9. Line 305: "We leave the other Ψ\Psi operations ideal." why? are the potentials not corrupted by noise?

5 comments

Thank you for your detailed and incisive comments. Responding to them gave us an opportunity to reflect on problems in our presentation and to reconsider issues we had overlooked.

Question 1

How realistic is a model of computation based on the five primitive operations in Table 1 (or four if you exclude ΨED\Psi_{ED} by the construction of Appendix B.2) is for analog chips? Ignoring the transformer-specific challenges for a minute, how realistic is the method of Stanojevic for FNNs or CNNs? Is this idea anywhere near usable with the hardware we have now?

Response

We consider the individual operators feasible to implement in analog circuits. Running a full continuous-time TTFS Transformer is more challenging because the operators must be connected and their timing coordinated.

We see a small feedforward network as a realistic starting point for implementing Stanojevic's method on current hardware. The neuron dynamics are relatively simple, and small TTFS networks have already been run on BrainScaleS-2 (Göltz et al., 2021).

Question 2

Can you explain the implications of this point on line 843? "Potentials must be held and supplied as constant currents during the next integration stage; this remains an operator-level assumption" Should I be worried about this?

Response

In deriving the SwiGLU products, we assume that the current supplied to the next stage is exactly proportional to the membrane potential from the preceding stage and remains constant throughout integration. In a physical circuit, holding the potential and converting it to a current can introduce errors. The derivation treats these steps as ideal.

Question 3

Likewise for line 862: "Every φ operation receives one synchronization event; a scalar reference is synchronized once per operator call and delivered to each destination." As I understand, this is necessary because some of the primitive operations must read the potential of one neuron at particular times, e.g. in equation 6. Is that correct? A motivation I have seen for spiking neural networks is precisely that they are asynchronous. Does this requirement limit the hardware so it is no longer as advantageous in, e.g., energy usage?

Response

Synchronization is required and may increase energy use, latency, or circuit complexity in a hardware implementation. One direction for future work is to explore whether incoming spikes can trigger some MLP operations asynchronously, reducing the number of synchronization events.

Question 4

Why are synaptic operations computed for only some rows of Table 3, and none of Table 4? Isn't that supposed to be a selling point of the model of computation chosen by the present work?

Response

We left some entries in Table 3 blank because the cited papers do not report total synaptic operation counts per image for those models. For example, SpikeZIP-TF reports power estimated from spike activity per simulation step.

The prior studies cited in Table 4 do not report comparable synaptic operation counts per inference for the text models listed there, so the new column will leave those entries blank. The table below gives our estimated synaptic operation counts for each converted text model processing a 128-token input.

ModelSynaptic operations per 128-token sequence (10910^9)
RoBERTa-B11.3611.36
RoBERTa-L39.9639.96
GPT-216.3116.31
Llama 2 7B851.58851.58

We will add these counts to Table 4 and describe how they were computed in the main text.

Question 5

Line 1232 "Too few NP circuits met the same criteria to summarize their variation...the four selected NP circuits...provide an exploratory estimate" Should I be worried about this? Why is it so hard to find more than four reliable NP circuits?

Response

We chose a setting that produced a large difference in spike times between low and high inputs while keeping timing variation across repeated trials small relative to that difference. When we applied this setting to 64 circuits on a single chip, only four fired at the lowest input code, which limited how many circuits we could select.

To see whether choosing settings for individual circuits could improve spike delivery, we reanalyzed our existing measurements. We chose a setting for each circuit using the calibration data and evaluated it on separate validation data. Of the 64 circuits, 46 met the validation criteria, including a spike miss rate of at most 5% across the three representative input codes combined. These results show that tuning settings for individual circuits can increase the number of circuits that fire reliably at the tested inputs. The noise estimate in the manuscript remains exploratory and reflects the circuits selected under the original common setting.

Question 6

How do the results in Fig 3a compare to Stanojevic's method for FNNs / CNNs? How about to the other methods in the literature proposed for transformers? I know it may be difficult to compare them since they use different operations which may require different error models. But basically, I want to know if Fig 3a reflects poorly on this method or poorly on the hardware / the field as a whole. Alternatively, is there any reason to think that α=10−2\alpha = 10^{-2} could be realistic?

Response

We evaluated the official TTFSFormer implementation using the same ViT-B/16 weights and 5,000 ImageNet-1k validation images as in Fig. 3(a). We matched TTFSFormer’s timing noise standard deviation relative to its time window to the corresponding fraction used for our linear ϕ\phi operations. At the noise level corresponding to α=0.10\alpha=0.10, TTFSFormer’s top-1 accuracy fell from 86.04% to 78.90%, and ours fell from 86.00% to 13.08%. Our configuration in Fig. 3(a) was more sensitive in this intermediate noise regime. Each implementation used its own noise injection sites and spike handling rules, and our logarithmic ϕ\phi operations used a larger, separately measured noise fraction. At the noise level corresponding to α=0.30\alpha=0.30, both methods fell below 0.2% accuracy.

For VGG on CIFAR-10, Stanojevic et al. report that accuracy remained at its noise-free value of 93.59% with a spike timing noise standard deviation of 10−410^{-4}, then fell to 59.29% at a standard deviation of 0.03 in their time units. The 10−410^{-4} condition corresponds to α≈0.0092\alpha\approx0.0092 when referenced to their input time window and our measured linear ϕ\phi noise fraction. Their protocol applies the same absolute standard deviation to layers with different time window widths, so this correspondence applies to the input window. Stanojevic et al., Fig. 5(b)

Sensitivity to spike timing noise is a common challenge for TTFS conversion across Transformers and CNNs. Whether physical circuits can achieve the noise level corresponding to α=0.01\alpha=0.01 remains an open question. To attain the accuracy observed at that setting in Fig. 3(a) on a physical system, further research should pursue both hardware improvements that reduce spike timing errors and software approaches, including calibration and noise-aware training, that improve the model’s tolerance to those errors.

Question 7

Is "Φ/Ψ framework" an existing term of art? I understand that you are using Φ to denote mappings from potential space to spike-time space and Ψ to denote mappings from spike-time space to potential space, but if this terminology is meant to suggest some important connection to previous work I missed it.

Response

We introduced the term “Φ/Ψ\Phi/\Psi framework” to formalize the two directions of mapping shared by the TTFS constructions of Stanojevic et al. and TTFSFormer. TTFSFormer derives receiving and firing kernels from target transformations, without restricting those kernels to a prescribed set of analog neuron dynamics and current responses. By contrast, we start from IF/LIF neuron dynamics and current responses with precedents in analog hardware. The chosen circuit dynamics constrain the form of the ϕ\phi and ψ\psi mappings in different ways. We exploit this asymmetry to obtain nonlinear operations and reuse a fixed set of primitive operators to construct the Transformer operations studied here.

Question 8

Line 303: Are the gaussianity and independence assumptions justified here? Is this a good statistical model of the noise? I don't see that justified in Appendix E

Response

Appendix E.3 estimates the magnitude of timing variation by taking the square root of the average variance across input codes. We chose a Gaussian reference distribution for the timing sensitivity analysis for two reasons.

  1. Among continuous distributions on the real line with zero mean and a specified variance, the Gaussian maximizes differential entropy.
  2. Stanojevic et al. (2024) also used Gaussian timing perturbations to assess the sensitivity of TTFS networks to timing jitter.

Where the Gaussian approximation breaks down. We examined two BrainScaleS-2 circuits with 128 validation repetitions per input code. We standardized the spike times separately for each code and compared the observed tail frequencies with those of Gaussian samples of the same size standardized in the same way.

  • NP 184. Spikes occurred more than two standard deviations early in 4.76% of repetitions and more than two standard deviations late in 0.35%. The Gaussian reference gives 2.21% in each tail, so it misses the observed asymmetry.
  • NL 1. Spikes occurred more than three standard deviations late in 1.11% of repetitions, about ten times the Gaussian reference of 0.115%.

We also observed correlated timing errors in some simultaneously recorded circuits, so independence remains a limitation of the simulation.

Question 9

Line 305: "We leave the other Ψ operations ideal." why? are the potentials not corrupted by noise?

Response

Membrane potentials can also be affected by noise. In the BrainScaleS-2 experiments, we used spike inputs to establish the membrane potential and then recorded the time of the first spike generated by the ϕ\phi operation. Thus, noise from the fixed-duration ψInt\psi_{\mathrm{Int}} operation used to charge the membrane potential was combined with noise from the ϕ\phi operation. We therefore ran the simulations treating each measured ϕ\phi noise estimate as including ψ\psi noise. However, these experiments did not separately measure the potential noise of the Ψ\Psi operations, and how potential noise arising in each actual Ψ\Psi operation affects the output spike time of the subsequent ϕ\phi operation remains a topic for future work.

Reviewer_4s2jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates the conversion of pretrained Transformers into time-to-first-spike (TTFS) continuous-time spiking neural networks (SNNs). The authors propose a Φ/Ψ conversion framework based on the dual relationship between membrane-potential space and spike-time space. Rather than designing specialized neuron dynamics for individual Transformer operations, the proposed framework composes these operations using five primitive operators. The authors evaluate the framework on several architectures, including ViT, RoBERTa, GPT-2, and Llama 2. The results show that the converted models closely preserve the performance of their pretrained ANN counterparts under ideal conditions, with a maximum reported classification accuracy reduction of approximately 0.11 percentage points.

Critical strengths

1). Rather than designing specialized neuron dynamics for each Transformer operation, the paper proposes a unified framework that constructs arithmetic and nonlinear functions by composing five primitive operators. I think this is an interesting approach toward generalizing ANN-to-SNN conversion. 2). The authors evaluate the proposed framework across several Transformer architectures, including both pre-norm and post-norm Transformers, different normalization mechanisms, and gated nonlinearities. From the results, it can be seen that the converted models closely preserve the pretrained ANN computations under ideal conditions.

Critical weaknesses

1). I feel that the novelty of the proposed framework relative to TTFSFormer needs to be more clearly differentiated. In its current form, it is not entirely clear to me what the main advantages of the proposed framework are over this existing approach. I suggest that the authors provide a clearer discussion of these differences. 2). Fig. 3(a) shows that model performance degrades significantly under timing noise.

Questions for authors

1). From Fig. 3(a), it can be seen that model performance degrades significantly under timing noise. Could the authors provide some insights into how this limitation could be addressed? 2). Since the converted SNN models achieve nearly identical accuracy to their pretrained ANN counterparts, could the authors provide a layer-wise analysis of the differences between the ANN activations and the corresponding SNN representations, along with an analysis of the samples where predictions differ? This would help understand where the remaining conversion errors originate. 3). Could the authors discuss whether the proposed conversion approach would generalize to other attention mechanisms, such as linear or sparse attention?

3 comments

Thank you for pointing out where the manuscript could be clearer. Your question about mitigating timing noise prompted us to examine which parts of the model are most sensitive to it.

We should have explained more clearly how our approach differs from TTFSFormer. TTFSFormer constructs Transformer operations by choosing current kernels for the target transformations. For example, its SiLU, multiplication, and LayerNorm constructions use kernels containing a sigmoid, a shifted reciprocal, and a power with exponent −3/2-3/2, respectively. The paper evaluates the conversion in simulation and leaves the circuit implementation of these waveforms open. Our framework composes Transformer operations from a fixed set of reusable primitive operators based on standard LIF and IF neuron models and circuit operations demonstrated in prior analog hardware, including integration of constant currents, pulse-width modulation, and exponential difference. We will add this comparison to the revised manuscript.

Question 1

From Fig. 3(a), it can be seen that model performance degrades significantly under timing noise. Could the authors provide some insights into how this limitation could be addressed?

Response

Timing noise could be mitigated by repeating the same encoding across time windows or by combining spikes from multiple neurons. Our MLP linear projections are particularly sensitive to timing errors in the shared zero-reference spike. In a simulation on 128 ImageNet-1k validation images at α=0.075\alpha=0.075 (7.5% of the measured timing noise scale), replacing the shared reference neuron with four neurons, each using one quarter of the original synaptic weight, raised mean top-1 accuracy from 56.56% to 67.66%. Relative to the 84.38% baseline without timing noise on the same images, this reduced the accuracy drop from 27.81 to 16.72 percentage points.

Following Qwkm's suggestion, we also found that using training percentiles to set the MLP projection input bounds improved accuracy at α=0.075\alpha=0.075 in a separate experiment (see our response to Qwkm Q2).

Question 2

Since the converted SNN models achieve nearly identical accuracy to their pretrained ANN counterparts, could the authors provide a layer-wise analysis of the differences between the ANN activations and the corresponding SNN representations, along with an analysis of the samples where predictions differ? This would help understand where the remaining conversion errors originate.

Response

On 5,000 ImageNet-1k validation images, we recorded corresponding ANN activations and SNN representations in ViT-B/16. For each image and block output, we divided the norm of the activation difference by the norm of the ANN activations, both computed across all tokens. As shown in the table below, the median ratio generally increased with depth, from 0.44% in block 1 to 1.57% in block 12. The cubic tanh approximation to GELU introduced most of the conversion error, and LayerNorm amplified differences already present in its inputs. To isolate the effect of this approximation, we replaced it with exact GELU in the converted SNNs for ViT-B/16, RoBERTa-B, and RoBERTa-L, keeping all other TTFS operations unchanged. The resulting SNNs matched their ANN counterparts' top-1 predictions on all 5,000 ImageNet-1k validation images for ViT-B/16 and all 872 SST-2 validation examples for each RoBERTa model. This included the five ViT-B/16 images and one RoBERTa-B example on which the original ANN and SNN predictions differed.

Block outputMedian ratio
10.44%
20.62%
30.76%
40.85%
50.90%
61.01%
71.06%
81.27%
91.49%
101.69%
111.87%
121.57%

Question 3

Could the authors discuss whether the proposed conversion approach would generalize to other attention mechanisms, such as linear or sparse attention?

Response

We used our existing TTFS operators to simulate linear attention in pretrained EfficientViT-B0 and sparse attention in pretrained Swin-T. We replaced only the attention calculations. In EfficientViT-B0, these were the two matrix products and normalization. In Swin-T, they were the score computation, Softmax, and weighted value sum within each window. We used 5,000 ImageNet-1k training images to calibrate Q/K/V in both models and the attention scores in Swin-T.

On 5,000 ImageNet-1k validation images without timing noise, top-1 accuracy before and after the replacement was 72.58% for EfficientViT-B0 and 82.04% for Swin-T.

The same TTFS operators can be composed for other attention mechanisms when their computations use supported operations and their intermediate values fit the operator domains. The hardware cost of selecting and routing those keys and values depends on the sparsity pattern.

I thank the authors for their rebuttal. I'll maintain my original score. I suggest authors to carefully review the manuscript once more, identify areas where the writing and clarity could be improved, and revise those sections accordingly in the final version.

Thank you for reviewing our response and for your advice. We will carefully revise the manuscript for clarity, particularly the distinction from TTFSFormer, the sources of conversion error, and the scope and limitations of the noise and attention experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.