- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper converts pretrained Transformers into TTFS-SNNs. It defines two encoders from potential to spike time and three decoders from spike time back to potential. Composing them gives multiplication, division, exponentiation, Softmax, normalization, GELU and SwiGLU. Without noise, the converted models (ViT up to Llama-2-7B) lose at most about 0.1% accuracy. The paper also injects timing noise scaled from BrainScaleS-2 measurements.
Critical strengths
- This paper proposes a clear set of primitives.
- Evaluate up to 7B.
- The time noise is based on hardware measurements and negative results are reported openly.
Critical weaknesses
- The declared ranges require a very large dynamic range. The GELU exponent input is [-80, 80], so is in $
$. Since windows and gains follow the declared range, every evaluation uses a 160-unit window and an gain, which float64 handles but analog hardware cannot.
- With the measured noise, all classifiers fall to chance. The paper does not explain why of propose any fix. Noise grows with window width, and the windows come from min–max calibration, so outliers are a likely cause. Clipping, per-channel ranges or noise-aware calibration should be tried.
- The noise measurement is thin. Only 4 of 63 NP circuits passed the selection criteria, and those criteria were relaxed after looking at the data. The assumption that noise scales linearly with window width is not tested.
Questions for authors
- Please report per-operator dynamic ranges and test a smaller clip.
- Does outlier-robust or noise-aware calibration change the needed to keep accuracy?
- Is the linear noise scaling supported by measurements at several window widths?
5 comments
Question 2
Does outlier-robust or noise-aware calibration change the α needed to keep accuracy?
Response
On ViT-B/16, we compared mean top-1 accuracy across three matched noise seeds. Percentile calibration of the two MLP projection inputs increased the largest tested noise multiplier that met our 85.4% mean accuracy target from to . The target was one percentage point below the noise-free min–max accuracy. At , the 0.01% cutoff per tail gave 85.87% accuracy versus 85.27% for min–max. The paired 95% confidence interval for the difference was percentage points.
We changed the calibration bounds for one group at a time using training percentiles. All four groups improved accuracy at , while LayerNorm calibration caused the largest loss in noise-free accuracy.
Table 1(a). ViT-B/16 top-1 accuracy (%) with a 0.01% cutoff per tail
| Min–max | Attention | LayerNorm | MLP projection inputs | GELU inputs | |
|---|---|---|---|---|---|
Table 1(b). ViT-B/16 top-1 accuracy (%) with a 0.1% cutoff per tail
| Min–max | Attention | LayerNorm | MLP projection inputs | GELU inputs | |
|---|---|---|---|---|---|
Table 1 reports seed 0 on 500 ImageNet-1k validation images.
We then applied the 0.01% cutoff jointly to attention, GELU inputs, and MLP projection inputs. The gains at intermediate noise levels were most pronounced for ViT. CCT-7 and RoBERTa-B showed smaller accuracy gains. GPT-2’s perplexity decreased at some higher noise levels, where it remained high, and increased without noise.
Table 2. Performance with joint calibration and change relative to min–max at the same
| ViT-B/16 | ViT-S/16 | CCT-7 | RoBERTa-B | GPT-2 | |
|---|---|---|---|---|---|
| % ( pp) | % ( pp) | % ( pp) | % ( pp) | (%) | |
| % ( pp) | % ( pp) | % ( pp) | % ( pp) | (%) | |
| % ( pp) | % ( pp) | % ( pp) | % ( pp) | (%) | |
| % ( pp) | % ( pp) | % ( pp) | % ( pp) | (%) | |
| % ( pp) | % ( pp) | % ( pp) | % ( pp) | (%) | |
| % ( pp) | % ( pp) | % ( pp) | % ( pp) | (%) | |
| % ( pp) | % ( pp) | % ( pp) | % ( pp) | (%) | |
| % ( pp) | % ( pp) | % ( pp) | % ( pp) | (%) | |
| — | % ( pp) | % ( pp) | % ( pp) | (%) |
*Classifier columns report accuracy and percentage-point changes. GPT-2 reports token-weighted perplexity and its relative percentage change, with lower values indicating better performance. Nonzero rows report three-seed means.
Question 3
Is the linear noise scaling supported by measurements at several window widths?
Response
On BrainScaleS-2, we measured the standard deviation of the first spike time at several time window widths, using three circuits for each of the and operators. For each circuit, we fit by minimizing the mean squared error between the measured standard deviations and the fitted values, where is the fitted signal span. The mean fitted exponent across circuits was 1.28 (range 1.07–1.44) for and 1.51 (range 1.43–1.55) for . For one circuit, fitting the coefficient with gave nearly the same mean squared error as fitting both parameters. All three circuits had fitted exponents above 1.
The linear scaling used in our original experiments underestimates spike-time noise at wider time windows, as observed in circuits for both operators. In practice, shorter time windows can help mitigate this noise, although achieving them while preserving the operator response may incur additional circuit costs.
We appreciate your close look at the dynamic range required by the GELU exponential. Your suggestions led us to test a narrower clip for its input and examine whether percentile calibration could improve robustness to timing noise.
Question 1
Please report per-operator dynamic ranges and test a smaller clip.
Response
Table 1 compares the observed ranges under the original [−80,80] clip and the narrower [−10,5] clip. We varied the lower and upper bounds of the exponential input within GELU independently. Table 2 reports accuracy across the sweep. The [−10,5] setting reduced the GELU exponential time window from 160 to 15 and preserved all predictions on the 500 validation images. With the [−10,2.5] clip, accuracy decreased from 86.4% to 85.8%, a drop of 0.6 percentage points.
Table 1. Observed operator ranges with the original and narrower GELU exponential input clips
Each interval gives the observed interval across the indicated operator calls on 500 validation images.
We report attention separately because a common shift of the scores in each row can rescale the exponential output range while preserving the Softmax weights in exact arithmetic. In both runs, the largest exponential value in attention was 1.84×1047, against a declared upper bound of 9.69×1072.
Table 2. Top-1 accuracy (%) across the GELU exponential input clips
Rows and columns give the lower and upper bounds, respectively.