- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the expressivity of spike-driven Transformers after a fixed LIF front end has partitioned inputs into finite-state regions. It introduces region-combination capacity and separates restrictions from shared branch state exposure and query–key product collisions. The resulting two-budget attention envelope is propagated through gated FFNs and depth. Finite-state enumeration and frozen SDT-V3 score probes support the local analysis. The central contribution is theoretical characterization of retained interaction traces, not a demonstrated improvement in full-network classification.
Critical strengths
- The distinction between input-region counts and assignments of finite downstream interaction traces is meaningful. The branch-versus-score decomposition explains why extra nominal product labels need not translate into additional jointly realizable traces.
- The authors enumerate counterexamples to tightness, examine score-limited and branch-limited regimes, and evaluate product collisions on frozen ImageNet-trained SDT-V3 representations. These are stronger checks than merely illustrating the algebra on independent labels.
Critical weaknesses
- The incremental explanatory value of the image-exposure cap for practical architectures remains weakly established. Appendix B.5 reports that its nominal Q/K exposure caps are inactive in all eight layers for both SDT-V3 profiles with 196 tokens; the corresponding envelopes are branch-limited for another reason, namely shared operand assignment. The nontrivial cap effects are mainly established analytically or in capped finite models rather than by measuring feasible simultaneous images of a concrete trained LIF branch. This limits, but does not invalidate, the new bound's demonstrated practical relevance.
- The formal observation counts retained score/value/FFN traces, not necessarily distinct network outputs or predictive functions. The frozen recovery study is intentionally an offline score diagnostic: it groups FP64 scores by exact equality, and the trained score-only probe is only a modest downstream test (for SDT-V3-S, 19.90% raw vs 21.73% vector on 100 classes; three runs). Appendix B.6 also reports that scalar recovery can reduce collisions while lowering probe accuracy. Thus, the evidence does not yet show when stronger trace distinguishability survives realistic quantization and subsequent value aggregation/spiking or yields a meaningful end-task benefit.
Questions for authors
- Can the authors instantiate a small, concrete shared LIF branch (rather than an abstract capped assignment model), enumerate the simultaneously realizable states/traces over fixed inputs, and compare the realized count with the exact or nominal image cap?
- For a realistic block, to what extent do recovered score distinctions survive value aggregation, post-attention spiking, and the fixed output readout? Please distinguish output-level evidence from the present score-only diagnostic.
- How robust are collision counts and trace gains when raw/scalar/vector scores are compared after the same realistic score quantizer/bit width? This is relevant because Appendix B.6 uses exact FP64 equality for the recovered scores.
6 comments
Response to Reviewer Voev — Part 2 of 3
2. Survival of recovered distinctions through downstream computation (W2 / Q2)
W2: “The formal observation counts retained score/value/FFN traces, not necessarily distinct network outputs or predictive functions.”
Q2: “For a realistic block, to what extent do recovered score distinctions survive value aggregation, post-attention spiking, and the fixed output readout? Please distinguish output-level evidence from the present score-only diagnostic.”
Thank you for asking how often a recovered score distinction reaches the fixed classifier. Theorem 1 identifies which distinctions recovery preserves at the score stage. As stated in Corollary A.13, if outputs factor through the retained traces by a fixed map , then and . For attention with fixed , a score difference survives aggregation exactly when . Each head of SDT-V3-L's final block has a value matrix of size , so the map from a score row to its aggregate has a null space of dimension at least 16. Its dimension alone does not determine how many recovered distinctions survive.
We traced individual score distinctions through the frozen final block and its original 1,000-class classifier. We took one held-out image from each of the 100 probe classes and obtained together in a fresh CPU forward pass. Its Q/K states differ at some coordinates from the archived states used in Q3. For one query per image and head, we sampled two distinct key vectors tied under the raw score, giving 800 pairs. Under each score map, we compared the unmodified row with the same row after swapping just those two scores. All other scores, values, residual inputs, and weights were held fixed within each comparison. We reused the same 800 pairs across the three recovery seeds. The raw control reproduced the original post-attention spikes and classifier inputs of this forward pass on all 100 images.
| Score map | Swaps tested | Score rows changed | changed (exact / FP32) | Post-attention MultiSpike changed | Fixed-head logits changed | Top-1 changed |
|---|---|---|---|---|---|---|
| Raw | 800 | 0 | 0 / 0 | 0 | 0 | 0 |
| Scalar | 2,400 | 2,181 | 1,902 / 1,995 | 29 | 8 | 1 |
| Vector | 2,400 | 2,400 | 2,088 / 2,183 | 54 | 14 | 0 |
For a swap at keys , the exact aggregation difference is . FP32 accumulation also produced small differences for some algebraically identical aggregates. All of these merged at MultiSpike. Of the 2,400 distinctions recovered by vector shifts, 87.0% survived aggregation algebraically, 2.25% changed a post-attention spike, and 0.58% changed the fixed classifier's logits (maximum absolute logit change above ). Top-1 stayed the same in all 2,400 comparisons. Thus, the largest observed loss occurred at post-attention spiking. These counts measure sensitivity to a single score swap within a real image. Appendix B.6, Table 8 reports these counts alongside the separately trained score-only probe in Table 5, which measures class information accessible to a new readout.
Response to Reviewer Voev — Part 3 of 3
3. Comparison under the same score quantizer and bit width (Q3)
Q3: “How robust are collision counts and trace gains when raw/scalar/vector scores are compared after the same realistic score quantizer/bit width? This is relevant because Appendix B.6 uses exact FP64 equality for the recovered scores.”
Thank you for highlighting the importance of comparing scores under the same quantization conditions. Theorem 1 characterizes distinguishability under exact products, while score quantization can further merge distinct product classes. Our new comparison applies the same floating-point format and conversion rule to all three score types, and vector recovery retains a substantial advantage under FP32, FP16, and BF16.
New experiment. We use Q/K states from the final attention layer of frozen SDT-V3-L, with 2,000 test images, eight heads, eight fixed queries per head, all 196 keys, and three recovery seeds. The query indices are fixed at [1, 16, 28, 54, 74, 93, 97, 119] across images, heads, recovery conditions, and precisions, using the protocol that reproduces Table 4. The FP64 baseline reproduces Table 4's SDT-V3-L final-layer raw/scalar losses (0.870/0.437 at the reported precision). All three score types use the same original Q/K states, query positions, floating-point format, and conversion rule, without condition-specific rescaling. Scores are computed in FP64 and then converted to FP32, FP16, or BF16; BF16 conversion proceeds through FP32 with round-to-nearest-even. This isolates score-representation rounding. The mean normalized distinct-key loss from Appendix B.6 is:
| Score representation | Raw loss | Scalar loss | Vector loss |
|---|---|---|---|
| FP64 | 0.8696 | 0.4366 | 0 |
| FP32 | 0.8696 | 0.4366 | 3.67e-06 |
| FP16 | 0.8696 | 0.4657 | 0.0415 |
| BF16 | 0.8696 | 0.5858 | 0.2292 |
FP32 preserves almost all recovered distinctions. FP16 and BF16 introduce additional collisions, but vector recovery has the lowest loss for all three seeds. Thus, the recovery advantage persists under matched finite-precision score representations, while exact collision freedom depends on the representation precision.
Appendix B.6, Table 7 gives the matched comparison and conversion protocol, and identifies the evaluated FP64 scores as the setting in which no vector collisions were observed.
Response to Reviewer Voev — Part 1 of 3
Dear Reviewer Voev,
We extend our sincere gratitude for dedicating your valuable time to reviewing our paper and for expressing your appreciation and support for our work. In the following, we provide a comprehensive response to your review comments.
1. Incremental contribution of the image-exposure cap in concrete LIF branches (W1 / Q1)
Thank you for highlighting the incremental contribution of the exposure cap. It turns a coexistence constraint imposed by parameter sharing into a capacity upper bound: states individually available across a branch family need not coexist under one parameter realization. As stated in Eq. 10 and Corollary 1,
N(N;Q,B)=∑j=1min(N,B)(jQ)j!S(N,j),
limN→∞N1logN(N;Q,B)=logB,
where the limit holds for fixed integers Q≥B≥1. Independent assignments have rate logQ, so B<Q changes the exponential growth rate of the envelope. This identifies simultaneous state exposure as a separate architectural resource: increasing the nominal alphabet alone need not increase the envelope's asymptotic rate.
For a finite number of slots N, the cap tightens QN exactly when B<min(N,Q). The SDT-V3 profiles in Appendix B.5 do not meet this condition, so their exposure caps provide no additional tightening; their branch-limited envelopes reflect shared operand assignments. The paper thus distinguishes these two structural restrictions and the regimes in which each contributes.
For a concrete shared LIF branch, the cap alone excludes 9.4% of independent assignments at four inputs and 96.0% at sixteen. We fix two count-decoded neurons with T=1, threshold 1, and zero initial potential. The inputs are the distinct ordered scalars x=0,…,N−1, so the first row uses {0,1,2,3}. Each neuron has its own weight and bias, shared across all inputs. The branch alphabet has Q=4 states. Two scalar thresholds can expose at most three joint states simultaneously, and this maximum is attainable, giving the exact global cap B=3. Varying only N gives the following exact combinatorial counts.
Eq. 10 gives N(N;4,3)=4⋅3N−6⋅2N+4. To obtain the realized count, observe that each neuron produces a prefix or suffix of ones on the ordered inputs. Counting the two constant patterns once gives exactly 2N patterns per neuron and (2N)2 assignments for the branch. The cap reduces the envelope's exponential rate from log4 to log3. The scalar threshold geometry further restricts the realized family to polynomial growth, with limN→∞N−1log(4N2)=0. These counts quantify the contribution of each restriction separately.
Our complementary two-input example tests attainment. At T=2, shared affine weights and biases can set arbitrary drives at x=0,1. All 81 assignments per two-neuron branch are realizable, and vector recovery preserves all 812=6,561 joint Q/K assignments. The exposure cap is inactive in this case. Appendix B.5, Table 6 gives the active-cap counts and derivation alongside the SDT-V3 profiles. Appendix B.7.2, Table 10 gives the complementary two-input attainment result.