acceptodds
Under review as a conference paper at ICLR 2027

How Expressive Are Spike-Driven Transformers? An Image-Capped Finite-Trace Analysis

Abstract

abstract We study the expressive power of spike-driven Transformers beyond conventional linear-region counting. Once spiking dynamics map continuous inputs to finite states, downstream Transformer layers need not create new input regions to realize different functions; instead, expressivity depends on how these states can be distinguished and recombined. We formalize this effect through region-combination capacity, which counts the finite interaction traces assignable to LIF-induced input regions. Our analysis reveals two fundamental bottlenecks. First, shared branches cannot assign their global state alphabets independently across tokens, imposing a state-exposure constraint. Second, multiplicative interactions can collapse distinct operand pairs into the same score. Together, these effects yield a two-budget attention envelope , which identifies whether branch sharing or score collisions limit the expressive gain of attention. Composing the surviving distinctions through values, gated FFNs, residual paths, and depth gives for homogeneous count-decoded models, with a tagged construction approaching this exponent. Finite enumeration validates the predicted sharing cap, product distinguishability, phase transition, and envelope tightness, while architecture-level studies on SDT-V3 and Spikformer show how the framework can diagnose the active expressive bottleneck. These results provide a finite-trace theory for understanding when Transformer interactions can, and cannot, overcome the discrete-state bottlenecks of SNNs.

Primary area:
learning theory
Keywords:
Spiking Neural Networks, Spike-driven Transformers, Neural Network Expressivity, Finite-state Analysis, Self-Attention, Linear Regions, Representation Capacity.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VoevOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the expressivity of spike-driven Transformers after a fixed LIF front end has partitioned inputs into finite-state regions. It introduces region-combination capacity and separates restrictions from shared branch state exposure and query–key product collisions. The resulting two-budget attention envelope is propagated through gated FFNs and depth. Finite-state enumeration and frozen SDT-V3 score probes support the local analysis. The central contribution is theoretical characterization of retained interaction traces, not a demonstrated improvement in full-network classification.

Critical strengths
  1. The distinction between input-region counts and assignments of finite downstream interaction traces is meaningful. The branch-versus-score decomposition explains why extra nominal product labels need not translate into additional jointly realizable traces.
  2. The authors enumerate counterexamples to tightness, examine score-limited and branch-limited regimes, and evaluate product collisions on frozen ImageNet-trained SDT-V3 representations. These are stronger checks than merely illustrating the algebra on independent labels.
Critical weaknesses
  1. The incremental explanatory value of the image-exposure cap for practical architectures remains weakly established. Appendix B.5 reports that its nominal Q/K exposure caps are inactive in all eight layers for both SDT-V3 profiles with 196 tokens; the corresponding envelopes are branch-limited for another reason, namely shared operand assignment. The nontrivial cap effects are mainly established analytically or in capped finite models rather than by measuring feasible simultaneous images of a concrete trained LIF branch. This limits, but does not invalidate, the new bound's demonstrated practical relevance.
  2. The formal observation counts retained score/value/FFN traces, not necessarily distinct network outputs or predictive functions. The frozen recovery study is intentionally an offline score diagnostic: it groups FP64 scores by exact equality, and the trained score-only probe is only a modest downstream test (for SDT-V3-S, 19.90% raw vs 21.73% vector on 100 classes; three runs). Appendix B.6 also reports that scalar recovery can reduce collisions while lowering probe accuracy. Thus, the evidence does not yet show when stronger trace distinguishability survives realistic quantization and subsequent value aggregation/spiking or yields a meaningful end-task benefit.
Questions for authors
  1. Can the authors instantiate a small, concrete shared LIF branch (rather than an abstract capped assignment model), enumerate the simultaneously realizable states/traces over fixed inputs, and compare the realized count with the exact or nominal image cap?
  2. For a realistic block, to what extent do recovered score distinctions survive value aggregation, post-attention spiking, and the fixed output readout? Please distinguish output-level evidence from the present score-only diagnostic.
  3. How robust are collision counts and trace gains when raw/scalar/vector scores are compared after the same realistic score quantizer/bit width? This is relevant because Appendix B.6 uses exact FP64 equality for the recovered scores.
6 comments
AuthorsOpenReview ↗

Response to Reviewer Voev — Part 1 of 3

Dear Reviewer Voev,

We extend our sincere gratitude for dedicating your valuable time to reviewing our paper and for expressing your appreciation and support for our work. In the following, we provide a comprehensive response to your review comments.

1. Incremental contribution of the image-exposure cap in concrete LIF branches (W1 / Q1)

W1: “The incremental explanatory value of the image-exposure cap for practical architectures remains weakly established.”

Q1: “Can the authors instantiate a small, concrete shared LIF branch (rather than an abstract capped assignment model), enumerate the simultaneously realizable states/traces over fixed inputs, and compare the realized count with the exact or nominal image cap?”

Thank you for highlighting the incremental contribution of the exposure cap. It turns a coexistence constraint imposed by parameter sharing into a capacity upper bound: states individually available across a branch family need not coexist under one parameter realization. As stated in Eq. 10 and Corollary 1,

N(N;Q,B)=∑j=1min⁡(N,B)(Qj)j!S(N,j)\mathcal N(N;Q,B)=\sum_{j=1}^{\min(N,B)}\binom{Q}{j}j!S(N,j),

lim⁡N→∞1Nlog⁡N(N;Q,B)=log⁡B\lim_{N\to\infty}\frac{1}{N}\log\mathcal N(N;Q,B)=\log B,

where the limit holds for fixed integers Q≥B≥1Q\geq B\geq1. Independent assignments have rate log⁡Q\log Q, so B<QB<Q changes the exponential growth rate of the envelope. This identifies simultaneous state exposure as a separate architectural resource: increasing the nominal alphabet alone need not increase the envelope's asymptotic rate.

For a finite number of slots NN, the cap tightens QNQ^N exactly when B<min⁡(N,Q)B<\min(N,Q). The SDT-V3 profiles in Appendix B.5 do not meet this condition, so their exposure caps provide no additional tightening; their branch-limited envelopes reflect shared operand assignments. The paper thus distinguishes these two structural restrictions and the regimes in which each contributes.

For a concrete shared LIF branch, the cap alone excludes 9.4% of independent assignments at four inputs and 96.0% at sixteen. We fix two count-decoded neurons with T=1T=1, threshold 1, and zero initial potential. The inputs are the distinct ordered scalars x=0,…,N−1x=0,\ldots,N-1, so the first row uses {0,1,2,3}\lbrace 0,1,2,3\rbrace. Each neuron has its own weight and bias, shared across all inputs. The branch alphabet has Q=4Q=4 states. Two scalar thresholds can expose at most three joint states simultaneously, and this maximum is attainable, giving the exact global cap B=3B=3. Varying only NN gives the following exact combinatorial counts.

Inputs NNIndependent assignments 4N4^NImage-capped envelope N(N;4,3)\mathcal N(N;4,3)Realizable branch assignments
425623264
865,53624,712256
1216,777,2162,101,192576
164,294,967,296171,793,6721,024

Eq. 10 gives N(N;4,3)=4⋅3N−6⋅2N+4\mathcal N(N;4,3)=4\cdot3^N-6\cdot2^N+4. To obtain the realized count, observe that each neuron produces a prefix or suffix of ones on the ordered inputs. Counting the two constant patterns once gives exactly 2N2N patterns per neuron and (2N)2(2N)^2 assignments for the branch. The cap reduces the envelope's exponential rate from log⁡4\log 4 to log⁡3\log 3. The scalar threshold geometry further restricts the realized family to polynomial growth, with lim⁡N→∞N−1log⁡(4N2)=0\lim_{N\to\infty}N^{-1}\log(4N^2)=0. These counts quantify the contribution of each restriction separately.

Our complementary two-input example tests attainment. At T=2T=2, shared affine weights and biases can set arbitrary drives at x=0,1x=0,1. All 81 assignments per two-neuron branch are realizable, and vector recovery preserves all 812=6,56181^2=6,561 joint Q/K assignments. The exposure cap is inactive in this case. Appendix B.5, Table 6 gives the active-cap counts and derivation alongside the SDT-V3 profiles. Appendix B.7.2, Table 10 gives the complementary two-input attainment result.

AuthorsOpenReview ↗

Response to Reviewer Voev — Part 2 of 3

2. Survival of recovered distinctions through downstream computation (W2 / Q2)

W2: “The formal observation counts retained score/value/FFN traces, not necessarily distinct network outputs or predictive functions.”

Q2: “For a realistic block, to what extent do recovered score distinctions survive value aggregation, post-attention spiking, and the fixed output readout? Please distinguish output-level evidence from the present score-only diagnostic.”

Thank you for asking how often a recovered score distinction reaches the fixed classifier. Theorem 1 identifies which distinctions recovery preserves at the score stage. As stated in Corollary A.13, if outputs factor through the retained traces by a fixed map GG, then O=G(T)\mathcal O=G(\mathcal T) and ∣O∣≤∣T∣|\mathcal O|\leq|\mathcal T|. For attention with fixed VV, a score difference survives aggregation exactly when (S−S′)V≠0(S-S')V\neq 0. Each head of SDT-V3-L's final block has a value matrix VV of size 196×180196\times180, so the map from a score row to its aggregate has a null space of dimension at least 16. Its dimension alone does not determine how many recovered distinctions survive.

We traced individual score distinctions through the frozen final block and its original 1,000-class classifier. We took one held-out image from each of the 100 probe classes and obtained Q,K,VQ,K,V together in a fresh CPU forward pass. Its Q/K states differ at some coordinates from the archived states used in Q3. For one query per image and head, we sampled two distinct key vectors tied under the raw score, giving 800 pairs. Under each score map, we compared the unmodified row with the same row after swapping just those two scores. All other scores, values, residual inputs, and weights were held fixed within each comparison. We reused the same 800 pairs across the three recovery seeds. The raw control reproduced the original post-attention spikes and classifier inputs of this forward pass on all 100 images.

Score mapSwaps testedScore rows changedSVSV changed (exact / FP32)Post-attention MultiSpike changedFixed-head logits changedTop-1 changed
Raw80000 / 0000
Scalar2,4002,1811,902 / 1,9952981
Vector2,4002,4002,088 / 2,18354140

For a swap at keys j,kj,k, the exact aggregation difference is (sj−sk)(vk−vj)(s_j-s_k)(v_k-v_j). FP32 accumulation also produced small differences for some algebraically identical aggregates. All of these merged at MultiSpike. Of the 2,400 distinctions recovered by vector shifts, 87.0% survived aggregation algebraically, 2.25% changed a post-attention spike, and 0.58% changed the fixed classifier's logits (maximum absolute logit change above 10−610^{-6}). Top-1 stayed the same in all 2,400 comparisons. Thus, the largest observed loss occurred at post-attention spiking. These counts measure sensitivity to a single score swap within a real image. Appendix B.6, Table 8 reports these counts alongside the separately trained score-only probe in Table 5, which measures class information accessible to a new readout.

AuthorsOpenReview ↗

Response to Reviewer Voev — Part 3 of 3

3. Comparison under the same score quantizer and bit width (Q3)

Q3: “How robust are collision counts and trace gains when raw/scalar/vector scores are compared after the same realistic score quantizer/bit width? This is relevant because Appendix B.6 uses exact FP64 equality for the recovered scores.”

Thank you for highlighting the importance of comparing scores under the same quantization conditions. Theorem 1 characterizes distinguishability under exact products, while score quantization can further merge distinct product classes. Our new comparison applies the same floating-point format and conversion rule to all three score types, and vector recovery retains a substantial advantage under FP32, FP16, and BF16.

New experiment. We use Q/K states from the final attention layer of frozen SDT-V3-L, with 2,000 test images, eight heads, eight fixed queries per head, all 196 keys, and three recovery seeds. The query indices are fixed at [1, 16, 28, 54, 74, 93, 97, 119] across images, heads, recovery conditions, and precisions, using the protocol that reproduces Table 4. The FP64 baseline reproduces Table 4's SDT-V3-L final-layer raw/scalar losses (0.870/0.437 at the reported precision). All three score types use the same original Q/K states, query positions, floating-point format, and conversion rule, without condition-specific rescaling. Scores are computed in FP64 and then converted to FP32, FP16, or BF16; BF16 conversion proceeds through FP32 with round-to-nearest-even. This isolates score-representation rounding. The mean normalized distinct-key loss from Appendix B.6 is:

Score representationRaw lossScalar lossVector loss
FP640.86960.43660
FP320.86960.43663.67e-06
FP160.86960.46570.0415
BF160.86960.58580.2292

FP32 preserves almost all recovered distinctions. FP16 and BF16 introduce additional collisions, but vector recovery has the lowest loss for all three seeds. Thus, the recovery advantage persists under matched finite-precision score representations, while exact collision freedom depends on the representation precision.

Appendix B.6, Table 7 gives the matched comparison and conversion protocol, and identifies the evaluated FP64 scores as the setting in which no vector collisions were observed.

Reviewer_fdQKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the expressivity of spike-driven Transformers through a finite-trace framework. It introduces region-combination capacity and identifies two bottlenecks: shared-branch state exposure and multiplicative product collisions. The resulting bounds are extended to deep networks and examined through finite-state experiments and frozen SDT-V3 models. While the distinction between operand availability and product distinguishability is useful, its connection to actual network expressivity remains insufficiently established.

Critical strengths
  1. The paper correctly distinguishes between the states that a branch can expose and the distinctions that survive multiplicative interactions. In particular, Theorem 1 and its treatment of raw, scalar-recovered, and vector-recovered products give a concrete characterization of when distinct operand pairs remain distinguishable. The discussion of independent versus tied recovery parameters is also important.
  2. The authors do not simply assume that their combinatorial bounds are attained, but they carefully distinguish between upper bounds and realized counts. The frozen-model experiments also separate product-collision measurements from downstream class-readout performance.
Critical weaknesses
  1. The relationship between retained-trace capacity and functional expressivity is insufficiently established. Definition 1 (Eq. 6) measures retained internal trace assignments rather than distinct input-output functions. Different traces may produce identical predictions, while a trainable readout can change outputs without changing these traces. Corollary A.13 only addresses a fixed readout. Thus, the full-network bounds in Eqs. 21–22 do not directly establish functional expressivity. A small-scale comparison between trace capacity and final-output capacity would clarify this relationship.
  2. The two-budget envelope may be loose for actual LIF networks. Eq. 10 counts all assignments satisfying the state-exposure cap but ignores additional geometric constraints imposed by shared LIF parameters. Appendix B.3 evaluates abstract capped assignments rather than realizable LIF mappings, while Appendix B.5 reports inactive state-exposure caps for the SDT-V3 profiles. Moreover, recovery can increase realized traces even when the envelope remains branch-limited (Figure 3(b)). Consequently, the proposed crossover may not identify the actual expressivity bottleneck. Testing a small parameterized LIF network against the predicted envelope would provide more direct evidence.
Questions for authors
  1. Corollary A.13 assumes a fixed readout. How should region-combination capacity be interpreted relative to final-output expressivity?
  2. Appendix B.3 uses abstract capped assignments rather than actual LIF mappings. Does the predicted bottleneck still hold for parameterized LIF branches, and how does the analysis extend to SDT-V3's MultiSpike representations?
  3. Appendix B.6 evaluates exact FP64 score equality, while vector recovery provides only modest classification gains (Table 5). How robust is the observed product recovery under a realistic finite-precision score representation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer fdQK — Part 1 of 4

Dear Reviewer fdQK,

We extend our sincere gratitude for dedicating your valuable time to reviewing our paper and for your valuable feedback. In the following, we provide a comprehensive response to your review comments.

1. Meaning of region-combination capacity and its connection to final-output expressivity (W1 / Q1)

W1: “The relationship between retained-trace capacity and functional expressivity is insufficiently established. … A small-scale comparison between trace capacity and final-output capacity would clarify this relationship.”

Q1: “Corollary A.13 assumes a fixed readout. How should region-combination capacity be interpreted relative to final-output expressivity?”

Thank you for highlighting the relationship between region-combination capacity and final-output expressivity. Our core contribution is an architecture-sensitive theory of finite interaction expressivity that characterizes structural capacity upper bounds and identifies mechanisms preserving internal computational distinctions. Unlike conventional region counting, our theory quantifies both the joint realizability of shared finite states and the distinguishability of their multiplicative interactions, yielding an image-capped assignment bound, a two-budget attention envelope, and a compositional full-network capacity bound. As stated in Definition 1,

TraceCombm(A;π)=\mathrm{TraceComb}_m(\mathcal A;\pi)=

sup⁡ϕ∈Φm∣{Γϕ,ψ(Pϕ):ψ∈Θdown(ϕ)}∣\sup_{\phi\in\Phi_m}|\{\Gamma_{\phi,\psi}(\mathcal P^\phi):\psi\in\Theta_{\mathrm{down}}(\phi)\}|

where Φm={ϕ:∣Pϕ∣≤m}\Phi_m=\left\lbrace\phi:|\mathcal P^\phi|\leq m\right\rbrace, and Γϕ,ψ\Gamma_{\phi,\psi} is a complete region-to-trace assignment for the given front-end partition. For a fixed front end exposing its complete finite state, region counts are insensitive to downstream architectural changes. Our metric instead counts the interaction mappings realizable as admissible parameters vary and exposes two restrictions that region counting alone cannot resolve: shared branches constrain which states can coexist, and multiplication can erase distinctions between those states. Theorem 1 characterizes product collisions and their recovery conditions, identifying interaction losses that architectural changes can remove.

As stated in Corollary A.13, for a fixed front end, if outputs factor through retained traces by a fixed map GG, independent of the varied parameters, then

O=G(T),∣O∣≤∣T∣,\mathcal O=G(\mathcal T),\qquad |\mathcal O|\leq |\mathcal T|,

with equality exactly when GG is injective on the realizable trace-assignment set T\mathcal T. Allowing the readout to vary requires including its family in the output-capacity analysis. Eqs. 21–22 characterize restrictions imposed by the interaction architecture; subsequent computation and the readout determine how retained distinctions translate into output functions. The theory therefore locates architectural losses of distinguishability and identifies mechanisms for preserving those distinctions.

Following the reviewer's suggestion, we performed a new exact comparison with a variable readout. We use two-neuron, T=1T=1 LIF branches on four or six ordered scalar inputs, with fixed recovery maps, and enumerate all realizable branch assignments and all output assignments obtainable with a variable affine binary score readout. We evaluate a fixed silent query with varying K-branch parameters, and independently variable Q/K branches. The table compares trace capacity with single-step binary-readout capacity; all triplets are ordered raw / scalar / vector.

SettingInputs NNRealizable trace assignmentsRealizable binary output assignmentsOutput-alphabet ceiling 2N2^N
Fixed silent query41 / 33 / 642 / 14 / 1416
Fixed silent query61 / 73 / 1442 / 32 / 3264
Independent Q/K451 / 1,662 / 4,09616 / 16 / 1616
Independent Q/K6183 / 8,720 / 20,73662 / 62 / 6264
AuthorsOpenReview ↗

Response to Reviewer fdQK — Part 2 of 4

The raw silent-query case illustrates why the two capacities must be distinguished: its single zero-score trace admits two constant binary outputs as the readout bias varies. This does not contradict Corollary A.13, which requires a fixed readout. Recovery enlarges the output set from 2 to 14 or 32 assignments in the silent-query setting. With independent Q/K branches, all three conditions cover all 16 binary assignments on four inputs. On six inputs, four threshold neurons permit at most four state changes, and all three conditions realize all 62 binary assignments satisfying this constraint. Restricting every query to be nonzero gives the same output sets. Output capacity thus depends jointly on the interaction architecture and the output observation; once a simple binary output saturates its realizable family, trace capacity still distinguishes the computations preserved by different interaction mechanisms.

Section 4.1 now links Definition 1 to Corollary A.13 and the exact comparison in Appendix B.7.1, Table 9. In particular, the structural bounds and recovery conditions explain how architectures constrain and preserve finite interactions, while the output comparison demonstrates their functional consequences under a specified readout family.

2. Realizable LIF mappings, the two-budget envelope, and MultiSpike representations (W2 / Q2)

W2: “The two-budget envelope may be loose for actual LIF networks. … Testing a small parameterized LIF network against the predicted envelope would provide more direct evidence.”

Q2: “Appendix B.3 uses abstract capped assignments rather than actual LIF mappings. Does the predicted bottleneck still hold for parameterized LIF branches, and how does the analysis extend to SDT-V3's MultiSpike representations?”

Thank you for highlighting realizability in parameterized LIF branches and the applicability to MultiSpike. Our theory provides structural upper bounds together with conditions and concrete constructions that attain or approach them. As stated in Theorem 2 (Eq. 15),

Dreal≤Dcap=min⁡{LQK,LS}.D_{\mathrm{real}}\leq D_{\mathrm{cap}}=\min\lbrace L_{QK},L_S\rbrace.

The two budgets constrain jointly available branch assignments and score assignments, respectively. Section 4.4 and Appendix A.3 already establish that a pair-injective fixed score map preserves every realizable joint Q/K assignment, giving Dreal=∣AQK∣D_{\mathrm{real}}=|\mathcal A_{QK}|. If the capped Q/K assignments are independently realizable, this yields

Dreal=∣AQK∣=LQK=Dcap.D_{\mathrm{real}}=|\mathcal A_{QK}|=L_{QK}=D_{\mathrm{cap}}.

Appendix A.4.4 provides constructive attainability results for parameterized LIF networks. Proposition A.2 states sufficient conditions for attaining the retained-trace product envelope. Corollary A.12 realizes arbitrary count assignments on the available count-code coordinates using persistent one-hot tags, shared linear mappings, and independently parameterized LIF branches. Combined with pair-injective recovery, this attains the corresponding local attention envelope. Accounting for the msms tag coordinates within total width D>msD>ms, the construction and Eq. 22 give

log⁡TraceComb⁡(L)(m)≤(3+2γ)msLDlog⁡(T+1),\log\operatorname{TraceComb}^{(L)}(m)\leq(3+2\gamma)msLD\log(T+1),

log⁡TraceComb⁡(L)(m)≥(3+2γ)msL(D−ms)log⁡(T+1).\log\operatorname{TraceComb}^{(L)}(m)\geq(3+2\gamma)msL(D-ms)\log(T+1).

The attained capacity exponent is at least a fraction 1−ms/D1-ms/D of the upper exponent, approaching one as D/(ms)→∞D/(ms)\to\infty. If tags are supplied outside the counted width, the upper exponent is attained exactly. Thus, the nominal full-network scaling bound is constructively near-tight under explicit architectural conditions.

Following the reviewer's suggestion, we also performed an exact enumeration of a small LIF attention head without slot tags. We use two fixed scalar input slots x=0,1x=0,1, so (m,s)=(1,2)(m,s)=(1,2), with independent Q/K branches, two neurons per branch, T=2T=2, threshold 1, zero initial membrane potential, and count decoding. We enumerate the continuous weight-and-bias family and verify exact rational witnesses through the original recurrence at β∈{1/4,1/2,1}\beta\in\{1/4,1/2,1\}. These leak values yield the same assignment family. Each neuron realizes all nine count assignments on the two inputs; each branch realizes all 81 state assignments, yielding 6,561 joint Q/K assignments. The global alphabet has nine states; the exact exposure cap of five and nominal cap of seven are inactive on two slots and both give LQK=6,561L_{QK}=6,561. The fixed raw/scalar/vector maps have 8/39/81 product classes. Complete score traces are counted using exact integer arithmetic:

AuthorsOpenReview ↗

Response to Reviewer fdQK — Part 3 of 4

(m,s)(m,s)RecoveryRealized traces DrealD_{\mathrm{real}}Envelope DcapD_{\mathrm{cap}}
(1,2)(1,2)Raw9764,096
(1,2)(1,2)Scalar2,9616,561
(1,2)(1,2)Vector6,5616,561

The new LIF example directly demonstrates the predicted two-budget transition. With LQK=6,561L_{QK}=6,561, raw scores are score-limited (Dcap=4,096D_{\mathrm{cap}}=4,096), whereas scalar and vector recovery shift the envelope into the branch-limited regime (Dcap=6,561D_{\mathrm{cap}}=6,561). Vector recovery exactly attains this envelope, demonstrating achievability by a concrete parameterized LIF network without persistent tags. Thus, the same LIF family illustrates both Theorem 2's active-budget transition and Theorem 1's recovery mechanism.

Attainment also follows analytically: the T=2T=2 count decoder has three nonempty drive intervals for every β∈(0,1]\beta\in(0,1], and arbitrary drives z0,z1z_0,z_1 at x=0,1x=0,1 are realized by b=z0b=z_0 and w=z1−z0w=z_1-z_0. Hence each neuron realizes all 32=93^2=9 count assignments; independent branches and pair-injective recovery give the stated equality. Appendix B.7.2 gives the piecewise count formula and construction, with the two-input result in Table 10 and the three- and four-input controls immediately below it. As already explained in Appendix B.3, the crossover concerns the active envelope budget; recovery can also increase realized capacity within an unchanged envelope.

The MultiSpike extension depends on the level of the analysis. Undoing SDT-V3's fixed 1/41/4 scale gives post-MultiSpike coordinates in {0,…,4}\lbrace 0,\ldots,4\rbrace, so Theorem 1's finite-alphabet product analysis applies directly, and assignment counting can use the nominal alphabet Q=5dQ=5^d and an appropriate exposure cap. The geometric formula for constant-drive LIF requires its dynamical assumptions. As already explained in Appendix B.5, an inactive exposure cap still leaves the shared-operand restriction: Q and K each occupy msms slots, whereas scores occupy ms2ms^2 interaction positions.

Appendix B.7.2 relates the local comparison to Corollary A.12, and Appendix B.5 specifies the applicability conditions for MultiSpike. The construction establishes attainable capacity growth, the concrete enumeration verifies local attainment without tags, and the preservation identity identifies exactly what recovery retains.

Reviewer_UERBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the expressive power of spike-driven Transformers from a finite-state perspective. The main argument is that linear-region counting does not capture what happens in downstream Transformer layers once the spiking front end has mapped continuous inputs into finite states. The authors introduce region-combination capacity to measure how many different assignments of interaction traces can be jointly realized over fixed input regions as the downstream parameters vary.

The analysis focuses on two constraints. Shared branches restrict which states can be exposed at the same time, while multiplicative interactions can produce identical scores from different operand pairs. These are combined into a two-budget attention envelope, with further analysis covering gated FFNs, residual connections, and network depth. The paper also includes finite-model enumeration, analytical experiments, and collision analysis on frozen spike-driven Transformers trained on ImageNet.

Critical strengths
  1. I like the way the paper approaches expressivity from the downstream side. Most linear-region analyses focus on how spiking neurons partition the continuous input space, but that does not tell us much about what attention can actually do with the resulting finite states. Even if the front end produces many distinct regions, the downstream layers may not be able to distinguish or combine all of them. Region-combination capacity tries to capture this difference, and I think it is a useful direction for studying spike-driven Transformers. It gives us something more specific to look at than simply counting input regions.

  2. The distinction between shared-state exposure and product distinguishability is probably the strongest part of the theoretical analysis. These are two different constraints, and treating them separately makes the attention bound easier to understand. In particular, Theorem 2 gives Dreal≤min⁡{LQK,LS}D_{\mathrm{real}} \leq \min\{L_{QK}, L_S\} What I find interesting here is that having more possible attention scores does not necessarily mean the model can realize more joint interaction traces. The analysis also goes beyond a single attention operation and considers gated FFNs, residual connections, and deeper networks.

The experiments help illustrate some of these results. In particular, the finite-model enumeration and frozen SDT-V3 experiments show how product collisions arise and how different recovery mechanisms can preserve distinctions that would otherwise be lost. I found these observations useful for understanding what the theoretical bounds are capturing.

Critical weaknesses
  1. My main concern is how region-combination capacity relates to functional expressivity in practice. The metric counts realizable assignments of intermediate interaction traces, rather than distinct input-output functions of the full network. The authors discuss this in Appendix A.4.4, and Corollary A.13 gives a relationship between retained-trace capacity and final-output capacity under a fixed output mapping. Still, I am not entirely sure how much we can infer about the functional expressivity of a standard network from an increase in this metric. The experimental results leave me with a similar question. For SDT-V3-S, vector recovery improves the score-only probe by 1.83 percentage points, and the improvements are not consistently monotonic. Scalar recovery also reduces collision loss without consistently improving classification accuracy. I understand that better classification performance is not required for a theoretical capacity measure to be useful. However, these results make it harder to tell when preserving additional state distinctions actually matters for the network's output. I would like to see a clearer explanation of this connection.

  2. The proposed bounds make sense as upper envelopes, but reaching them depends on whether the required state assignments can actually be realized with the available parameters and architectural constraints. The finite-model experiments in Appendix B.4 demonstrate tightness in several controlled settings. However, the asymptotic attainment construction in Appendix A.4.4 relies on persistent slot-specific tags, sufficiently expressive lookup mappings, and independently addressable parameters. I am not sure how closely these assumptions match the architectures normally used in practice, especially when parameters are shared. The frozen-model experiments are helpful for understanding local score collisions, but they do not directly show how close a complete trained network gets to the theoretical upper bound. I think the paper would benefit from a more concrete discussion of how loose these bounds might become in standard architectures, and what the current experiments can tell us about that gap.

Questions for authors
  1. Corollary A.13 establishes a conditional relationship between retained-trace capacity and final-output capacity. Under what conditions would an increase in region-combination capacity actually lead to more distinct input-output functions? I am also curious how the authors interpret the relatively small and non-monotonic improvements in the score-only probe. Do these results suggest that some of the additional state distinctions are not being used by the downstream network?
  2. The asymptotic attainment construction assumes persistent slot-specific tags and fairly strong realizability conditions. How tight would the two-budget envelope be for a standard spike-driven Transformer without these assumptions? Also, how much can we conclude about this from the current finite-model and frozen-model experiments?
3 comments
AuthorsOpenReview ↗

Response to Reviewer UERB — Part 1 of 2

Dear Reviewer UERB,

We extend our sincere gratitude for dedicating your valuable time to reviewing our paper and for expressing your appreciation and support for our work. In the following, we provide a comprehensive response to your review comments.

1. Practical significance and relationship to functional expressivity (W1 / Q1)

W1: “The practical interpretation of region-combination capacity remains somewhat limited.”

Q1: “Could the authors clarify when an increase in region-combination capacity is expected to produce a meaningful increase in functional expressivity? In particular, how should the relatively modest and non-monotonic improvements in the score-only probe be interpreted in this context?”

Thank you for raising this important question. Our contribution is to establish an architecture-level theory of finite interaction expressivity, rather than to use internal trace capacity as a proxy for task performance. Region-combination capacity characterizes how many distinct interaction patterns an architecture can realize over fixed LIF-induced input regions, capturing an aspect of expressivity that conventional linear-region counting cannot distinguish.

The main value of this framework is to characterize the structural bottlenecks imposed by different architectural constraints and provide guidance on how these constraints may be alleviated. In particular, Theorem 2 establishes a two-budget attention bound,

Dreal≤min⁡{LQK,LS},D_{\mathrm{real}}\leq \min \lbrace L_{QK},L_S \rbrace ,

which separates limitations from shared-branch realizability and multiplicative product collisions. These two regimes have fundamentally different implications for architecture design. In the score-limited regime, improving product distinguishability through recovery can enlarge the capacity envelope. In the branch-limited regime, further reducing product collisions cannot enlarge this envelope; improving the jointly realizable states of shared Q/K branches becomes more relevant. Theorem 3 further propagates these structural restrictions through attention, gated FFNs, residual paths, and network depth. Thus, our theory provides architecture-level upper bounds on finite interaction capacity and characterizes how shared-branch constraints and product collisions affect these bounds, as well as when recovery can improve them.

However, an increase in trace capacity does not necessarily lead to an increase in final functional expressivity. Different internal traces may still produce the same output because subsequent computation can merge them. Corollary A.13 formalizes this relationship: under a fixed front end and a fixed readout, output capacity equals trace capacity if and only if the readout is injective over the realizable trace assignments. Consequently, additional trace assignments increase functional expressivity only when they produce previously unattainable outputs. At the attention stage, pair-injective recovery guarantees that every realizable joint Q/K assignment remains distinguishable in the complete score trace. This removes product collisions as a source of information loss, although subsequent modules may still merge these distinctions.

This distinction also explains the modest and non-monotonic probe results. Vector recovery improves mean score-only Top-1 accuracy by 1.83 and 1.13 percentage points for SDT-V3-S and SDT-V3-L, respectively. However, scalar recovery reduces collision loss while lowering probe accuracy. This is not contradictory: preserving more distinctions increases the available internal information, but does not guarantee that the additional information is relevant to the target labels or can be effectively exploited by a particular readout. Moreover, the probe measures information accessible to a separately trained classifier, rather than end-to-end improvements in the original network (Appendix B.6, Table 5).

The exact output-capacity comparison in Appendix B.7.1 (Table 9) provides a concrete illustration of this distinction: increased trace capacity can produce additional output assignments, but need not do so when the specified output family is already saturated.

In summary, region-combination capacity is an architectural expressivity measure, not a task-performance metric. Its theoretical significance lies in identifying structural limitations, characterizing the conditions under which they can be alleviated, and establishing how internal computational distinctions relate to final functional expressivity.

AuthorsOpenReview ↗

Response to Reviewer UERB — Part 2 of 2

2. Tightness, applicability, and implications for architectural design (W2 / Q2)

W2: “The tightness and applicability of the theoretical bounds to practical architectures need further discussion.”

Q2: “The asymptotic attainment construction uses persistent slot-specific tags and specific realizability assumptions. How tight are the proposed bounds expected to be for standard spike-driven Transformer architectures without these conditions? Could the authors clarify what the existing finite-model and frozen-model experiments establish about this gap?”

Thank you for raising this important question. Our results establish architecture-level upper bounds, but we do not claim that they are uniformly tight for standard SDT architectures. Attaining these bounds requires sufficient joint realizability of the finite states across shared branches and layers.

Corollary A.12 provides a constructive result at the full-network level. By introducing persistent slot-specific tags, the construction preserves the identity of each region–token pair across layers, allowing independently controlled branch assignments and injective recovery. Although the tags consume (ms) dimensions of the total width (D), the resulting capacity exponent reaches at least a fraction (1-ms/D) of the upper exponent, approaching one as (D/(ms)\to\infty). This establishes that our full-network bound can be approached under explicit architectural conditions, rather than assuming that these conditions hold in standard SDTs.

Importantly, tightness is not restricted to this special tag-based construction. Table 10 provides a concrete example of a parameterized LIF attention head without persistent tags. With two input slots and independently parameterized two-neuron Q/K branches at (T=2), vector recovery realizes all 6,561 possible joint score traces, exactly attaining the two-budget envelope: [ D_{\mathrm{real}}=D_{\mathrm{cap}}=6,561. ] In comparison, raw and scalar recovery realize 976 and 2,961 traces, respectively, below their corresponding bounds. This demonstrates that the proposed bound can also be tight in concrete LIF architectures under appropriate conditions, without requiring persistent slot-specific tags.

The finite-model experiments in Appendix B.4 further illustrate how product collisions affect the attainability of the envelope and how pair-injective recovery can remove this restriction. Meanwhile, the frozen-model experiments in Appendix B.6 confirm that product collisions and their recovery also occur in learned SDT representations. These experiments support the relevance of our theoretical mechanisms to practical SNN representations, although they do not establish a uniform tightness ratio for complete trained SDTs.

Together, these results show that our bounds are attainable under explicit conditions and can also be attained by concrete LIF attention heads without special tags, while their tightness for general SDT architectures remains dependent on architectural realizability.

Dear Reviewers,

Thank you again for your time and consideration.

As the rebuttal discussion period is approaching its deadline, I would like to kindly follow up on our responses. We have carefully addressed the concerns raised in the reviews and provided additional clarifications and supporting explanations.

We would greatly appreciate it if you could take a moment to review our rebuttal and let us know if any concerns remain or if further clarification would be helpful before the discussion period closes.

Thank you for your time and consideration.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear Area Chair and Reviewers,

We sincerely thank you for the time and care you devoted to evaluating our work. We have responded in detail to every concern and question, and we believe that these responses, together with the corresponding revisions, address the issues raised and clarify the scope and contribution of our work. Your constructive suggestions have helped us improve the manuscript. We revised the paper accordingly and marked all changes in blue. To make the revisions easy to locate, each point-by-point response ends with precise references to the relevant sections and tables. We are again grateful for your thoughtful feedback and would be happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.