acceptodds
Under review as a conference paper at ICLR 2027

Are Scaling Laws Universal in Spiking Large Language Models?

Abstract

The scaling law is a foundational property of large language models (LLMs), dictating predictable performance scaling in foundation models. Spiking LLMs are often presented as neuromorphic analogues of quantized models which convert continuous-valued LLM activations into discrete temporal spike trains. However, whether Spiking LLMs genuinely exhibit consistent scaling laws remains an open and unresolved question. In this paper, we demonstrate that the validity of scaling laws in Spiking LLMs depends fundamentally on whether the system is analyzed under a static conversion regime or a temporal dynamical regime. Under the static accumulate-then-fire (ATF) regime, where synaptic input is fully accumulated prior to firing, conversion acts effectively as a static quantizer. The excess loss exhibits a predictable representation-limited scaling law governed by network scale and spike-code capacity . In contrast, when operating in the temporal dynamical regime via online charge-and-fire (OCF), which represents the asynchronous parallel mode native to neuromorphic hardware, scaling laws become strictly conditional. In this dynamical regime, small time windows and coarse threshold levels let temporal unevenness dominate, causing excess loss to deviate from power-law scaling against or . Power-law scaling is only restored when and code capacity exceed critical dynamic thresholds. Our findings reveal that scaling claims for SLLMs cannot treat them simply as bio-inspired quantized LLMs, but must explicitly specify the dynamical regime and the condition under OCF.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spiking Neural Networks, Scaling law, Language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A4L1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the conversion loss of pretrained large language models turned into spiking neural networks follows a scaling law with respect to model size and spike encoding capacity. The authors distinguish ATF mode, which accumulates input before firing, from OCF mode, which fires while receiving input, and propose that temporal unevenness introduced by online timing affects the applicable range of the representation-capacity power law. Experiments are conducted on the OPT series, three corpora, and multiple neuron encodings. Some OCF configurations show good temporal power-law fits under larger time windows and threshold counts.

Critical strengths
  1. The paper clearly separates static encoding capacity from online temporal dynamics, which helps explain why conversion results similar to quantization may not directly extend to online spiking computation.

  2. The ATF and OCF experiments in MTH share weights and calibration, allowing a fairly direct comparison of the differences between operating modes.

  3. The authors examine multiple model sizes, three corpora, and three encoding types, and disclose fitting failures and cases where parameters hit optimization bounds in the appendix, making it easier to judge the scope of the conclusions.

Critical weaknesses
  1. The main text defines conversion error as whole-network cross-entropy increase, but Appendix D.3 Eqs. (80)–(82) derive single-neuron MSE. There is no analysis of how local error propagates to cross-entropy, and no clear justification for the additive representation-plus-temporal error decomposition at the cross-entropy level. This weakens the theorem's link to the actual metrics.

  2. Theorem 3.1 only gives a sufficient condition for preserving the power law when representation error already follows it; it does not prove no power law exists otherwise. The claim that "a scaling law exists only when the time window and threshold count are large enough" needs stronger support or a narrower wording.

  3. ATF MTH model-size fits only reach R2=0.729–0.787R^2=0.729–0.787, and phase coding on WikiText-2 reaches R2=0.32R^2=0.32, yet ATF scaling is still claimed strongly. Some OCF fits reach R2=0.819–0.885R^2=0.819–0.885 but are judged as no scaling law. The paper needs a unified functional form, fitting range, and decision rule, plus uncertainty and predictive-power analysis.

  4. Positive OCF temporal power-law results mainly come from OPT-125M, and cross-model-size OCF experiments only cover the low-threshold region with m=1,2,4m=1,2,4. So the paper does not directly show restored model-size scaling at high thresholds, nor that the restored exponent matches the ATF result. Using only OPT also limits generality.

Questions for authors
  1. Can you give a clear derivation from the single-neuron mean squared error in Eq. (80) to the whole-network cross-entropy increase? What assumptions about error bias, cross-layer correlation, and network sensitivity are needed for this step?

  2. Is the condition in Theorem 3.1 only sufficient? If so, would the authors be willing to change "no scaling law exists" to "within the examined range, the representation-capacity power law does not fully explain the observed results"?

  3. Can you add cross-model-size OCF experiments at larger mm, and paired comparisons with the same mTmT but different (T,m)(T,m), and test whether the theoretical condition can predict the performance of configurations not used in fitting?

  4. Under the calibration constraint mθM=RMm\theta_M=R_M, increasing mm does not raise the single-step maximum firing amplitude. How do the authors reconcile this with the explanation that "a larger threshold count helps handle bursty residuals"? Can this mechanism be verified by measuring actual membrane-potential residuals or backlog?

5 comments
AuthorsOpenReview ↗

Response to Reviewer A4L1-Part 1

We thank the reviewer for recognizing the controlled ATF–OCF comparison and for identifying the specific gaps between the local analysis, the global evaluation metric, and the statistical evidence. These concerns materially affect the strength of the paper's claims, and the responses below distinguish provable statements from empirical hypotheses. Our supplementary experimental results will be provided in the anonymous link https://anonymous.4open.science/r/iclr2027-snn-rebuttal-875B, and we will also include hyperlinks pointing to the corresponding experimental results when responding to specific questions.

Response to W1&Q1:

We thank the reviewer for identifying this important theoretical gap, which is closely related to the concern raised by Reviewer X41F (W2).

We have connected the local neuron-level MSE bound to whole-network cross-entropy using layerwise error propagation and a bias-aware loss expansion. Starting from Appendix D.3, the squared discrepancy EM,loc\mathcal E_{M,\mathrm{loc}} of an MTH neuron satisfies EM,loc≤KMΨM(T,m)\mathcal E_{M,\mathrm{loc}}\leq K_M\Psi_M(T,m) with ΨM(T,m)=(AM+BM/m2)T−2\Psi_M(T,m)=(A_M+B_M/m^2)T^{-2}. We compare adjacent ATF/OCF hybrid networks with matched block inputs, aggregate the per-coordinate errors and bound the downstream operators. A telescoping decomposition and Minkowski's inequality then yield

\leq K_{\mathrm{net}}\Psi_M(T,m), \qquad K_{\mathrm{net}}=\left(\sum_j\chi_j\sqrt{\kappa_j}\right)^2.$$ Here $\chi_j$ captures downstream sensitivity and $\kappa_j$ aggregates within-block error bounds. This propagation argument allows arbitrary cross-layer correlations and requires controlled network sensitivity along the evaluated trajectories, including suitable stability conditions around hard firing thresholds. Applying a second-order Taylor expansion of cross-entropy, whose logit Hessian is positive semidefinite with spectral norm at most $1/2$, gives $$E_{M,\mathrm{net}} =F_{\mathrm{OCF}}-F_{\mathrm{ATF}} =b_M+r_M^{\mathrm{CE}},\qquad 0\leq r_M^{\mathrm{CE}}\leq \frac{K_{\mathrm{net}}}{4}\Psi_M(T,m),$$ where $b_M=\mathbb E[\langle\nabla_z\ell(z_{\mathrm{ATF}},y),z_{\mathrm{OCF}}-z_{\mathrm{ATF}}\rangle]$ and $|b_M|\leq\sqrt{2K_{\mathrm{net}}\Psi_M(T,m)}$. The network-level loss decomposition is therefore exact: $E_{M,\mathrm{OCF}}=E_{M,\mathrm{rep}}+b_M+r_M^{\mathrm{CE}}$, with $E_{M,\mathrm{rep}}=F_{\mathrm{ATF}}-F_{\mathrm{ANN}}$. **The global temporal contribution may be positive or negative; only the second-order remainder is necessarily nonnegative.** The full derivation and the explicit regularity and model-sensitivity requirements are provided in the **Corollary E.1 of revised manuscript**. **Response to W2&Q2:** We thank the reviewer for highlighting this important theoretical distinction, which is closely related to the concern raised by Reviewer X41F (Q1). **We clarify that Theorem 3.1 originally provides a sufficient condition and supplement it with a conditional failure result and its contrapositive necessary condition.** As detailed in our response to Reviewer X41F (Q1), we consider preservation of the representation-based law $E_{M,\mathrm{rep}}=C_MN^{\beta_M}(mT)^{-\alpha_M}$ within a relative tolerance $\eta$. Under an additional non-degeneracy lower bound $|E_{M,\mathrm{net}}|\geq\nu_M(A_M+B_M/m^2)T^{-2}$, we prove that the representation-based approximation necessarily fails the strict tolerance test whenever $$T\leq\left[\frac{\nu_MF_M(m)}{\eta C_MN^{\beta_M}}\right]^{\frac1{2-\alpha_M}}, \qquad F_M(m)=A_Mm^{\alpha_M}+B_Mm^{\alpha_M-2}.$$ Consequently, preserving this law requires $$T>\left[\frac{\nu_MF_M(m)}{\eta C_MN^{\beta_M}}\right]^{\frac1{2-\alpha_M}}, \qquad m>\frac1T\left[\frac{\nu_MB_M}{\eta C_MN^{\beta_M}}\right]^{\frac1{2-\alpha_M}}.$$ The necessary and sufficient bounds need not coincide: configurations between the guaranteed-failure and guaranteed-preservation regions remain theoretically undetermined, rather than being classified as having or lacking a scaling law. Furthermore, the local $O(T^{-2})$ bound does not itself imply the non-degeneracy lower bound, and the network-level sufficient criterion also accounts for the first-order bias $b_M$, as explained in our response to A4L1 (W1/Q1). **We have accordingly narrowed the abstract and main-text wording to state that, within the examined range, the representation-capacity power law does not fully explain the observed results outside the well-supported scaling regime.** The complete proof and scope clarification are included in the **Corollary E.4 of revised manuscript**.
AuthorsOpenReview ↗

Response to Reviewer A4L1-Part 2

Response to W3:

We thank the reviewer for this important observation. We clarify the differences in fitting behavior between ATF and OCF and further strengthen the consistency of our scaling-law evaluation.

1. The relatively low fitting accuracy of phase coding is partly attributable to its built-in adaptive threshold calibration. Unlike conventional fixed-resolution quantization, the original multi-granularity phase-coding method incorporates learnable coding bases and threshold parameters to better match activation distributions. Such adaptive calibration can introduce model-dependent changes in the effective quantization behavior, potentially weakening the simple power-law relationship with model size. Moreover, the original phase-coding fit on WikiText-2 contains only three model-size points, making the estimated R2R^2 particularly sensitive to individual observations.

2. ATF follows the same representation-limited mechanism as static quantization, for which a simple power law is an approximation rather than an exact universal relationship. As discussed in Section 3.2, ATF eliminates temporal unevenness and behaves analogously to a static quantizer. Its conversion error is therefore influenced by quantization resolution, clipping, activation distributions, and model-dependent error propagation. Consequently, the simple model-size relationship E=CNβE=CN^\beta may exhibit imperfect fitting accuracy. Indeed, Appendix C shows that the ATF MTH representation-capacity curves are substantially better described by a saturating scaling function, achieving R2=0.946R^2=0.946–0.9860.986 across the evaluated configurations. These results support the representation-limited scaling interpretation while indicating that the appropriate functional form and fitting range must be considered. We have therefore clarified that the ATF model-size results demonstrate an approximate scaling trend rather than an exact universal power law.

3. Our OCF results support conditional representation-based scaling, and we further clarify the fitting criteria to avoid interpreting moderate R2R^2 as the complete absence of scaling laws. Our claim is not that no scaling relationship can exist outside the identified conditions, but that the original representation-based power law is not consistently preserved when temporal unevenness becomes significant. In Appendix C, the moderate R2=0.819R^2=0.819–0.8850.885 values occur at m=4m=4, representing an intermediate regime rather than definitive evidence against all power-law behavior. In contrast, smaller thresholds (m=1,2m=1,2) exhibit considerably weaker fits, whereas m≥6m\geq6 and T≥8T\geq8 yield consistently higher fitting accuracy (R2≥0.91R^2\geq0.91) across all three corpora. This systematic improvement supports our central finding that temporal and representational conditions determine when the representation-based scaling approximation becomes reliable.

Response to W4:

We thank the reviewer for this valuable suggestion. To address these concerns, we have substantially extended our experiments in three complementary directions.

1. We extend the OCF temporal sweep to OPT-6.7B. We additionally evaluate OCF under larger temporal horizons TT and threshold numbers mm on OPT-6.7B in Figure OPT-6.7B. On OPT-6.7B, ATF lowers excess loss when m or T is raised independently; OCF, by contrast, improves only when T and m are increased jointly.

2. We also extend the OCF size sweep to m∈{8,16}m\in\{8,16\} and T∈{16,32}T\in\{16,32\}. We extend the OCF experiments to m∈{8,16}m\in\{8,16\} and T∈{16,32}T\in\{16,32\} across OPT models ranging from 125M to 30B as shown in Figure m ≥\geq 8 and T ≥\geq 16.

3.We further supplement our study with results on Llama and Qwen, thereby broadening the evidence across model families. These results are shown in Figure Llama and Qwen.

AuthorsOpenReview ↗

Response to Reviewer A4L1-Part 3

Response to Q3:

We thank the reviewer for these constructive suggestions. We have extended our experiments to evaluate model-size scaling and the distinct effects of TT and mm.

1. We extend the OCF model-size experiments to larger threshold numbers. As discussed in our response to W4, we additionally evaluate OCF with m∈{8,16}m\in\{8,16\} and T∈{16,32}T\in\{16,32\} across OPT models from 125M to 30B.

2. We introduce paired experiments with identical representation capacity mTmT but different (T,m)(T,m) combinations to isolate the effects of temporal dynamics. Specifically, we compare configurations such as (T,m)=(8,16),(16,8),(32,4),(64,2)(T,m)=(8,16),(16,8),(32,4),(64,2) in Figure same product The does not predict OCF's performance. It constrains whether a representation law stays within a tolerance. Raising TT does not stand in for raising mm, and a fit on one set of cells does not recover the measured loss on another.

Response to Q4:

We thank the reviewer for this insightful observation. We clarify the role of the threshold number mm and provide additional membrane-potential measurements to verify the underlying mechanism.

Increasing mm improves the precision of membrane-potential discharge rather than its maximum discharge capacity, thereby reducing the discretization component of temporal unevenness. We agree that under the calibration constraint mθM=RMm\theta_M=R_M, the maximum single-step discharge amplitude remains RMR_M, independent of mm. However, increasing mm reduces the threshold spacing θM=RM/m\theta_M=R_M/m, enabling finer-grained membrane-potential adjustments and reducing the residual error caused by discrete threshold levels. As derived in Appendix D.3, the temporal unevenness bound contains two components:

\lesssim \left(A_M+\frac{B_M}{m^2}\right)T^{-2},$$ where the stochastic backlog component $A_M$ is independent of $m$, while the finite-threshold discretization component decreases as $m^{-2}$. To further verify this mechanism, **the inference residual was measured on OPT-125M at $T=16$**. The residual membrane can be seen in [Figure Membrane residual](https://anonymous.4open.science/r/iclr2027-snn-rebuttal-875B/figures/membrane_vs_m.png). The mean absolute membrane left after the last step is 0.126 at $m=1$, 0.130 at $m=4$, 0.079 at $m=8$, and 0.042 at $m=16$.
Reviewer_X41FOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper classifies the conversion of large spiking models into two types, ATF and OCF, and analyzes how the additional losses of these two types of models vary with model size and spike representation capacity. Experiments show that the conversion loss of ATF generally decreases or plateaus as spike representation capacity increases, whereas OCF is affected by interference from temporal dynamics and exhibits approximately power-law decay again only in some configurations with sufficiently many time steps and threshold levels.

Critical strengths
  1. The research direction of this paper is novel and meaningful. Scaling laws for large spiking models are indeed an important direction to explore.
  2. The theoretical analysis is based on proofs.
Critical weaknesses
  1. ATF is essentially no different from a quantized ANN, and its scaling laws have already been studied in papers such as “Low-Bit Quantization Favors Undertrained LLMs.[1]” Therefore, the core contribution of this paper mainly lies in analyzing the scaling laws of the OCF quantization method.
  2. The relationship between the local mean squared error in Eq. (80) and the cross-entropy increase in the main text, as well as the required assumptions, needs further clarification. Although the OCF–ATF difference is a nonnegative local mean squared error, this does not guarantee that the global cross-entropy difference is also nonnegative. In some cases, OCF may have lower error than ATF.
  3. The conclusion that “OCF recovers power-law decay at larger T and m” is mainly based on experiments with a 125M model, and experimental results on larger models are lacking.
  4. The largest model used across all experiments has only 30B parameters, far fewer than current mainstream large models. Whether the conclusions of this paper apply to models with more parameters remains debatable.

[1]Ouyang X, Ge T, Hartvigsen T, et al. Low-bit quantization favors undertrained LLMs[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 32338-32348.

Questions for authors
  1. Does Theorem 3.1 provide sufficient conditions or necessary conditions? That is, does it mean that “satisfying these conditions guarantees recovery of the original scaling law,” or that “a scaling law can exist only if these conditions are satisfied”? The authors are advised to clarify the wording and ensure consistency with the relevant statements in the abstract.
7 comments
AuthorsOpenReview ↗

Response to Reviewer X41F-Part 1

We thank the reviewer for recognizing the importance of studying scaling laws for spiking LLMs and for identifying the main contribution as the distinction between static representation and temporal dynamics. We address the theoretical scope and missing large-model evidence below. Our supplementary experimental results will be provided in the anonymous link https://anonymous.4open.science/r/iclr2027-snn-rebuttal-875B, and we will also include hyperlinks pointing to the corresponding experimental results when responding to specific questions.

Response to W1:

We thank the reviewer for this insightful comment and agree that ATF shares fundamental similarities with conventional quantization. Our primary contribution is not to establish a new scaling law for ATF, but to reveal the conditional nature of scaling laws under OCF, with ATF serving as an essential reference for isolating temporal effects. As correctly pointed out by the reviewer, ATF aggregates temporal inputs before spike generation and is therefore functionally analogous to static quantization, whose scaling behavior has been investigated in prior work. We explicitly acknowledge this connection and use ATF as a controlled reference rather than claiming its scaling behavior as an independent theoretical novelty. Specifically, ATF characterizes representation-limited conversion error, whereas OCF additionally introduces temporal unevenness arising from causal membrane integration and spike emission. By comparing these two regimes under identical weights, calibration, and representation capacity, we demonstrate that scaling behavior under OCF cannot be determined solely by nominal encoding capacity. More importantly, our theoretical analysis establishes sufficient conditions on the temporal horizon TT and threshold number mm under which representation-based scaling behavior can be preserved despite temporal dynamics. Our experiments further investigate how violations of these conditions are associated with deviations from the representation-based scaling relationship. Therefore, the novelty of our work lies in extending the scaling-law perspective from static representation capacity to conditional temporal dynamics, with ATF providing the necessary foundation for this distinction.

Response to W2:

We thank the reviewer for highlighting this important theoretical issue.

We have extended the neuron-level MSE bound to network-level cross-entropy and explicitly accounted for the first-order loss bias. Appendix D.3 establishes the local activation error EM,loc≤KMΨM(T,m)\mathcal E_{M,\mathrm{loc}}\leq K_M\Psi_M(T,m), where ΨM(T,m)=(AM+BM/m2)T−2\Psi_M(T,m)=(A_M+B_M/m^2)T^{-2}. We first compare adjacent hybrid networks that differ only in the conversion mechanism of one block and apply the local bound to their matched inputs. Under bounded downstream sensitivity along the evaluated trajectories, the hybrid-network telescoping argument and Minkowski's inequality give

\leq K_{\mathrm{net}}\Psi_M(T,m),\qquad K_{\mathrm{net}}=\left(\sum_{j=1}^{J}\chi_j\sqrt{\kappa_j}\right)^2,$$ where $\kappa_j$ aggregates the local errors within block $j$ and $\chi_j$ bounds the sensitivity of its downstream map. These bounds follow from the operators in normalization, attention, feed-forward and residual blocks on a controlled forward region; hard thresholds require a stability or margin condition. We then use the convexity and $1/2$-smoothness of token-level cross-entropy with respect to logits. Writing $E_{M,\mathrm{net}}=F_{\mathrm{OCF}}-F_{\mathrm{ATF}}$, its exact Taylor decomposition and bounds are $$E_{M,\mathrm{net}}=b_M+r_M^{\mathrm{CE}},\qquad 0\leq r_M^{\mathrm{CE}}\leq\frac{K_{\mathrm{net}}}{4}\Psi_M(T,m),$$ $$b_M=\mathbb E\\!\left[\left\langle \nabla_z\ell(z_{\mathrm{ATF}},y), z_{\mathrm{OCF}}-z_{\mathrm{ATF}} \right\rangle\right],\qquad \lvert b_M\rvert\leq\sqrt{2K_{\mathrm{net}}\Psi_M(T,m)}.$$ **Thus, the second-order loss contribution is nonnegative and decays as $T^{-2}$, whereas the overall cross-entropy difference can have either sign because of $b_M$.** Preserving the original representation-based scaling law requires controlling both the bias-to-representation-error ratio and the network sensitivity across the considered model sizes. The complete layerwise derivation, operator bounds, and cross-entropy analysis are provided in the **Corollary E.1 of revised manuscript**.
AuthorsOpenReview ↗

Response to Reviewer X41F-Part 2

Response to W3:

We extended the OCF temporal sweep to OPT-6.7B to test whether the OPT-125M decay persists at this scale. In the original manuscript, the positive evidence for OCF temporal scaling primarily comes from OPT-125M. To examine whether this phenomenon persists at larger model scales, we additionally conduct OCF experiments on OPT-6.7B, which is shown in Figure OPT-6.7B. For OPT-6.7B, it shows that ATF lowers loss when m or T is raised independently; while OCF improves only when T and m are increased jointly.

Response to W4:

We thank the reviewer for highlighting the importance of validating our findings on larger and more recent LLMs. We added a few WikiText-2 cells on OPT-66B and OPT-175B, which are larger models in the same OPT family. These results are shown in Table Models beyond 30B. The measured cells use the same conversion. The measured excesses are 8.94 for OPT-66B ATF at (4,8)(4,8), 6.35 for OPT-66B OCF at (16,8)(16,8), and 8.68 to 8.92 for the four OPT-175B OCF cells at T,m∈{2,4}T,m \in \{2,4\}. Response to Q1:

We thank the reviewer for pointing out this important distinction between sufficient and necessary conditions.

1. The original Theorem 3.1 states a sufficient condition, and we have clarified its scope at the network-loss level. The theorem compares an O(T−2)O(T^{-2}) neuron-level temporal-error bound with the representation-based law EM,rep=CMNβM(mT)−αME_{M,\mathrm{rep}}=C_MN^{\beta_M}(mT)^{-\alpha_M}. As detailed in our response to W2, the global cross-entropy deviation additionally contains a first-order bias bMb_M; accordingly, the network-level sufficient criterion controls both ∣bM∣/EM,rep|b_M|/E_{M,\mathrm{rep}} and the second-order remainder. We have revised the abstract and theoretical statements to distinguish preservation of the original representation-based approximation from the possible existence of other scaling relationships.

2. We further derive a conditional necessary criterion by first establishing a regime where representation-based scaling must fail. Under the explicit network-level non-degeneracy condition

\nu_M\left(A_M+\frac{B_M}{m^2}\right)T^{-2}, \qquad \nu_M>0,$$ let $\rho_M=|E_{M,\mathrm{OCF}}-E_{M,\mathrm{rep}}|/E_{M,\mathrm{rep}}$ and $F_M(m)=A_Mm^{\alpha_M}+B_Mm^{\alpha_M-2}$. Then $$\rho_M\geq \frac{\nu_M}{C_M}F_M(m)N^{-\beta_M}T^{\alpha_M-2}.$$ If this lower bound is at least a prescribed tolerance $\eta$, the original representation-based law cannot satisfy the strict relative-error requirement $\rho_M<\eta$. Taking the contrapositive yields the necessary thresholds $$T>\left[\frac{\nu_MF_M(m)}{\eta C_MN^{\beta_M}}\right]^{\frac{1}{2-\alpha_M}}, \qquad m>\frac1T\left[\frac{\nu_MB_M}{\eta C_MN^{\beta_M}}\right]^{\frac{1}{2-\alpha_M}}.$$ The non-degeneracy lower bound is an additional condition, not a consequence of the neuron-level upper bound. The necessary and network-level sufficient bounds delimit a guaranteed-failure region, a guaranteed-preservation region, and an intermediate region that is not resolved by these estimates. **This analysis concerns preservation of the representation-based law within a specified tolerance; it does not rule out other scaling laws.** The proof and the corresponding clarification of the theorem's scope are included in the **Corollary E.4 of revised manuscript**.
AuthorsOpenReview ↗

Thank You for Your Valuable Feedback and Further Discussion

Dear Reviewer X41F,

Thank you again for your thoughtful feedback, particularly regarding the connection between neuron-level MSE and network-level cross-entropy, and the distinction between sufficient and necessary scaling conditions.

In response to your concerns, we have developed two additional theoretical results (Corollaries E.1 and E.4), including a bias-aware network-level error analysis and complementary necessary conditions for preserving representation-based scaling. We have also extended our experiments to OPT-6.7B, OPT-66B, and OPT-175B.

As the discussion period is approaching its end, we would be very grateful for any further feedback on whether these theoretical extensions and additional experiments address your concerns. We would be happy to discuss any remaining questions and provide further explanations if helpful.

Thank you very much for your time and constructive suggestions.

Reviewer_qPJSOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether ANN-to-SNN converted LLMs follow scaling laws in their excess loss E=FSNN−FANNE = F_{\text{SNN}} - F_{\text{ANN}}, borrowing the QiD framing from the quantization scaling-law literature. The central move is to split conversion into two regimes: accumulate-then-fire (ATF), which is essentially a static quantizer, and online charge-and-fire (OCF), where causal firing adds an extra "temporal unevenness" error.

The authors claim that ATF gives a smooth law in model size NN and code capacity L(T,ϕ)L(T, \phi). Under OCF, which they test only with MTH neurons, the law becomes "conditional" and only shows up when both TT and mm are large. On the theory side, a reflected-random-walk bound on the unevenness error and Thm 3.1 give sufficient conditions for the law to survive. Experiments convert OPT models from 125M to 30B with GIF, phase-coding, and MTH neurons, and evaluate on WikiText-2, RefinedWeb, and Dolma.

Critical strengths
  • The question is timely and interesting, and the ATF vs OCF split is a clear, useful framing. Pushing the field to state the dynamical regime instead of saying "spiking LLMs scale" is healthy, and pointing out that many "spiking" LLMs are effectively GEMM-style quantizers is worth saying out loud.
  • The sweep is wide: three neuron families, three corpora, and models up to 30B.
  • The decoupling intuition is nice: more thresholds increase resolution but not per-step discharge capacity (mθ=Rm\theta = R). That explains why mm and TT aren't interchangeable online.
Critical weaknesses

My biggest concern is that the "laws" are fitted in a regime where the converted models are basically broken, and the conversion itself may be the cause:

  • In Fig. 2 the ATF excess loss is ~5–7 nats, and the OCF m=1m = 1 floor is ~5.26 nats on WikiText-2. That puts SNN loss around 8–10 nats, while ln⁡(vocab size)≈10.8\ln(\text{vocab size}) \approx 10.8. Fitting or not fitting power laws near random-guess loss isn't very informative; I'd much rather see the analysis where excess loss is small (say <0.5< 0.5 nats), which right now only phase coding reaches.
  • MTH (T,m)=(4,8)(T, m) = (4, 8) gives about 65 levels (roughly 6 bits), yet it produces 5+ nats of excess loss. That's surprisingly bad compared to PTQ and to what SpikeLLM and phase-coding papers report. OPT models at 6.7B and above are known for big activation outliers (LLM.int8()), so per-tensor min-max calibration with uniform thresholds might be the real culprit, which could also explain β>0\beta > 0 and the 13B outlier point.
  • Calibration details are basically missing (which tensors, what granularity, how big the calibration set is, how attention, softmax, and LayerNorm are handled). Also, everything is post-training conversion of a single, older model family, so "Spiking LLMs" in the title feels broader than the evidence; Pythia or Llama/Qwen would make the conclusions much more convincing.
Questions for authors
  1. What are the absolute FANNF_{\text{ANN}} and FSNNF_{\text{SNN}} values in Figs. 2, 5, and 6? Can you redo the key analysis in a regime where excess loss is small?
  2. How exactly is calibration done (granularity, calibration set, which ops are spiking, how attention and LayerNorm are handled)? Does outlier handling (per-channel ranges, clipping, SmoothQuant-style tricks) change β\beta or the 13B point?
  3. Can you show the OCF NN-sweep at large mm (≥8\ge 8) and TT (≥16\ge 16) to actually demonstrate that the NN-law reappears?
5 comments
AuthorsOpenReview ↗

Response to Reviewer qPJS-Part 1

We thank the reviewer for recognizing the value of distinguishing ATF from OCF, the breadth of the experiments, and the conceptual separation between representational resolution and temporal discharge capacity. We agree that a scaling analysis is more compelling when the converted models retain useful predictive quality. Below, we clarify the claims supported by the current manuscript and specify the additional evidence required. Our supplementary experimental results will be provided in the anonymous link https://anonymous.4open.science/r/iclr2027-snn-rebuttal-875B, and we will also include hyperlinks pointing to the corresponding experimental results when responding to specific questions.

Response to W1:

We thank the reviewer for this insightful comment. We would like to clarify the underlying reasons for the relatively high conversion loss and provide additional evidence supporting our conclusions.

1. The high OCF loss under insufficient temporal capacity is precisely the phenomenon motivating our conditional scaling-law analysis. Unlike static quantization, OCF introduces temporal unevenness arising from causal membrane integration and spike emission. Our analysis shows that the representation-based scaling law is preserved when temporal unevenness is sufficiently small relative to representation error. Conversely, when the temporal horizon TT or threshold number mm are insufficient, temporal unevenness may become dominant, leading to substantial conversion degradation. In this regime, increasing the nominal representation capacity does not necessarily produce the expected power-law behavior. This explains why the OCF configurations with small TT or mm in our experiments exhibit relatively high loss and irregular scaling trends. Therefore, these observations directly motivate our central contribution: identifying the conditions under which representation-based scaling laws remain valid in Spiking LLMs.

2. Larger MTH codes lower the excess on some models. We evaluated MTH at larger TT and mm in Figure ATF excess and model size, Figure OCF excess and model size. At ATF (32,16)(32,16), the excess nat is 0.123 on OPT-1.3B and 0.328 on OPT-2.7B. OPT-13B and OPT-30B remain near 4.55. OCF excess falls below 0.5 on OPT-125M, for example 0.088 at (128,16)(128,16).

3. The relatively high ATF loss primarily reflects limited representation capacity, and studying scaling behavior at relatively high loss is also common in the literature. Unlike OCF, ATF does not introduce additional temporal unevenness, and its conversion loss is primarily determined by representation errors, including quantization and clipping. Relatively large loss values do not inherently invalidate scaling-law analysis, as demonstrated by previous LLM scaling studies.** Scaling laws characterize systematic relationships between loss and scaling variables, rather than requiring loss values below a fixed threshold. For example, Kaplan et al. [1] (Figs. 20-21) analyze per-token losses reaching approximately 8 nats, while Hoffmann et al. [2] (Fig. 2) investigate training losses reaching approximately 6 nats. Similarly, Shen et al. [3] (Fig. 1) study scaling behavior over losses of approximately 6.5 nats, and Yan et al. [4] (Figs. 3 and 7) examine cross-entropy behavior, including high-loss training stages. More directly related to our work, Ouyang et al. [5] (Figs. 3–4) investigate quantization-induced excess loss reaching approximately 5 nats while establishing scaling relationships. Although absolute loss and conversion-induced excess loss are different metrics, these studies illustrate that relatively large loss values do not necessarily preclude meaningful scaling analysis.

References

[1] Kaplan, J., et al. "Scaling Laws for Neural Language Models." arXiv:2001.08361, 2020.

[2] Hoffmann, J., et al. "Training Compute-Optimal Large Language Models." NeurIPS, 2022.

[3] Shen, X., et al. "Scaling Laws for Linear Complexity Language Models." EMNLP, 2024.

[4] Yan, J., et al. "What Scales in Cross-Entropy Scaling Law?" ICLR, 2026.

[5] Ouyang, X., et al. "Low-Bit Quantization Favors Undertrained LLMs." ACL, 2025.

AuthorsOpenReview ↗

Response to Reviewer qPJS-Part 2

Response to W2&Q2:

We thank the reviewer for these valuable comments. We agree that calibration details and activation outliers are important for understanding conversion accuracy and the observed scaling behavior. We clarify our calibration procedure and introduce additional ablation experiments as follows.

1.Comparing MTH directly against PTQ methods at a comparable bit-width, or against SpikeLLM and phase coding, is not an apples-to-apples comparison. On the one hand, MTH amounts to a form of static activation quantization, whereas most PTQ methods either quantize weights only or compute activation scales dynamically. On the other hand, the SpikeLLM and phase-coding papers incorporate additional techniques, such as salient channels and learnable coding bases, whereas our focus is on the differences between spike codings themselves.

2. Our calibration procedure uses a consistent representation budget to isolate the effect of model scale on conversion error. Specifically, we convert the non-embedding transformer stack of pretrained OPT models and estimate activation ranges using a calibration set. For MTH neurons, the calibrated activation range [amin⁡,amax⁡][a_{\min},a_{\max}] determines the center cM=(amax⁡+amin⁡)/2c_M=(a_{\max}+a_{\min})/2 and half-range RM=(amax⁡−amin⁡)/2R_M=(a_{\max}-a_{\min})/2, with uniformly spaced thresholds satisfying θM=RM/m\theta_M=R_M/m. This yields LM=2mT+1L_M=2mT+1 representable levels, while keeping the same (T,m)(T,m) across model sizes to ensure a consistent encoding budget. For MTH, ATF and OCF use identical weights and calibration parameters, differing only in their temporal dynamics. We will further clarify the exact calibration granularity, calibration dataset and sample count, converted operations, and treatment of attention, Softmax, and LayerNorm in the revised manuscript. Importantly, the nominal number of representable levels does not directly guarantee PTQ-equivalent accuracy, since conversion errors also depend on activation distributions, clipping, and error propagation.

3. We additionally evaluate whether outlier-aware calibration changes the scaling exponent and the anomalous OPT-13B result. Following the reviewer's suggestion, we introduce three calibration variants: (i) per-channel activation ranges, (ii) percentile-based activation clipping, and (iii) SmoothQuant-style activation smoothing. We conduct these experiments under the same MTH configuration (T,m)=(4,8)(T,m)=(4,8), keeping the model weights, evaluation data, and encoding budget consistent wherever applicable. These results is shown in Figure Calibration part. We first compare the excess loss of OPT-13B under different calibration strategies to determine whether the anomalous point can be mitigated. We then repeat the calibration variants across the OPT model-size sweep and refit the scaling relationship E=CNβE=CN^\beta to examine changes in β\beta and fitting quality. We additionally report the deviation of the 13B point from the fitted scaling curve to quantify its sensitivity to activation outliers. On the five OPT models with an archive point, E=CNβE=CN^{\beta} gives β=0.073\beta=0.073, R2=0.785R^2=0.785 for the original calibration; β=0.105\beta=0.105, R2=0.485R^2=0.485 for per-channel ranges; β=0.077\beta=0.077, R2=0.346R^2=0.346 for 99.9% clipping; and β=0.015\beta=0.015, R2=0.053R^2=0.053 for SmoothQuant at α=0.5\alpha=0.5. Leaving OPT-13B out, its excess lies 1.06 nats above the original curve, 0.33 below the per-channel curve, 3.08 above the percentile curve, and 0.86 below the SmoothQuant curve. Per-channel calibration lowers the 13B excess and does not raise R2R^2.

Response to W3:

We thank the reviewer for this valuable suggestion. Llama and Qwen measurements extend the same conversion beyond OPT. As detailed in our response to W2/Q2, we have clarified the calibration procedure, including its granularity, calibration data, converted operations, and treatment of attention and normalization layers. To address the concern regarding model diversity, we have further extended our experiments to two widely adopted LLM families, Llama and Qwen, which differ from OPT in their architectures and training configurations. The results are shown in Figure Qwen and Llama. On Qwen2.5 from 0.5B to 14B, OCF excess at m=8m=8 stays between about 6 and 11 nats at both T=16T=16 and T=32T=32. No Qwen or Llama OCF cell has excess below 2 nats. The smallest are 3.36 on Qwen2.5-0.5B at (64,64)(64,64) and 5.49 on Llama-2-13B at (32,16)(32,16). Two ATF cells fall below 1 nat: 0.476 on Qwen2.5-0.5B at (64,64)(64,64) and 0.909 on Llama-3-8B at (64,64)(64,64).

AuthorsOpenReview ↗

Response to Reviewer qPJS-Part 3

Response to Q1:

We thank the reviewer for this suggestion. We provide the absolute loss values. The low-excess cells cover only part of the model range, as detailed below.

1. We additionally report the absolute ANN and SNN losses to provide a more comprehensive assessment of conversion quality. In the original manuscript, we focus on the conversion-induced excess loss E=FSNN−FANNE=F_{\mathrm{SNN}}-F_{\mathrm{ANN}}, following the quantization-induced degradation (QiD) literature, to isolate the effect of SNN conversion from the intrinsic performance of the pretrained ANN. However, we agree that reporting absolute losses is important for evaluating the practical quality of the converted models. We therefore supplement Figs. 2, 5, and 6 with the corresponding FANNF_{\mathrm{ANN}}, FSNNF_{\mathrm{SNN}}, and excess loss values, which as shown in Table Absolute losses. All measurements are computed on identical evaluation sequences to ensure fair comparisons. These additional results allow readers to directly assess both the scaling behavior and the absolute language-modeling performance of each configuration.

Response to Q3:

We thank the reviewer for this valuable suggestion. We extended the sweep range to m∈{8,16}m\in \lbrace 8,16 \rbrace and T∈{16,32}T\in \lbrace 16,32 \rbrace on OPT models from 125M to 30B. These results are shown in Figure m ≥\geq 8 and T ≥\geq 16. This sweep is as discussed in the response to W1. The larger m and T do help explore the behavioral limits of models.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Manuscript Revisions and Additional Experimental Results

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. During the rebuttal period, we have substantially extended our theoretical analysis, conducted additional experiments, and clarified the scope of our conclusions.

To facilitate further discussion, we summarize the major revisions and additional experimental results below.

1. Summary of Manuscript Revisions

  • Extension from neuron-level MSE to network-level cross-entropy (Corollary E.1). We establish a theoretical connection between local temporal unevenness and global cross-entropy through layerwise error propagation and a bias-aware loss decomposition, explicitly accounting for network sensitivity and cross-layer correlations.

  • Complementary necessary-condition analysis (Corollary E.4). Beyond the original sufficient conditions in Theorem 3.1, we derive a conditional failure criterion and its corresponding necessary conditions on TT and mm, further characterizing the admissible regime for representation-based scaling under OCF.

  • Detailed implementation and calibration protocols (Appendix E.2). We have added a dedicated subsection describing the implementation of spiking layers, temporal embeddings, nonlinear activation functions, attention, Softmax, LayerNorm, and RMSNorm in OPT, Llama, and Qwen. We further specify the calibration dataset, sample size, calibration granularity, activation-range estimation, and mean-square-error-based scale refinement.

2. Summary of Additional Experimental Results

  • Low-excess-loss experiments ( Figure ATF excess and model size, Figure OCF excess and model size): Larger MTH configurations achieve ATF excess losses of 0.123 and 0.328 nats on OPT-1.3B and OPT-2.7B, respectively, and an OCF excess loss of 0.088 nats on OPT-125M.

  • Additional calibration ablations ( Figure Calibration part): We evaluate per-channel calibration, percentile clipping, and SmoothQuant-style smoothing, examining their effects on the scaling exponent, fitting accuracy, and the anomalous OPT-13B result.

  • Extended OCF temporal scaling experiments (Figure OPT-6.7B): We supplement the temporal sweep on OPT-6.7B to investigate whether the conditional behavior observed on OPT-125M extends to larger models.

  • High-threshold OCF model-size experiments (Figure m ≥\geq 8 and T ≥\geq 16): We extend the model-size sweep to m∈{8,16}m\in\{8,16\} and T∈{16,32}T\in\{16,32\} across OPT models from 125M to 30B.

  • Additional large-scale model evaluations (Table Models beyond 30B): We extend the empirical investigation beyond 30B parameters by including OPT-66B and OPT-175B.

  • Additional model families (Figure Qwen and Llama): We conduct experiments on Llama and Qwen to investigate the applicability and limitations of the observed scaling relationships beyond OPT.

  • Matched representation-capacity experiments (Figure same product): We compare different (T,m)(T,m) configurations with identical mTmT, further examining the distinction between nominal representation capacity and temporal dynamics.

  • Membrane-potential residual measurements (Figure Membrane residual): On OPT-125M with T=16T=16, the mean absolute terminal membrane residual decreases from 0.126 at m=1m=1 to 0.042 at m=16m=16, providing additional evidence for the role of threshold resolution.

  • Absolute cross-entropy measurements (Table Absolute losses): We additionally report the absolute ANN and SNN losses corresponding to the key experimental figures.

All supplementary experimental results are available in our anonymous repository.

We believe these revisions strengthen the theoretical foundation and empirical scope of our work while clarifying its limitations. We greatly appreciate the reviewers' time and consideration, and we would welcome any further questions or suggestions during the discussion period.

Area_Chair_W6EzOpenReview ↗

reminder

Dear reviewers,

This is a gentle reminder to read the authors’ rebuttals and respond in the discussion.

Thank you for your time and effort throughout the review process, and for your contributions to the community.

Best Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.