- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the conversion loss of pretrained large language models turned into spiking neural networks follows a scaling law with respect to model size and spike encoding capacity. The authors distinguish ATF mode, which accumulates input before firing, from OCF mode, which fires while receiving input, and propose that temporal unevenness introduced by online timing affects the applicable range of the representation-capacity power law. Experiments are conducted on the OPT series, three corpora, and multiple neuron encodings. Some OCF configurations show good temporal power-law fits under larger time windows and threshold counts.
Critical strengths
-
The paper clearly separates static encoding capacity from online temporal dynamics, which helps explain why conversion results similar to quantization may not directly extend to online spiking computation.
-
The ATF and OCF experiments in MTH share weights and calibration, allowing a fairly direct comparison of the differences between operating modes.
-
The authors examine multiple model sizes, three corpora, and three encoding types, and disclose fitting failures and cases where parameters hit optimization bounds in the appendix, making it easier to judge the scope of the conclusions.
Critical weaknesses
-
The main text defines conversion error as whole-network cross-entropy increase, but Appendix D.3 Eqs. (80)–(82) derive single-neuron MSE. There is no analysis of how local error propagates to cross-entropy, and no clear justification for the additive representation-plus-temporal error decomposition at the cross-entropy level. This weakens the theorem's link to the actual metrics.
-
Theorem 3.1 only gives a sufficient condition for preserving the power law when representation error already follows it; it does not prove no power law exists otherwise. The claim that "a scaling law exists only when the time window and threshold count are large enough" needs stronger support or a narrower wording.
-
ATF MTH model-size fits only reach , and phase coding on WikiText-2 reaches , yet ATF scaling is still claimed strongly. Some OCF fits reach but are judged as no scaling law. The paper needs a unified functional form, fitting range, and decision rule, plus uncertainty and predictive-power analysis.
-
Positive OCF temporal power-law results mainly come from OPT-125M, and cross-model-size OCF experiments only cover the low-threshold region with . So the paper does not directly show restored model-size scaling at high thresholds, nor that the restored exponent matches the ATF result. Using only OPT also limits generality.
Questions for authors
-
Can you give a clear derivation from the single-neuron mean squared error in Eq. (80) to the whole-network cross-entropy increase? What assumptions about error bias, cross-layer correlation, and network sensitivity are needed for this step?
-
Is the condition in Theorem 3.1 only sufficient? If so, would the authors be willing to change "no scaling law exists" to "within the examined range, the representation-capacity power law does not fully explain the observed results"?
-
Can you add cross-model-size OCF experiments at larger , and paired comparisons with the same but different , and test whether the theoretical condition can predict the performance of configurations not used in fitting?
-
Under the calibration constraint , increasing does not raise the single-step maximum firing amplitude. How do the authors reconcile this with the explanation that "a larger threshold count helps handle bursty residuals"? Can this mechanism be verified by measuring actual membrane-potential residuals or backlog?
5 comments
Response to Reviewer A4L1-Part 2
Response to W3:
We thank the reviewer for this important observation. We clarify the differences in fitting behavior between ATF and OCF and further strengthen the consistency of our scaling-law evaluation.
1. The relatively low fitting accuracy of phase coding is partly attributable to its built-in adaptive threshold calibration. Unlike conventional fixed-resolution quantization, the original multi-granularity phase-coding method incorporates learnable coding bases and threshold parameters to better match activation distributions. Such adaptive calibration can introduce model-dependent changes in the effective quantization behavior, potentially weakening the simple power-law relationship with model size. Moreover, the original phase-coding fit on WikiText-2 contains only three model-size points, making the estimated particularly sensitive to individual observations.
2. ATF follows the same representation-limited mechanism as static quantization, for which a simple power law is an approximation rather than an exact universal relationship. As discussed in Section 3.2, ATF eliminates temporal unevenness and behaves analogously to a static quantizer. Its conversion error is therefore influenced by quantization resolution, clipping, activation distributions, and model-dependent error propagation. Consequently, the simple model-size relationship may exhibit imperfect fitting accuracy. Indeed, Appendix C shows that the ATF MTH representation-capacity curves are substantially better described by a saturating scaling function, achieving – across the evaluated configurations. These results support the representation-limited scaling interpretation while indicating that the appropriate functional form and fitting range must be considered. We have therefore clarified that the ATF model-size results demonstrate an approximate scaling trend rather than an exact universal power law.
3. Our OCF results support conditional representation-based scaling, and we further clarify the fitting criteria to avoid interpreting moderate as the complete absence of scaling laws. Our claim is not that no scaling relationship can exist outside the identified conditions, but that the original representation-based power law is not consistently preserved when temporal unevenness becomes significant. In Appendix C, the moderate – values occur at , representing an intermediate regime rather than definitive evidence against all power-law behavior. In contrast, smaller thresholds () exhibit considerably weaker fits, whereas and yield consistently higher fitting accuracy () across all three corpora. This systematic improvement supports our central finding that temporal and representational conditions determine when the representation-based scaling approximation becomes reliable.
Response to W4:
We thank the reviewer for this valuable suggestion. To address these concerns, we have substantially extended our experiments in three complementary directions.
1. We extend the OCF temporal sweep to OPT-6.7B. We additionally evaluate OCF under larger temporal horizons and threshold numbers on OPT-6.7B in Figure OPT-6.7B. On OPT-6.7B, ATF lowers excess loss when m or T is raised independently; OCF, by contrast, improves only when T and m are increased jointly.
2. We also extend the OCF size sweep to and . We extend the OCF experiments to and across OPT models ranging from 125M to 30B as shown in Figure m 8 and T 16.
3.We further supplement our study with results on Llama and Qwen, thereby broadening the evidence across model families. These results are shown in Figure Llama and Qwen.
Response to Reviewer A4L1-Part 3
Response to Q3:
We thank the reviewer for these constructive suggestions. We have extended our experiments to evaluate model-size scaling and the distinct effects of and .
1. We extend the OCF model-size experiments to larger threshold numbers. As discussed in our response to W4, we additionally evaluate OCF with and across OPT models from 125M to 30B.
2. We introduce paired experiments with identical representation capacity but different combinations to isolate the effects of temporal dynamics. Specifically, we compare configurations such as in Figure same product The does not predict OCF's performance. It constrains whether a representation law stays within a tolerance. Raising does not stand in for raising , and a fit on one set of cells does not recover the measured loss on another.
Response to Q4:
We thank the reviewer for this insightful observation. We clarify the role of the threshold number and provide additional membrane-potential measurements to verify the underlying mechanism.
Increasing improves the precision of membrane-potential discharge rather than its maximum discharge capacity, thereby reducing the discretization component of temporal unevenness. We agree that under the calibration constraint , the maximum single-step discharge amplitude remains , independent of . However, increasing reduces the threshold spacing , enabling finer-grained membrane-potential adjustments and reducing the residual error caused by discrete threshold levels. As derived in Appendix D.3, the temporal unevenness bound contains two components:
\lesssim \left(A_M+\frac{B_M}{m^2}\right)T^{-2},$$ where the stochastic backlog component $A_M$ is independent of $m$, while the finite-threshold discretization component decreases as $m^{-2}$. To further verify this mechanism, **the inference residual was measured on OPT-125M at $T=16$**. The residual membrane can be seen in [Figure Membrane residual](https://anonymous.4open.science/r/iclr2027-snn-rebuttal-875B/figures/membrane_vs_m.png). The mean absolute membrane left after the last step is 0.126 at $m=1$, 0.130 at $m=4$, 0.079 at $m=8$, and 0.042 at $m=16$.
Response to Reviewer A4L1-Part 1
We thank the reviewer for recognizing the controlled ATF–OCF comparison and for identifying the specific gaps between the local analysis, the global evaluation metric, and the statistical evidence. These concerns materially affect the strength of the paper's claims, and the responses below distinguish provable statements from empirical hypotheses. Our supplementary experimental results will be provided in the anonymous link https://anonymous.4open.science/r/iclr2027-snn-rebuttal-875B, and we will also include hyperlinks pointing to the corresponding experimental results when responding to specific questions.
Response to W1&Q1:
We thank the reviewer for identifying this important theoretical gap, which is closely related to the concern raised by Reviewer X41F (W2).
We have connected the local neuron-level MSE bound to whole-network cross-entropy using layerwise error propagation and a bias-aware loss expansion. Starting from Appendix D.3, the squared discrepancy EM,loc of an MTH neuron satisfies EM,loc≤KMΨM(T,m) with ΨM(T,m)=(AM+BM/m2)T−2. We compare adjacent ATF/OCF hybrid networks with matched block inputs, aggregate the per-coordinate errors and bound the downstream operators. A telescoping decomposition and Minkowski's inequality then yield
\leq K_{\mathrm{net}}\Psi_M(T,m), \qquad K_{\mathrm{net}}=\left(\sum_j\chi_j\sqrt{\kappa_j}\right)^2.$$ Here $\chi_j$ captures downstream sensitivity and $\kappa_j$ aggregates within-block error bounds. This propagation argument allows arbitrary cross-layer correlations and requires controlled network sensitivity along the evaluated trajectories, including suitable stability conditions around hard firing thresholds. Applying a second-order Taylor expansion of cross-entropy, whose logit Hessian is positive semidefinite with spectral norm at most $1/2$, gives $$E_{M,\mathrm{net}} =F_{\mathrm{OCF}}-F_{\mathrm{ATF}} =b_M+r_M^{\mathrm{CE}},\qquad 0\leq r_M^{\mathrm{CE}}\leq \frac{K_{\mathrm{net}}}{4}\Psi_M(T,m),$$ where $b_M=\mathbb E[\langle\nabla_z\ell(z_{\mathrm{ATF}},y),z_{\mathrm{OCF}}-z_{\mathrm{ATF}}\rangle]$ and $|b_M|\leq\sqrt{2K_{\mathrm{net}}\Psi_M(T,m)}$. The network-level loss decomposition is therefore exact: $E_{M,\mathrm{OCF}}=E_{M,\mathrm{rep}}+b_M+r_M^{\mathrm{CE}}$, with $E_{M,\mathrm{rep}}=F_{\mathrm{ATF}}-F_{\mathrm{ANN}}$. **The global temporal contribution may be positive or negative; only the second-order remainder is necessarily nonnegative.** The full derivation and the explicit regularity and model-sensitivity requirements are provided in the **Corollary E.1 of revised manuscript**. **Response to W2&Q2:** We thank the reviewer for highlighting this important theoretical distinction, which is closely related to the concern raised by Reviewer X41F (Q1). **We clarify that Theorem 3.1 originally provides a sufficient condition and supplement it with a conditional failure result and its contrapositive necessary condition.** As detailed in our response to Reviewer X41F (Q1), we consider preservation of the representation-based law $E_{M,\mathrm{rep}}=C_MN^{\beta_M}(mT)^{-\alpha_M}$ within a relative tolerance $\eta$. Under an additional non-degeneracy lower bound $|E_{M,\mathrm{net}}|\geq\nu_M(A_M+B_M/m^2)T^{-2}$, we prove that the representation-based approximation necessarily fails the strict tolerance test whenever $$T\leq\left[\frac{\nu_MF_M(m)}{\eta C_MN^{\beta_M}}\right]^{\frac1{2-\alpha_M}}, \qquad F_M(m)=A_Mm^{\alpha_M}+B_Mm^{\alpha_M-2}.$$ Consequently, preserving this law requires $$T>\left[\frac{\nu_MF_M(m)}{\eta C_MN^{\beta_M}}\right]^{\frac1{2-\alpha_M}}, \qquad m>\frac1T\left[\frac{\nu_MB_M}{\eta C_MN^{\beta_M}}\right]^{\frac1{2-\alpha_M}}.$$ The necessary and sufficient bounds need not coincide: configurations between the guaranteed-failure and guaranteed-preservation regions remain theoretically undetermined, rather than being classified as having or lacking a scaling law. Furthermore, the local $O(T^{-2})$ bound does not itself imply the non-degeneracy lower bound, and the network-level sufficient criterion also accounts for the first-order bias $b_M$, as explained in our response to A4L1 (W1/Q1). **We have accordingly narrowed the abstract and main-text wording to state that, within the examined range, the representation-capacity power law does not fully explain the observed results outside the well-supported scaling regime.** The complete proof and scope clarification are included in the **Corollary E.4 of revised manuscript**.