acceptodds
Under review as a conference paper at ICLR 2027

When Weak Attention Goes Silent: Repairing Context-Amplified Attention Distortion in Spiking Language Models

Abstract

Spiking neural networks (SNNs) offer sparse, event-driven inference for large language models (LLMs). Converting pretrained artificial neural networks (ANNs) to SNNs with few timesteps can preserve performance on mainstream language-modeling and question-answering benchmarks, yet we find that converted models increasingly fail to use relevant information as input length grows. Controlled interventions identify the few-spike encoding of post-Softmax attention as the dominant recoverable bottleneck: its zero-response region suppresses weak but collectively substantial attention probabilities. Uniformly rescaling the response to recover these probabilities compromises coverage of larger ones. We propose the Downstream-Calibrated Phase Neuron (DCPN) to repair both ends of the response at a fixed deployment budget. DCPN adjusts the final phase's threshold and readout to make weak probabilities accessible with appropriately small outputs, while coordinating earlier-phase readouts to preserve coverage of salient probabilities. It screens candidate responses by reconstructing attention-weighted value outputs, then confirms them using validation negative log-likelihood in the progressively converted model. DCPN changes only post-Softmax response parameters, preserving pretrained weights, timestep count, response-parameter storage, and few-spike inference structure. It improves conventional language-modeling and short-input task performance across evaluated settings. At eight timesteps, RULER and NeedleBench scores increase by 38.69 and 52.97 percentage points on LLaMA-2-7B at 4K, and by 24.44 and 38.07 points on LLaMA-3-8B at 8K, respectively, relative to the conversion baseline.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Network, Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_gXkkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the performance degradation of ANN-to-SNN converted language models on long-context tasks. It identifies the loss of weak but collectively important attention probabilities in few-spike post-Softmax encoding as the primary bottleneck. To address this, the authors propose the Downstream-Calibrated Phase Neuron (DCPN), which optimizes phase thresholds and readouts to recover weak attention signals while preserving salient responses. Without modifying pretrained weights, timestep counts, or storage requirements, DCPN improves both standard language modeling and long-context understanding. Experiments demonstrate substantial gains on RULER and NeedleBench, with improvements of up to 52.97 percentage points at eight timesteps.

Critical strengths
  1. This paper is in good written and easy to follow.

  2. This paper focuses on SNN, a bio-inspired method, which is an interesting topic to save energy.

  3. In the experimental results, this paper improves results in LLAMA-2 and LLAMA-3 in the NeedleBench compared with the proposed baselines.

Critical weaknesses
  1. More comprehensive experimental comparisons are needed. Beyond the baseline considered in this paper, the authors should provide more detailed experimental comparisons with existing SNN attention mechanisms, along with further theoretical analysis.

  2. The effectiveness of SNN attention on modern long-context tasks deserves further investigation. For example, modern LLMs have demonstrated strong reasoning capabilities. However, this work primarily focuses on LLaMA-based models and lacks further exploration of more challenging long-context / response tasks, particularly those requiring complex reasoning.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Reply to Reviewer gXkk — W1. Additional SNN Attention Comparisons and Theoretical Analysis

We thank the reviewer for suggesting broader comparisons with existing spiking attention mechanisms and further theoretical analysis.

1. Additional comparisons with existing SNN methods.

Beyond DMPC, we evaluated LAS (T=16), SpikedAttention (T=40), FAS (T=16), and SpikeLLM (T=2, W4A4) on the same LLaMA models and long-context benchmarks.

LLaMA-2-7B:

MethodWT2 PPL ↓Needle 1K ↑Needle 2K ↑Needle 4K ↑RULER 4K ↑
ANN5.472183.0173.7567.2275.23
DMPC T86.108279.1069.144.895.80
SpikeLLM T2/W4A411.60098.587.461.064.29
LAS T165.970871.3053.8653.1341.39
FAS T168.766366.8149.674.0731.77
SpikedAttention T405.974380.6667.7149.1737.31
DCPN T85.515387.5282.0657.8644.49

LLaMA-3-8B:

MethodWT2 PPL ↓Needle 1K ↑Needle 2K ↑Needle 4K ↑Needle 8K ↑RULER 4K ↑
ANN6.135897.9295.8397.1697.7393.26
DMPC T86.298990.0877.1963.1821.7472.57
SpikeLLM T2/W4A417.83841.170.850.000.000.46
LAS T167.450193.1685.2781.0446.4285.83
FAS T169.231476.3279.2379.5740.5976.12
SpikedAttention T406.637183.5478.6263.9141.0277.89
DCPN T86.225395.1790.2386.8259.8189.40

DCPN at T=8 achieves the highest long-context scores among these evaluated SNN configurations. Compared with LAS T=16, DCPN improves 4K NeedleBench/RULER by 4.73/3.10 percentage points on LLaMA-2 and 5.78/3.57 points on LLaMA-3. It also improves LLaMA-3 NeedleBench 8K by 13.39 points.

These comparisons extend beyond the original DMPC baseline. Since the methods employ different timestep counts, neuron designs, and conversion procedures, they are cross-method performance comparisons rather than strictly matched-compute comparisons.


2. Theoretical interpretation of weak-attention distortion.

Consider one attention row aa and its value matrix VV. Let F(a)F(a) denote the spike-decoded attention response and e=F(a)−ae=F(a)-a. The resulting attention-output error satisfies

\|\Delta o\|_2^2=e^\top VV^\top e.$$ Thus, elementwise attention reconstruction error and downstream attention-output error are not equivalent: the latter also depends on the geometry of the value vectors. Let $Z$ denote attention entries mapped to zero, with erased mass $m_Z=\sum_{i\in Z}a_i$. Their removed contribution is $\sum_{i\in Z}a_iv_i$. For example, if $k$ weak entries each carry mass $m/k$ with similar value vectors, once $m/k$ falls below the zero-response threshold, their **aggregate removed contribution remains approximately $m\bar v$**, despite each individual probability being small. This illustrates why a fixed zero-response region can become consequential in longer contexts. The original response diagnostics support this mechanism: on LLaMA-2 at T=6, erased attention mass increases from **16.35% at 512 tokens to 47.52% at 4K**, while AV NMSE increases from **0.0739 to 0.3599**. DCPN addresses this representation problem by adjusting the final phase's threshold and readout to recover weak probabilities, while coordinating earlier-phase readouts to preserve the coverage of larger probabilities. The AV screening objective directly measures the resulting attention-output distortion. We will incorporate the expanded comparisons and this theoretical explanation into the revised manuscript.
AuthorsOpenReview ↗

Reply to Reviewer gXkk — W2. Generalization

We thank the reviewer for encouraging broader evaluation on modern LLMs and challenging long-context tasks, particularly those involving reasoning and response generation.

1. Generalization across model families and longer contexts.

We extended our T=8 evaluation from LLaMA-2/3 to Qwen2.5-7B and Qwen3-8B, covering contexts up to 16K.

NeedleBench and WikiText-2 results:

ModelMethodWT2 PPL ↓Needle 2K ↑Needle 4K ↑Needle 8K ↑Needle 16K ↑
Qwen2.5-7BANN6.848696.9798.1197.1696.69
DMPC T87.215391.0281.8050.950.00
DCPN T86.992293.5694.5389.3860.17
Qwen3-8BANN6.994696.6996.2993.8494.22
DMPC T87.767797.3596.5786.2712.31
DCPN T87.459396.2195.9395.7285.64

RULER results:

ModelMethodRULER 2K ↑RULER 4K ↑RULER 8K ↑RULER 16K ↑
Qwen2.5-7BANN94.6092.2390.6188.23
DMPC T891.3075.3540.941.36
DCPN T893.7587.8375.2652.12
Qwen3-8BANN95.3393.9593.6292.15
DMPC T893.8777.1149.8211.45
DCPN T894.2488.8775.1859.34

RULER contains 13 tasks and 6,500 examples per configuration, while NeedleBench contains 208 cases across three equally weighted task groups.

At 16K, DCPN improves NeedleBench/RULER over DMPC by 60.17/50.76 percentage points on Qwen2.5 and 73.33/47.89 points on Qwen3. Although Qwen3 shows small NeedleBench decreases at 2K/4K, its improvements at 8K/16K are substantial. These results extend the observed benefits beyond LLaMA and demonstrate stronger long-context capability retention.


2. Long-context multi-hop reasoning and response generation.

We additionally evaluated Qwen3-8B at T=8 on three LongBench multi-hop QA tasks: HotpotQA, 2WikiMQA, and MuSiQue. Each task contains 200 examples per method, with input lengths spanning 1,018–17,898 tokens, evaluated using official LongBench F1.

TaskANNDMPC T8DCPN T8Δ DCPN−DMPC95% paired CI
HotpotQA57.5530.5543.71+13.16[7.49, 18.88]
2WikiMQA44.5438.8540.29+1.44[−3.81, 6.78]
MuSiQue31.7211.9722.80+10.83[6.08, 15.75]
Mean44.6027.1235.60+8.48[5.46, 11.59]

DCPN improves the three-task mean by 8.48 F1 points (95% paired CI [5.46, 11.59]). The improvements on HotpotQA (+13.16) and MuSiQue (+10.83) have confidence intervals entirely above zero, while the smaller gain on 2WikiMQA (+1.44) is not statistically resolved.

These tasks require integrating evidence from multiple locations within long inputs, providing more challenging evaluations than single-fact retrieval. The remaining gap to ANN indicates partial recovery of long-context multi-hop QA capability.

Our original LongBench evaluation also covers document QA, summarization, and code completion. At T=8, the sample-weighted score improves from 42.62 to 53.43 on LLaMA-2-7B and 44.78 to 48.04 on LLaMA-3-8B (DMPC → DCPN), using untruncated native-window subsets.


3. Cross-family mechanism diagnostics.

We further conducted paired response diagnostics across 17 model–length configurations, using identical ANN attention/value tensors for DMPC and DCPN within each model. All configurations use T=8 and G=2.

Representative results at each model's longest evaluated context are shown below (DMPC → DCPN, all ↓):

ModelLengthLost mass (%)Zero-response (%)AV NMSE
LLaMA-2-7B4K14.45 → 11.2799.13 → 98.04.025123 → .018503
LLaMA-3-8B8K14.06 → 11.0399.55 → 99.06.027375 → .012566
Qwen2.5-7B16K23.79 → 19.8799.75 → 99.46.035427 → .010074
Qwen3-8B16K18.30 → 15.0099.74 → 99.49.047654 → .010703

Across all 17 configurations, DCPN consistently reduces lost attention mass, zero-response ratio, and AV NMSE. At 16K, AV NMSE decreases by 71.56% on Qwen2.5 and 77.54% on Qwen3.

These diagnostics support the transferability of weak-attention repair across model families and link the observed capability recovery to improved attention–value reconstruction.

Together, our results demonstrate improvements beyond LLaMA and single-fact retrieval, extending to 16K long-context benchmarks, multi-hop QA, and response generation. The observed recovery is substantial but remains incomplete relative to ANN.

We will incorporate the expanded evaluations, multi-hop QA results, and cross-family mechanism diagnostics into the revised manuscript.

Reviewer_TvToOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies low-timestep ANN-to-SNN conversion of LLaMA-2-7B and LLaMA-3-8B. Short-context perplexity and QA remain close to the ANN, while RULER and NeedleBench degrade sharply as context grows. Module-replacement experiments attribute most of the recoverable error to the few-spike encoding of post-Softmax attention: probabilities inside its zero-response region are decoded as zero, and their total mass grows with length. Uniform rescaling can admit these weak probabilities only by reducing coverage of larger ones. DCPN keeps the pretrained weights, timestep count, parameter storage, and few-spike structure. It changes the final-phase threshold and readout to represent weak probabilities, coordinates earlier-phase readouts for salient probabilities, screens candidates by attention–value reconstruction, and accepts updates by validation NLL in layer order. At (T=8), the reported gains over DMPC are 38.69 and 52.97 points on LLaMA-2-7B at 4K, and 24.44 and 38.07 points on LLaMA-3-8B at 8K, for RULER and NeedleBench respectively.

Critical strengths

The diagnosis connects a real evaluation gap to a concrete mechanism. Figure 1 shows that a converted model can nearly match the ANN on ordinary perplexity and QA while scoring 5.80 against 75.23 on RULER at 4K. The length-wise replacement experiments make the post-Softmax bottleneck specific rather than a generic claim that conversion error increases with length.

The repair is tested at a fixed budget. DCPN does not add timesteps or response parameters. On LLaMA-2-7B at (T=8), weak-end repair accounts for most of the NeedleBench recovery, from 4.89 to 50.28 at 4K, and an unstructured search with the same candidate budget falls to 2.10. The gain over DMPC is therefore not just additional search.

Critical weaknesses

The long-context comparison does not establish an advance over other spiking LLMs. DMPC is the only strong baseline on RULER and NeedleBench. The LAS and SpikedAttention rows are described in Appendix Table 32 as framework-matched encoding controls, not reproductions of the published systems. A RULER score of 44.49 against 0.48 therefore cannot be read as outperforming published SpikedAttention. FAS, published LAS, and SpikeLLM are absent, so the paper does not show whether the zero-response failure is specific to the DMPC phase neuron.

Attention–value screening is not supported as a core component. Appendix B.4 states that it does not outperform MSE: dual-endpoint WikiText-2 perplexity is 5.607 with MSE and 5.688 with attention–value screening. Table 11 reports a correlation of only (\rho=0.356) with validation NLL, and the top-two shortlist retains the model-best candidate for 12.5% of layers. Forward confirmation is the best of the four tested schedules, but that does not show that the committed parameters are optimal. Because candidate screening and confirmation are listed as a contribution, this gap affects the soundness of the method, not only its efficiency.

Questions for authors
  1. I may have misunderstood the LAS and SpikedAttention rows. Appendix Table 32 describes them as framework-matched controls. Could you clarify in the main text whether these rows use fixed binary phase coding inside the DMPC framework, and how their neurons, timestep limits, and Softmax treatment differ from the published methods? It would also help readers if the main discussion distinguished these controls from the original papers.

  2. The long-context gains over DMPC are clear. To see how far the claim extends, would it be possible to include one published converter under the same RULER and NeedleBench protocol? If that is difficult, a short discussion would already help: whether DCPN is meant to repair the post-Softmax phase neuron in DMPC, rather than spiking attention in general, and whether FAS or SpikeLLM has a similar zero-response region.

  3. I would like to understand the role of attention–value screening a little better. Appendix B.4 finds that it does not outperform MSE on WikiText-2 perplexity. If it is convenient, could you compare MSE screening, attention–value screening, and validation NLL on the same candidate pool with RULER or NeedleBench? If attention–value screening is not stronger there, it might be clearer to describe it as one practical choice rather than a necessary part of the method. It would also be helpful to report the candidate-pool size, the number of full-model NLL evaluations, and the GPU hours used for construction and confirmation.

  4. The behavior on LLaMA-2-7B and LLaMA-3-8B is already clear. It might help to say that the present claim refers to these two models and their native 4K and 8K windows, since a gap to the ANN remains at the longest lengths. A result on one non-LLaMA model, or on a larger model, would be a welcome addition, but I see it as further evidence rather than something needed to interpret the current experiments.

  5. I read the energy numbers as a theoretical 45 nm MAC/AC estimate, with DCPN staying within about 1% of DMPC. Could you note that the phase readout uses real coefficients and that memory energy is not included? A hardware simulator would be interesting if it is easy to add, but it is not needed for this comparison.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer TvTo — W1/Q1/Q2. Published Converters and Scope of Comparison

We thank the reviewer for highlighting the distinction between framework-matched controls and published conversion systems. We clarify the original comparisons and provide additional evaluations of published methods.

1. Clarification of the original encoding controls.

The LAS and SpikedAttention rows in our original Tables 1–2 and Appendix Tables 32–33 are framework-matched encoding controls implemented following the simplified LLaMA baseline adaptations in DMPC (Zheng et al., 2026), rather than reproductions of the complete published systems.

Both use fixed binary phase coding with θt=ht=dt=τ2−t\theta_t=h_t=d_t=\tau 2^{-t}. Following DMPC's adaptation, the SpikedAttention control relaxes the original single-spike restriction to allow at most two output spikes. These controls do not implement LAS's OAT/HG neurons and spike-equivalent Transformer components or SpikedAttention's winner-oriented Softmax and trace-driven matrix multiplication.


2. Comparisons with published converters.

We additionally evaluated SpikeLLM (T=2), LAS (T=16), FAS (T=16), and SpikedAttention (T=40) on the same RULER and NeedleBench protocols.

LLaMA-2-7B:

MethodConfigurationWT2 PPL ↓Needle 1K ↑Needle 2K ↑Needle 4K ↑RULER 4K ↑
ANN—5.472183.0173.7567.2275.23
SpikeLLMT=211.60098.587.461.064.29
LAST=165.970871.3053.8653.1341.39
FAST=168.766366.8149.674.0731.77
SpikedAttentionT=405.974380.6667.7149.1737.31
DMPCT=86.108279.1069.144.895.80
DCPN (ours)T=85.515387.5282.0657.8644.49

LLaMA-3-8B:

MethodConfigurationWT2 PPL ↓Needle 1K ↑Needle 2K ↑Needle 4K ↑Needle 8K ↑RULER 4K ↑
ANN—6.135897.9295.8397.1697.7393.26
SpikeLLMT=217.83841.170.850.000.000.46
LAST=167.450193.1685.2781.0446.4285.83
FAST=169.231476.3279.2379.5740.5976.12
SpikedAttentionT=406.637183.5478.6263.9141.0277.89
DMPCT=86.298990.0877.1963.1821.7472.57
DCPN (ours)T=86.225395.1790.2386.8259.8189.40

DCPN at T=8 achieves the highest retrieval scores among the evaluated spiking methods on both models. Compared with LAS T=16, it improves 4K NeedleBench/RULER by 4.73/3.10 percentage points on LLaMA-2 and 5.78/3.57 points on LLaMA-3, with a further 13.39-point gain on LLaMA-3 NeedleBench 8K.

We emphasize that these are cross-method evaluations under their respective configurations, not strictly matched-budget comparisons: the methods use different timestep counts, quantization settings, neuron designs, and conversion procedures. The controlled same-budget comparison remains DCPN versus DMPC.


3. Generality of the weak-attention failure mechanism.

DCPN addresses a fundamental representation challenge in low-timestep spike coding: weak attention probabilities can fall within the neuron's zero-response region, causing information loss that accumulates with context length.

Although DCPN is implemented within DMPC's post-Softmax FS neurons, our additional experiments demonstrate that this phenomenon and the proposed repair extend across model families. On LLaMA-2-7B, LLaMA-3-8B, Qwen2.5-7B, and Qwen3-8B, paired diagnostics across 17 model–length configurations consistently show reductions in lost attention mass, zero-response ratio, and AV NMSE. At 16K, DCPN reduces AV NMSE by 71.56% on Qwen2.5 and 77.54% on Qwen3.

Other converters use different spike representations and nonlinear operator implementations, so the manifestation of weak-attention loss may vary. Our results establish the cross-family transferability of the proposed repair under low-timestep FS phase coding, while the additional published-converter comparisons demonstrate its competitiveness beyond the DMPC baseline.

We will clarify the baseline implementations, comparison scope, and generalization evidence in the revised manuscript.


References

[1] Zheng et al. Distribution-Aware Multi-Granularity Phase Coding: Towards Lower Conversion Error for Spike-Driven Large Language Models. ICLR 2026.

[2] Xing et al. SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking. ICLR 2025.

[3] Chen et al. LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models. AAAI 2026.

[4] Chen et al. FAS: Fast ANN-SNN Conversion for Spiking Large Language Models. arXiv:2502.04405, 2025.

[5] Hwang et al. SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation. NeurIPS 2024.

AuthorsOpenReview ↗

Reply to Reviewer TvTo — W2/Q3. AV Screening and Model-Level Confirmation

We thank the reviewer for examining the role of AV screening and its relationship to model-level NLL. We provide full-procedure retrieval comparisons, candidate-ranking diagnostics, and computational costs.

1. Full sequential AV vs. MSE comparison.

We compare AV and MSE screening under the complete two-pass sequential confirmation procedure, using identical initial DMPC checkpoints, candidate-generation rules, data splits, layer order, shortlist quotas, acceptance criteria, and confirmation-budget caps.

After accepted updates, each method recollects subsequent-layer activations from its updated model; thus, the realized downstream candidates may differ.

Full-procedure results:

ModelScreeningWT2 PPL ↓Needle 4K ↑Needle 8K ↑RULER 4K ↑RULER 8K ↑
LLaMA-2-7BAV5.515357.86—44.49—
MSE5.521253.13—41.47—
LLaMA-3-8BAV6.225386.8259.8189.4061.35
MSE6.254183.0341.9683.8457.87

Paired 95% confidence intervals (AV − MSE):

ComparisonDifference (pp)95% CI
LLaMA-2, RULER 4K+3.02[2.21, 3.82]
LLaMA-3, RULER 4K+5.56[5.01, 6.12]
LLaMA-3, RULER 8K+3.48[2.76, 4.19]
LLaMA-2, NeedleBench 4K+4.73[3.24, 5.77]
LLaMA-3, NeedleBench 4K+3.79[−2.58, 9.77]
LLaMA-3, NeedleBench 8K+17.85[9.68, 26.20]

We use 10,000 stratified paired cluster-bootstrap resamples, jointly resampling matched question-family clusters and preserving the original equal-weight macro averages (1/3 per NeedleBench task group; 1/13 per RULER task).

Five of six intervals exclude zero, including all three RULER comparisons and the +17.85-point improvement on LLaMA-3 NeedleBench 8K. The LLaMA-3 NeedleBench 4K interval includes zero.

These results demonstrate that AV screening improves long-context retrieval under the evaluated full sequential procedure, despite the different PPL rankings in the controlled experiments of Tables 12–13.


2. Candidate ranking and the role of validation NLL.

AV measures attention–value reconstruction, whereas MSE measures elementwise activation error. Both are local proxies and need not agree with model-level validation NLL.

In Table 11, Dual/AV achieves higher Spearman correlation (0.356 vs. 0.312) and Recall@2 (12.5% vs. 3.1%) than Dual/MSE, but its confirmed NLL regret is slightly higher (0.00375 vs. 0.00284). This demonstrates that improved local ranking does not necessarily translate into better model-level selection.

In the full sequential procedure, structured search considers 259 non-current candidates plus the current response per layer. Local screening forms a shortlist, while validation NLL evaluates proposals in the progressively updated model under the prescribed acceptance criterion.

Table 11 evaluates candidate retention under a fixed surrounding model, rather than the complete sequential trajectory. We therefore do not claim global optimality. Instead, local screening and model-level confirmation serve complementary roles, with the new full-procedure results directly demonstrating AV's empirical benefit for long-context retrieval.


3. Confirmation budgets and offline costs.

Both objectives use the same confirmation-budget cap. The realized AV/MSE candidate-confirmation counts are:

ModelAVMSE
LLaMA-2-7B399404
LLaMA-3-8B402396

Thus, AV's retrieval improvements are obtained with closely matched realized confirmation counts.

At T=8, complete offline calibration requires 27.27/30.98 GPU·h on NVIDIA L20 GPUs for LLaMA-2-7B/LLaMA-3-8B. The two post-Softmax search passes account for 24.03/25.36 GPU·h, respectively, including selection and confirmation operations. These are complete offline workflow costs, not solely incremental DCPN overhead.

Overall, the full-procedure results provide direct empirical evidence for AV screening's value in long-context retrieval, with five of six paired confidence intervals excluding zero. Although local screening does not guarantee the globally optimal response, AV captures downstream attention–value distortion, while model-level NLL confirmation evaluates shortlisted candidates in the progressively updated model. Together, these stages form a practical downstream-calibrated selection procedure that improves retrieval under matched calibration budgets.

We will incorporate these results, ranking diagnostics, confirmation costs, and clarified contribution claims into the revised manuscript.

AuthorsOpenReview ↗

Reply to Reviewer TvTo — Q4. Generalization Across Model Families and Longer Contexts

We thank the reviewer for recognizing the results on the original LLaMA models and encouraging broader generalization experiments. We extended our evaluation to Qwen2.5-7B and Qwen3-8B at T=8, covering contexts up to 16K, and conducted additional mechanistic diagnostics across all four models.

1. Cross-family and long-context evaluation.

We evaluate ANN, DMPC, and DCPN under matched model-specific protocols. RULER contains 13 tasks and 6,500 examples per configuration, equally weighted across tasks. NeedleBench contains 208 cases, comprising 110 single-retrieval, 10 multi-retrieval, and 88 multi-reasoning cases, with equal task-group weights.

NeedleBench (%, ↑):

ModelMethod2K4K8K16K
Qwen2.5-7BANN96.9798.1197.1696.69
DMPC91.0281.8050.950.00
DCPN (ours)93.5694.5389.3860.17
Qwen3-8BANN96.6996.2993.8494.22
DMPC97.3596.5786.2712.31
DCPN (ours)96.2195.9395.7285.64

RULER (%, ↑):

ModelMethod2K4K8K16K
Qwen2.5-7BANN94.6092.2390.6188.23
DMPC91.3075.3540.941.36
DCPN (ours)93.7587.8375.2652.12
Qwen3-8BANN95.3393.9593.6292.15
DMPC93.8777.1149.8211.45
DCPN (ours)94.2488.8775.1859.34

At 16K, DCPN improves NeedleBench/RULER over DMPC by 60.17/50.76 percentage points on Qwen2.5 and 73.33/47.89 points on Qwen3. At 8K, the corresponding gains are 38.43/34.32 and 9.45/25.36 points.

The improvements become particularly pronounced as context length increases. On Qwen2.5, DMPC's NeedleBench score drops from 91.02 at 2K to 0.00 at 16K, whereas DCPN retains 60.17. On Qwen3, DMPC falls from 97.35 to 12.31, while DCPN achieves 85.64 at 16K.

DCPN also improves RULER at every evaluated length on both models. On Qwen3, NeedleBench exhibits small decreases at 2K/4K (−1.14/−0.64 points), but substantial gains at 8K/16K, indicating that the main benefit lies in preserving long-context information use.

Language-modeling performance also improves: WikiText-2 PPL decreases from 7.2153 to 6.9922 on Qwen2.5 and from 7.7677 to 7.4593 on Qwen3. Thus, the long-context recovery is accompanied by improved language-modeling fidelity.


2. Cross-model mechanistic validation.

To further validate the proposed mechanism, we conducted matched response diagnostics on LLaMA-2-7B, LLaMA-3-8B, Qwen2.5-7B, and Qwen3-8B, using T=8, G=2, and no additional cap on K with the existing checkpoints.

Context lengths include the static prefix. We use eight shared held-out source-text starting positions with model-specific tokenization. Within each model, DMPC and DCPN share identical ANN attention/value tensors.

We measure lost attention mass, zero-response ratio, and AV NMSE to characterize the frequency, probability-mass impact, and downstream distortion of zero responses.

Representative results at each model's longest evaluated context are shown below (DMPC → DCPN, all ↓):

ModelLengthLost mass (%)Zero-response (%)AV NMSE
LLaMA-2-7B4K14.45 → 11.2799.13 → 98.04.025123 → .018503
LLaMA-3-8B8K14.06 → 11.0399.55 → 99.06.027375 → .012566
Qwen2.5-7B16K23.79 → 19.8799.75 → 99.46.035427 → .010074
Qwen3-8B16K18.30 → 15.0099.74 → 99.49.047654 → .010703

Across all 17 model–length configurations, DCPN consistently reduces all three metrics. The complete results will be provided in the revised appendix.

At 16K, DCPN reduces lost attention mass by 3.92/3.30 percentage points and AV NMSE by 71.56%/77.54% on Qwen2.5/Qwen3, respectively. Notably, despite zero-response ratios remaining above 99%, the substantial reductions in AV NMSE demonstrate that the impact of zero responses depends not merely on their frequency, but also on the probability mass and downstream distortion associated with them.

These findings provide cross-family mechanistic evidence consistent with DCPN's weak-attention repair strategy, complementing the original module-intervention experiments.

We will incorporate the new benchmark results, complete cross-model diagnostics, and updated limitations into the revised manuscript.

Reviewer_KWuJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies context-length-dependent failures in few-timestep ANN-to-SNN conversion of language models. It identifies the loss of small post-Softmax attention probabilities as an important source of degradation and proposes the Downstream-Calibrated Phase Neuron (DCPN). DCPN adjusts the final-phase threshold and readout to recover weak attention, while coordinating earlier-phase readouts to retain coverage of larger probabilities.

Critical strengths

easy to fellow and good writting

The paper identifies context-dependent failures missed by conventional benchmarks. Component interventions and attention diagnostics provide convincing evidence for targeting post-Softmax weak-attention loss.

DCPN achieves substantial gains at matched deployment budgets. At (T=8), LLaMA-2-7B’s 4K RULER and NeedleBench scores improve from 5.80 to 44.49 and 4.89 to 57.86, respectively. Targeted ablations support the response design, with improvements extending to LLaMA-3-8B and conventional tasks.

Critical weaknesses

Important details of the main calibration procedure remain unspecified. Sections B.1–B.2 and D.1 do not provide the complete numerical candidate grids, the higher-timestep reference used for each deployment setting, the head-readout compensation rule, or the main shortlist and acceptance settings. The general-search control matches candidate counts, but its search ranges and sampling procedure are unclear. Additionally, Appendix A.4 distinguishes released and refitted DMPC checkpoints, whereas the main tables only label the baseline as “DMPC.” These omissions make it difficult to reproduce the main results and assess how much improvement comes from targeted response construction versus recalibration of the baseline.

The incremental value of AV screening over activation MSE is not established for the main retrieval results. In Table 12, dual-endpoint construction with MSE achieves lower perplexity than with AV, 5.607 versus 5.688. The model-confirmed comparison in Table 13 also favors Dual/MSE over Dual/AV, 5.6171 versus 5.6325. The paper acknowledges this result, which strengthens its transparency, but leaves the choice of AV screening insufficiently justified. A matched retrieval comparison using the complete sequential confirmation procedure would clarify which selection objective is preferable.

Uncertainty is reported more thoroughly for diagnostic experiments than for the smaller downstream improvements. NeedleBench uses 208 cases per length, including only 10 multi-retrieval cases, while weighting the three task groups equally. Consequently, the smaller differences between weak-only and dual repair, or between repair-phase placements, would benefit from paired confidence intervals. The large baseline-to-DCPN gains remain compelling, but the reliability of these smaller design advantages is less clear.

Questions for authors

Could you provide the complete configuration used for Tables 1–3, including the reference timestep, candidate grids, head compensation, shortlist size, acceptance variant and threshold, and calibration/selection splits? Please also identify whether the reported DMPC baseline is released or refitted, and report the offline calibration cost.

With the same candidate pool, sequential update schedule, and model-level confirmation budget, how do MSE and AV screening compare on RULER and NeedleBench? Does the objective preferred by perplexity also preserve retrieval performance?

Could you report task-group results and stratified paired confidence intervals for the endpoint and phase-placement ablations? This would clarify the reliability of the smaller gains beyond weak-response repair.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer KWuJ — W1/Q1. Reproducibility and Baseline Fairness

We thank the reviewer for highlighting the reproducibility and baseline-fairness concerns. We clarify the calibration settings, search budgets, checkpoint provenance, and offline costs.

1. Candidate construction and numerical grids.

All experiments preserve the original FS inference structure. We use G=2 head/tail coefficient groups for thresholds, resets, and readouts. The higher-timestep references are T_ref = 7, 8, 9, 10 for deployment T = 5, 6, 7, 8, respectively.

The structured pool contains native, range, weak-end, and joint candidates. The complete numerical grids are:

  • Range (20): {0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.75, 2.0, 2.5, 3.0, 4.0}.
  • Weak-end (13): {0.35, 0.5, 0.7, 0.85, 1.0, 1.15, 1.3, 1.5, 1.75, 2.0, 2.5, 3.0, 4.0}.

After determining the weak-end target from the higher-timestep reference, head-readout compensation solves

Σ(t=1,...,T−1) τ·b^(−t) + q_tail ≈ R_max(source)

using 80 bisection iterations over b∈[1.01,8], followed by verification of the realized FP16 response maximum.


2. General-search control and fairness.

General search uses fixed-seed multiplicative random sampling (seed 42):

ParameterMultiplier distribution
τLogUniform[0.5, 2]
d_g, h_g, θ_gexp(U[−0.35, 0.35]), clipped to [1.05, 8]
v₀exp(U[−1, 1])

Multipliers are sampled independently across parameters and coefficient groups. The resulting (d_g, h_g, θ_g) bases are clipped to [1.05, 8]. A fixed bank of 260 multiplier configurations is reused across layers, applied to each layer's own source parameters.

Structured search includes 259 non-current candidates plus the current response (when range = weak-end = 1), while general search samples 260 non-current candidates. Both use local AV reconstruction screening, the same confirmation-budget limits, and two forward sequential passes with activation recapture.

Thus, the control matches the candidate-evaluation scale and confirmation procedure, not the structured method's search space or sampling prior. We will clarify thisin the revised manuscript.


3. Shortlist, acceptance, and data separation.

The shortlist retains up to two candidates per non-native family, one native candidate, and the current response. We perform two forward sequential confirmation passes, recollecting subsequent-layer activations after accepted updates.

Updates are accepted when the upper bound of the paired 95% CI for candidate-minus-current validation NLL is below zero (20,000 block-bootstrap resamples, block size 4, δ=0).

Data are separated into:

  • Calibration: first 90% of the WikiText-2 training split.
  • Model-level selection: remaining 10% of the training split.
  • Testing: independent WikiText-2 test split.
  • Local calibration: 16 windows of 2,048 tokens.
  • Local diagnostic holdout: 8 windows of 2,048 tokens for AV-NMSE evaluation.
  • Data sampling seed: 1.

Calibration, selection, and final testing are separated.


4. DMPC checkpoint provenance.

We clarify that the main T=8 LLaMA-2-7B and LLaMA-3-8B comparisons use the released DMPC checkpoints, with DCPN starting from those exact checkpoints. Thus, the reported improvements at T=8 are not attributable to an additional DMPC refitting stage.

For T=5–7, corresponding released checkpoints were unavailable, so we reproduced DMPC using our implementation.

To validate this reproduction procedure, we independently reproduced the T=8 DMPC calibration with three seeds. Relative to the released checkpoints, the WikiText-2 PPL discrepancy was at most 1.57%, and the discrepancy in mean accuracy across PIQA, ARC-Easy, ARC-Challenge, and WinoGrande was at most 0.63 percentage points.

These checks support the fidelity of our reproduced lower-timestep baselines. We will explicitly distinguish released and reproduced DMPC configurations in the revised tables.


5. Offline calibration cost.

We measured the complete offline calibration workflow on NVIDIA L20 GPUs:

Deployment TReference TLLaMA-2-7BLLaMA-3-8BQwen2.5-7BQwen3-8B
5719.7421.63——
6822.1826.05——
7925.2829.02——
81027.2730.9826.9731.69

All costs are in GPU·h.

At T=8, the two post-Softmax search passes account for 24.03, 25.36, 22.98, and 27.47 GPU·h across the four models, respectively. The remaining costs cover reference construction, operator fitting, and other offline calibration stages. The reported costs include the associated selection and confirmation operations.

These measurements quantify the complete offline workflow rather than only the incremental DCPN search cost. Calibration does not change pretrained weights or introduce additional inference-time validation.

We will incorporate these configurations, checkpoint distinctions, and measured costs into the revised manuscript.

AuthorsOpenReview ↗

Reply to Reviewer KWuJ — W2/Q2. AV vs. MSE Screening

We thank the reviewer for this suggestion. We agree that the perplexity comparisons in Tables 12–13 do not establish AV screening's advantage for long-context retrieval. We therefore conducted a new comparison using the complete sequential confirmation procedure, holding candidate construction, starting checkpoints, data splits, and confirmation budgets fixed.

1. Matched full-procedure comparison.

AV and activation MSE screening use the same initial DMPC checkpoint, candidate-construction rules, layer order, shortlist quotas, acceptance criterion, and maximum full-model confirmation budget.

Both methods perform two forward sequential passes. After each accepted update, subsequent-layer activations are recollected from the corresponding updated model. Thus, while candidate-generation rules and initial states are identical, the realized downstream candidates can differ as the two trajectories evolve.

Complete evaluation results:

ModelScreeningWT2 PPL ↓Needle 1K ↑Needle 2K ↑Needle 4K ↑Needle 8K ↑RULER 4K ↑RULER 8K ↑
LLaMA-2-7BAV5.515387.5282.0657.86—44.49—
MSE5.521270.8772.3153.13—41.47—
LLaMA-3-8BAV6.225395.1790.2386.8259.8189.4061.35
MSE6.254194.1784.3983.0341.9683.8457.87

Under the complete procedure, AV achieves lower PPL and higher RULER/NeedleBench scores on both models.

On LLaMA-2-7B, AV improves RULER 4K by approximately 3.02 percentage points and NeedleBench 4K by 4.73 points over MSE.

On LLaMA-3-8B, AV improves RULER by 5.56/3.48 points at 4K/8K, respectively, and NeedleBench by 3.79/17.85 points at 4K/8K.


2. Paired statistical comparisons.

Statistical protocol. For both RULER and NeedleBench, we use 10,000 stratified paired cluster-bootstrap resamples (seed 42). Within each task stratum, we jointly resample matched AV/MSE question-family clusters and their associated depth variants, preserving example-level pairing.

Each bootstrap replicate retains the original macro-averaging rule:

  • NeedleBench: 208 cases per length (110 single-retrieval, 10 multi-retrieval, and 88 multi-reasoning), with equal weights across the three task groups (1/3 each).
  • RULER: 13 tasks with 500 cases each (6,500 total), with equal weights across tasks (1/13 each).

Paired 95% confidence intervals (AV − MSE):

Comparison (AV − MSE)Difference (pp)Paired 95% CI
LLaMA-2, RULER 4K+3.02[2.21, 3.82]
LLaMA-3, RULER 4K+5.56[5.01, 6.12]
LLaMA-3, RULER 8K+3.48[2.76, 4.19]
LLaMA-2, NeedleBench 4K+4.73[3.24, 5.77]
LLaMA-3, NeedleBench 4K+3.79[−2.58, 9.77]
LLaMA-3, NeedleBench 8K+17.85[9.68, 26.20]

All three RULER comparisons and both the LLaMA-2 4K and LLaMA-3 8K NeedleBench comparisons have confidence intervals entirely above zero. In particular, AV improves LLaMA-3 NeedleBench 8K by 17.85 points [9.68, 26.20], supporting its value for longer-context information retrieval.


3. Confirmation-budget accounting.

Both screening objectives use the same full-model confirmation budget cap. The actual numbers of candidate confirmations were:

ModelAVMSE
LLaMA-2-7B399404
LLaMA-3-8B402396

The small differences arise from the realized sequential trajectories. Both methods remain subject to the same confirmation-budget limit, and AV's improvements do not require a substantially larger number of full-model evaluations.


4. Interpretation of Tables 12–13.

Tables 12–13 examine controlled candidate-selection settings, whereas the new experiment evaluates the complete two-pass sequential workflow.

Their earlier PPL rankings therefore do not directly predict the final rankings after iterative model updates and activation recapture. The new results show that AV screening is beneficial under the evaluated full-procedure settings, with statistically supported improvements across multiple RULER and NeedleBench comparisons.

AV remains a local proxy rather than a guarantee of globally optimal candidate selection. Its advantage here is empirical: attention–value reconstruction screening better preserves downstream information use under the tested sequential calibration protocol.

We will incorporate these results, paired uncertainty estimates, and confirmation-budget details into the revised manuscript.

AuthorsOpenReview ↗

Reply to Reviewer KWuJ — W3/Q3. Statistical Reliability of Ablations

We thank the reviewer for this suggestion. We provide additional task-group results and stratified paired confidence intervals for the endpoint and phase-placement ablations.

1. Statistical protocol.

NeedleBench contains 208 cases per context length: 110 single-retrieval, 10 multi-retrieval, and 88 multi-reasoning cases, with the three groups equally weighted.

We use 10,000 stratified paired cluster-bootstrap resamples (seed 42). Within each group, question families and their depth variants are jointly resampled, preserving method pairing. The groups contain 10/10/8 question families, respectively, and each replicate retains equal group weights (1/3 each).

We focus on the 4K setting and additionally report LLaMA-3-8B at 8K.


2. Endpoint ablation: Dual vs. Weak-only.

We extend the comparison to both LLaMA models using matched calibration data and acceptance rules.

Overall evaluation results:

ModelMethodWT2 PPL ↓Needle 1KNeedle 2KNeedle 4KNeedle 8KRULER 4K
LLaMA-2-7BDual5.515387.5282.0657.86—44.49
Weak-only5.552084.0276.1450.28—44.18
LLaMA-3-8BDual6.225395.1790.2386.8259.8189.40
Weak-only6.246696.0687.0578.4840.0381.17

Task-group scores (%) and paired 95% CIs:

Model / LengthMethod / ContrastSingle retrievalMulti-retrievalMulti-reasoningMacro average
LLaMA-2 / 4KDual60.9157.2755.4057.86
Weak-only62.7332.7355.4050.28
Δ [95% CI]−1.82 [−9.09, 5.45]+24.55 [8.18, 43.64]0.00 [−4.55, 4.55]+7.58 [1.36, 14.47]
LLaMA-3 / 4KDual80.9195.4584.0986.82
Weak-only58.1895.4581.8278.48
Δ [95% CI]+22.73 [6.36, 38.18]0.00 [−4.55, 4.55]+2.27 [−2.27, 7.95]+8.34 [2.35, 14.17]
LLaMA-3 / 8KDual43.6468.1867.6159.81
Weak-only19.0944.5556.4440.03
Δ [95% CI]+24.55 [11.82, 37.27]+23.64 [7.27, 40.91]+11.17 [0.38, 21.97]+19.78 [12.01, 27.73]

Δ = Dual − Weak-only, in percentage points. Positive values favor Dual.

At 4K, Dual improves the macro average by 7.58 points [1.36, 14.47] on LLaMA-2 and 8.34 points [2.35, 14.17] on LLaMA-3, with both intervals excluding zero.

The gains vary across task groups. LLaMA-2 benefits primarily from multi-retrieval, while LLaMA-3 shows its largest 4K improvement in single retrieval. At 8K, LLaMA-3 exhibits positive confidence intervals across all three groups, yielding a +19.78-point macro improvement [12.01, 27.73].

These results support the additional benefit of coverage repair beyond weak-response repair, while revealing task-dependent contributions.


3. Repair-phase placement: Front vs. Middle vs. Tail.

We extend Table 5 with RULER results and task-group statistics.

Overall retrieval results:

PlacementNeedle 1KNeedle 2KNeedle 4KRULER 1KRULER 2KRULER 4K
Front84.3079.8754.6886.5778.2643.65
Middle84.1579.2653.2686.6778.0543.53
Tail (DCPN)87.5282.0657.8690.2180.2144.49

NeedleBench 4K task-group scores (%) and paired 95% CIs:

Placement / ContrastSingle retrievalMulti-retrievalMulti-reasoningMacro average
Front53.6452.7357.6754.68
Middle50.9150.9157.9553.26
Tail60.9157.2755.4057.86
Tail − Front [95% CI]+7.27 [1.82, 11.82]+4.55 [−2.73, 15.45]−2.27 [−4.55, 0.24]+3.18 [0.16, 7.12]
Tail − Middle [95% CI]+10.00 [5.45, 14.55]+6.36 [−3.64, 17.27]−2.56 [−8.24, 2.27]+4.60 [0.53, 9.02]

Contrasts are in percentage points; positive values favor the first method.

At 4K, Tail improves the macro average over Front and Middle by 3.18 [0.16, 7.12] and 4.60 [0.53, 9.02] points, respectively. Both paired intervals exclude zero.

The improvement is strongest in single retrieval. Multi-retrieval point estimates favor Tail but have intervals including zero, while multi-reasoning shows no statistically resolved Tail advantage.


4. Interpretation.

The additional analysis support two conclusions:

  • dual-endpoint repair provides statistically supported macro-average gains beyond weak-response repair, although the contributing task groups differ across models and context lengths.

  • tail placement yields smaller but statistically supported macro-average gains over front and middle placement at 4K, consistent with our mechanism-based motivation for final-phase weak-response repair.

We will incorporate these task-group results, paired confidence intervals, and statistical details into the revised manuscript.

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the improved reproducibility details, the full-procedure AV vs. MSE comparison, and the additional statistical analysis.

The new results provide stronger evidence for AV screening and the benefits of dual-endpoint repair. However, some of the improvements remain relatively modest, and the uncertainty in certain task-group comparisons, particularly those involving the small multi-retrieval subset, makes it difficult to fully establish the advantages of some design choices.

Overall, the rebuttal has addressed most of my technical concerns and strengthened the experimental evidence. Nevertheless, I still have some reservations about the robustness and significance of the smaller design improvements. I therefore maintain my original rating.

Reviewer_wJ9yOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses severe long‑context performance drop for ANN‑to‑SNN converted spiking LLMs. Few‑spike encoding erases collectively‑important weak attention weights; naive scaling fails to handle both weak and strong attention. DCPN only modifies post‑Softmax neuron parameters: the final phase recovers weak attention while earlier phases preserve strong‑attention representation. It selects parameters via two‑stage pipeline: AV‑loss screening followed by model‑level NLL validation. On LLaMA‑2‑7B and LLaMA‑3‑8B, it greatly boosts long‑context retrieval, improves conventional tasks with negligible energy overhead.

Critical strengths

DCPN only adjusts post‑Softmax phase‑neuron parameters. It preserves original timesteps, parameter count, few‑spike inference pipeline and pretrained weights. No full‑model retraining is required.

Critical weaknesses
  1. As shown in Limitation in paper, only LLaMA‑2‑7B and LLaMA‑3‑8B are tested within their native 4K / 8K windows. Generalization to larger models and extremely long contexts remains unproven.
  2. Lack of real neuromorphic hardware measurement: Energy results are purely theoretical estimates; end‑to‑end latency and power consumption are not measured on physical spiking hardware.
  3. Extra offline calibration cost: Though pretrained weights are frozen, disjoint calibration/validation data are required. Layer‑wise candidate search and model‑level validation add computational overhead in the offline tuning phase.
Questions for authors

What is the point of developing large spiking‑neural‑network (SNN) models today? Dedicated neuromorphic chips are not yet ready for practical deployment, and SNNs show no performance advantages when run on GPUs.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer wJ9y — W1. Generalization

We thank the reviewer for highlighting the importance of generalization across model families, parameter scales, and longer contexts.

1. Cross-family and long-context evaluation.

We extended our experiments to Qwen2.5-7B and Qwen3-8B at T=8, covering context lengths from 2K to 16K.

NeedleBench (%, ↑):

ModelMethod2K4K8K16K
Qwen2.5-7BANN96.9798.1197.1696.69
DMPC91.0281.8050.950.00
DCPN (ours)93.5694.5389.3860.17
Qwen3-8BANN96.6996.2993.8494.22
DMPC97.3596.5786.2712.31
DCPN (ours)96.2195.9395.7285.64

RULER (%, ↑):

ModelMethod2K4K8K16K
Qwen2.5-7BANN94.6092.2390.6188.23
DMPC91.3075.3540.941.36
DCPN (ours)93.7587.8375.2652.12
Qwen3-8BANN95.3393.9593.6292.15
DMPC93.8777.1149.8211.45
DCPN (ours)94.2488.8775.1859.34

At 16K, DCPN improves NeedleBench/RULER over DMPC by 60.17/50.76 percentage points on Qwen2.5 and 73.33/47.89 points on Qwen3. At 8K, the corresponding gains are 38.43/34.32 and 9.45/25.36 points.

WikiText-2 PPL also improves from 7.2153 to 6.9922 on Qwen2.5 and from 7.7677 to 7.4593 on Qwen3, indicating that long-context recovery is accompanied by improved language-modeling fidelity.


2. Cross-model mechanistic validation.

To examine whether the same weak-attention distortion persists across architectures, we conducted additional response diagnostics on LLaMA-2-7B, LLaMA-3-8B, Qwen2.5-7B, and Qwen3-8B.

We use T=8, G=2, retaining the existing checkpoints. Context length is defined as the total token count. We use the same eight held-out source-text starting positions across models, with model-specific tokenization. Within each model, DMPC and DCPN share identical ANN attention/value tensors.

The following table reports DMPC → DCPN; all three metrics are lower-is-better.

ModelLengthLost mass (%) ↓Zero-response (%) ↓AV NMSE ↓
LLaMA-2-7B1K9.55 → 5.9396.28 → 91.83.013592 → .010225
2K12.19 → 8.3298.24 → 95.67.019472 → .014412
4K14.45 → 11.2799.13 → 98.04.025123 → .018503
LLaMA-3-8B1K9.05 → 6.2596.27 → 92.71.010274 → .004349
2K11.54 → 8.4698.15 → 96.16.017803 → .007452
4K13.21 → 10.0599.08 → 98.05.023209 → .010383
8K14.06 → 11.0399.55 → 99.06.027375 → .012566
Qwen2.5-7B1K12.37 → 9.2095.54 → 91.92.009435 → .003357
2K16.81 → 12.8197.74 → 95.40.017974 → .005055
4K19.54 → 15.5698.94 → 97.76.025228 → .006929
8K21.15 → 17.3699.50 → 98.93.027903 → .007945
16K23.79 → 19.8799.75 → 99.46.035427 → .010074
Qwen3-8B1K10.26 → 7.5196.07 → 92.92.015900 → .004273
2K13.37 → 10.1497.99 → 96.10.027393 → .005451
4K15.82 → 12.4598.99 → 97.99.036638 → .006709
8K16.92 → 13.6199.49 → 99.00.038679 → .008605
16K18.30 → 15.0099.74 → 99.49.047654 → .010703

All 17 model–length configurations exhibit reductions in lost attention mass, zero-response ratio, and AV NMSE.

Notably, at 16K:

  • Qwen2.5-7B: Lost mass decreases by 3.92 percentage points, while AV NMSE falls by 71.56%.
  • Qwen3-8B: Lost mass decreases by 3.30 percentage points, while AV NMSE falls by 77.54%.

Although the zero-response ratio remains high for long attention sequences, DCPN substantially reduces the probability mass erased by those responses and the resulting attention–value distortion. This distinction is important: the impact of zero responses depends not only on their frequency but also on the probability mass and downstream information they carry.

These diagnostics provide direct cross-family evidence that context-amplified weak-attention loss persists beyond LLaMA and can be mitigated by DCPN. Together with the new 16K benchmark results, they support the transferability of the proposed response-repair mechanism.


3. Larger parameter scales.

Due to time and computational resource constraints during the rebuttal period, we have not yet evaluated larger parameter scales, such as 14B. We plan to extend our experiments to these scales and include the results in the revised manuscript.

Overall, the new results extend our evaluation from two LLaMA models to four models spanning the LLaMA and Qwen lineages, including Qwen2.5 and Qwen3, and increase the maximum evaluated context length to 16K.

AuthorsOpenReview ↗

Reply to Reviewer wJ9y — W2. Energy Estimation and Hardware Efficiency

We thank the reviewer for highlighting the distinction between theoretical computational energy and practical hardware efficiency. We clarify our methodology, provide measured spike-activity statistics, and specify the scope of our energy claims.

1. Established methodology for theoretical energy estimation.

Our energy analysis follows the MAC/AC operation-counting methodology adopted in previous SNN studies, including DMPC [1], Spikformer [3], and LAS [4].

Specifically, we follow DMPC (Zheng et al., 2026) [1] and use the 45 nm CMOS reference costs from Horowitz (2014) [2]:

Etotal=EMACNMAC+EACNACE_{\mathrm{total}} = E_{\mathrm{MAC}} N_{\mathrm{MAC}} + E_{\mathrm{AC}} N_{\mathrm{AC}}

where EMAC=4.6,pJE_{\mathrm{MAC}} = 4.6\\,\mathrm{pJ} and EAC=0.9,pJE_{\mathrm{AC}} = 0.9\\,\mathrm{pJ}.

Here, MAC denotes dense multiply–accumulate operations, while AC denotes spike-driven accumulations. This methodology enables theoretical comparisons under consistent operation-counting assumptions rather than estimating the actual power consumption of GPUs or neuromorphic chips.


2. Measured spike activity and theoretical energy.

Beyond theoretical operation costs, we directly count spike events throughout the converted model and calculate their corresponding synaptic operations, accounting for fan-out where applicable.

For LLaMA-2-7B at T=8, we obtained:

ContextMethodAll FS events (B)Theoretical energy (J)Δ vs. DMPC
1KDMPC3.864418.1531—
DCPN (ours)3.910318.1467−0.035%
2KDMPC7.758036.7229—
DCPN (ours)7.861536.6976−0.069%
4KDMPC15.527674.7585—
DCPN (ours)15.839874.8797+0.162%

At 4K, DCPN increases post-Softmax spike events from 0.2383B to 0.4548B, enabling more weak attention responses to be represented. Nevertheless, whole-model spike activity increases by only 2.01%, with merely +0.162% theoretical energy overhead.

Across all original model–timestep settings, DCPN's theoretical computational energy differs from DMPC by −0.25% to +1.03%. Under the same MAC/AC model, at T=8, DCPN's estimated energy is 44.33% below ANN on LLaMA-2-7B at 4K and 45.97% below ANN on LLaMA-3-8B at 8K.

These results demonstrate substantial long-context capability recovery under nearly unchanged theoretical computational energy relative to DMPC.


3. Hardware interpretation and limitations.

We distinguish operation-level energy estimation from physical hardware measurements. Actual hardware efficiency depends on memory access, data movement, communication, control overhead, and implementation precision. Neuromorphic hardware research [5] further emphasizes the importance of hardware–algorithm co-design.

Notably, DMPC also provides an extended memory read/write energy analysis in Appendix D.7 [1]. Our reported estimates specifically follow its Sec. 6.3 MAC/AC methodology.

We will clarify that memory-access energy, communication overhead, and real-valued phase-readout implementation costs are not included in our estimates. These results do not represent measured end-to-end GPU energy, neuromorphic power consumption, or hardware latency.


4. Contribution under a fixed deployment budget.

DCPN preserves the original deployment timesteps, response-parameter storage, and few-spike inference structure, without introducing additional inference modules.

Our contribution is therefore recovering substantially more model capability at nearly unchanged theoretical computational energy, rather than claiming immediate hardware acceleration. This addresses a fundamental algorithmic requirement for practical spiking LLMs: preserving essential model capabilities under constrained spike budgets.

We will incorporate the clarified methodology, supporting references, and hardware limitations into the revised manuscript.


References

[1] Zheng et al. Distribution-Aware Multi-Granularity Phase Coding: Towards Lower Conversion Error for Spike-Driven Large Language Models. ICLR 2026.

[2] Horowitz. Computing's Energy Problem (and What We Can Do About It). ISSCC 2014.

[3] Zhou et al. Spikformer: When Spiking Neural Network Meets Transformer. ICLR 2023.

[4] Chen et al. LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models. AAAI 2026.

[5] Muir and Sheik. The Road to Commercial Success for Neuromorphic Technologies. Nature Communications, 16, 3586, 2025.

AuthorsOpenReview ↗

Reply to Reviewer wJ9y — W3. Offline Calibration Cost

We thank the reviewer for highlighting the importance of quantifying offline calibration overhead. We measured the calibration costs on NVIDIA L20 GPUs across four LLMs and report both total computational costs and their breakdown.

1. Total offline calibration cost.

The measurements cover the corresponding offline calibration workflows, including reference construction, operator fitting, and response selection and confirmation.

TLLaMA-2-7B (GPU·h)LLaMA-3-8B (GPU·h)Qwen2.5-7B (GPU·h)Qwen3-8B (GPU·h)
519.7421.63——
622.1826.05——
725.2829.02——
827.2730.9826.9731.69

For the LLaMA models, total calibration costs range from 19.74 to 30.98 GPU·h across T=5–8. At T=8, the corresponding costs for Qwen2.5-7B and Qwen3-8B are 26.97 and 31.69 GPU·h, respectively.


2. Cost breakdown and deployment implications.

At T=8, we further separate the two post-Softmax search passes from the remaining calibration stages:

ComponentLLaMA-2-7BLLaMA-3-8BQwen2.5-7BQwen3-8B
Two post-Softmax search passes24.0325.3622.9827.47
Remaining offline stages3.245.623.994.22
Total (GPU·h)27.2730.9826.9731.69

The two post-Softmax search passes account for 88.1%, 81.9%, 85.2%, and 86.7% of the total calibration costs, respectively. The remaining costs arise from reference construction, operator fitting, and other calibration operations. These results identify post-Softmax response search as the primary source of offline computational overhead across all four models.

Importantly, calibration is performed once per model–timestep configuration, rather than during inference. The pretrained weights remain frozen, and only existing post-Softmax response coefficients are modified.

The resulting models therefore preserve the original deployment timestep count, response-parameter storage, and few-spike inference structure, without requiring additional model-level validation or optimization during inference.

These measurements quantify the offline computational tradeoff for recovering long-context capabilities under a fixed deployment budget. We will incorporate the complete calibration costs and stage-wise accounting into the revised manuscript.

AuthorsOpenReview ↗

Reply to Reviewer wJ9y — Q1. Motivation for Large-Scale Spiking LLMs

We appreciate this fundamental question. We agree that current GPU implementations do not establish an immediate efficiency advantage for large-scale SNNs, and practical neuromorphic deployment still faces significant challenges.

However, the scientific value of developing large-scale spiking models should not be determined solely by the maturity of today's hardware. We see three important motivations.


1. Energy-efficient large-scale intelligence is an important research objective.

The growing computational and energy demands of LLMs motivate research into alternative computing paradigms. The International Energy Agency [1] identifies AI as a major driver of increasing data-center electricity demand.

SNNs offer a promising direction through sparse, event-driven computation and temporal information encoding, which may enable more energy-efficient inference on suitable hardware. Whether these advantages can be realized at LLM scale remains an open research problem.

Importantly, limited current hardware support is a reason to advance algorithm–hardware co-design, not to abandon the investigation of potentially more efficient representations. As emphasized by Schuman et al. [2] and Muir and Sheik [3], neuromorphic computing requires the joint development of algorithms, hardware, and applications. Algorithms capable of preserving LLM functionality under limited spike budgets are an essential part of this research.


2. Spiking models offer opportunities beyond conventional GPU inference.

SNNs also have potential relevance to computational neuroscience, neural signal processing, and brain–computer interfaces (BCIs). Their spike-based communication and temporal dynamics are naturally compatible with important characteristics of biological neural signaling. Existing work has investigated SNN-based neural decoding for brain–machine interfaces [4].

Developing more capable spike-based models may eventually help connect event-driven neural processing with the high-level semantic and reasoning capabilities of language models, creating opportunities for future brain-inspired intelligent systems and neural interfaces.

We present this as a longer-term research motivation, not as a demonstrated BCI application or a claim that converted LLMs faithfully model biological neurons.


3. DCPN addresses a concrete bottleneck in making large-scale spiking models useful.

An energy-efficient representation is valuable only if it preserves the capabilities required by the application. Our work shows that conventional language-modeling metrics can conceal severe long-context degradation after low-timestep conversion.

On LLaMA-2-7B, DMPC at T=8 achieves a WikiText-2 PPL of 6.108, relatively close to the ANN's 5.472, yet its 4K RULER and NeedleBench scores fall to 5.80 and 4.89, compared with 75.23 and 67.22 for ANN.

DCPN addresses this failure without increasing the inference timestep count or response-parameter storage. At T=8, it achieves 44.49 on RULER and 57.86 on NeedleBench, even exceeding DMPC at T=10 (40.20 and 49.91, respectively).

These results demonstrate that improving the quality of spike-based information representation can be more effective than simply increasing the temporal computation budget.

Our contribution is therefore not a claim of immediate GPU acceleration or experimentally verified neuromorphic energy savings. Rather, DCPN advances the algorithmic feasibility of capable, low-timestep spiking LLMs—an important prerequisite for investigating their eventual efficiency benefits and broader applications.

In short, developing accurate large-scale SNNs and advancing neuromorphic hardware are complementary goals. Establishing how modern LLM capabilities can be preserved in spike-based representations is a scientifically meaningful objective in its own right.


References

[1] International Energy Agency. Energy and AI. 2025.

[2] Schuman et al. Opportunities for Neuromorphic Computing Algorithms and Applications. Nature Computational Science, 2022.

[3] Muir and Sheik. The Road to Commercial Success for Neuromorphic Technologies. Nature Communications, 2025.

[4] Taeckens and Shah. A Spiking Neural Network with Continuous Local Learning for Robust Online Brain Machine Interface. Journal of Neural Engineering, 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Abstract: Summary of Updates and Response to Reviewer Feedback

We thank all reviewers for their thoughtful feedback and constructive suggestions. We have uploaded a revised manuscript and updated the abstract to reflect the expanded evidence and clarify the paper’s contributions.

The main revisions are:

  • Expanded evaluation. We incorporated Qwen results extending to 16K on RULER and 64K on NeedleBench, together with Qwen3 multi-hop QA results and comparisons with additional spiking LLM methods. The main DCPN–DMPC comparisons consistently use eight timesteps.
  • Stronger analysis and transparency. We added cross-model response diagnostics, task-group ablations and paired confidence intervals, clarified calibration and evaluation protocols, and reported offline calibration costs. We also distinguish theoretical energy estimates from measured hardware efficiency and explicitly discuss remaining performance gaps.
  • Improved organization. We reorganized the appendix, consolidated redundant material, and clarified the relationship between the main results and supplementary diagnostic experiments.

These revisions benefited substantially from the reviewers’ suggestions. We hope the updated manuscript and our responses address your concerns and make the evidence easier to assess. If you find that your concerns have been satisfactorily resolved, we would be grateful if you would consider updating your assessment and score accordingly. We welcome any remaining questions and sincerely appreciate your time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.