- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the performance degradation of ANN-to-SNN converted language models on long-context tasks. It identifies the loss of weak but collectively important attention probabilities in few-spike post-Softmax encoding as the primary bottleneck. To address this, the authors propose the Downstream-Calibrated Phase Neuron (DCPN), which optimizes phase thresholds and readouts to recover weak attention signals while preserving salient responses. Without modifying pretrained weights, timestep counts, or storage requirements, DCPN improves both standard language modeling and long-context understanding. Experiments demonstrate substantial gains on RULER and NeedleBench, with improvements of up to 52.97 percentage points at eight timesteps.
Critical strengths
-
This paper is in good written and easy to follow.
-
This paper focuses on SNN, a bio-inspired method, which is an interesting topic to save energy.
-
In the experimental results, this paper improves results in LLAMA-2 and LLAMA-3 in the NeedleBench compared with the proposed baselines.
Critical weaknesses
-
More comprehensive experimental comparisons are needed. Beyond the baseline considered in this paper, the authors should provide more detailed experimental comparisons with existing SNN attention mechanisms, along with further theoretical analysis.
-
The effectiveness of SNN attention on modern long-context tasks deserves further investigation. For example, modern LLMs have demonstrated strong reasoning capabilities. However, this work primarily focuses on LLaMA-based models and lacks further exploration of more challenging long-context / response tasks, particularly those requiring complex reasoning.
Questions for authors
N/A
2 comments
Reply to Reviewer gXkk — W2. Generalization
We thank the reviewer for encouraging broader evaluation on modern LLMs and challenging long-context tasks, particularly those involving reasoning and response generation.
1. Generalization across model families and longer contexts.
We extended our T=8 evaluation from LLaMA-2/3 to Qwen2.5-7B and Qwen3-8B, covering contexts up to 16K.
NeedleBench and WikiText-2 results:
| Model | Method | WT2 PPL ↓ | Needle 2K ↑ | Needle 4K ↑ | Needle 8K ↑ | Needle 16K ↑ |
|---|---|---|---|---|---|---|
| Qwen2.5-7B | ANN | 6.8486 | 96.97 | 98.11 | 97.16 | 96.69 |
| DMPC T8 | 7.2153 | 91.02 | 81.80 | 50.95 | 0.00 | |
| DCPN T8 | 6.9922 | 93.56 | 94.53 | 89.38 | 60.17 | |
| Qwen3-8B | ANN | 6.9946 | 96.69 | 96.29 | 93.84 | 94.22 |
| DMPC T8 | 7.7677 | 97.35 | 96.57 | 86.27 | 12.31 | |
| DCPN T8 | 7.4593 | 96.21 | 95.93 | 95.72 | 85.64 |
RULER results:
| Model | Method | RULER 2K ↑ | RULER 4K ↑ | RULER 8K ↑ | RULER 16K ↑ |
|---|---|---|---|---|---|
| Qwen2.5-7B | ANN | 94.60 | 92.23 | 90.61 | 88.23 |
| DMPC T8 | 91.30 | 75.35 | 40.94 | 1.36 | |
| DCPN T8 | 93.75 | 87.83 | 75.26 | 52.12 | |
| Qwen3-8B | ANN | 95.33 | 93.95 | 93.62 | 92.15 |
| DMPC T8 | 93.87 | 77.11 | 49.82 | 11.45 | |
| DCPN T8 | 94.24 | 88.87 | 75.18 | 59.34 |
RULER contains 13 tasks and 6,500 examples per configuration, while NeedleBench contains 208 cases across three equally weighted task groups.
At 16K, DCPN improves NeedleBench/RULER over DMPC by 60.17/50.76 percentage points on Qwen2.5 and 73.33/47.89 points on Qwen3. Although Qwen3 shows small NeedleBench decreases at 2K/4K, its improvements at 8K/16K are substantial. These results extend the observed benefits beyond LLaMA and demonstrate stronger long-context capability retention.
2. Long-context multi-hop reasoning and response generation.
We additionally evaluated Qwen3-8B at T=8 on three LongBench multi-hop QA tasks: HotpotQA, 2WikiMQA, and MuSiQue. Each task contains 200 examples per method, with input lengths spanning 1,018–17,898 tokens, evaluated using official LongBench F1.
| Task | ANN | DMPC T8 | DCPN T8 | Δ DCPN−DMPC | 95% paired CI |
|---|---|---|---|---|---|
| HotpotQA | 57.55 | 30.55 | 43.71 | +13.16 | [7.49, 18.88] |
| 2WikiMQA | 44.54 | 38.85 | 40.29 | +1.44 | [−3.81, 6.78] |
| MuSiQue | 31.72 | 11.97 | 22.80 | +10.83 | [6.08, 15.75] |
| Mean | 44.60 | 27.12 | 35.60 | +8.48 | [5.46, 11.59] |
DCPN improves the three-task mean by 8.48 F1 points (95% paired CI [5.46, 11.59]). The improvements on HotpotQA (+13.16) and MuSiQue (+10.83) have confidence intervals entirely above zero, while the smaller gain on 2WikiMQA (+1.44) is not statistically resolved.
These tasks require integrating evidence from multiple locations within long inputs, providing more challenging evaluations than single-fact retrieval. The remaining gap to ANN indicates partial recovery of long-context multi-hop QA capability.
Our original LongBench evaluation also covers document QA, summarization, and code completion. At T=8, the sample-weighted score improves from 42.62 to 53.43 on LLaMA-2-7B and 44.78 to 48.04 on LLaMA-3-8B (DMPC → DCPN), using untruncated native-window subsets.
3. Cross-family mechanism diagnostics.
We further conducted paired response diagnostics across 17 model–length configurations, using identical ANN attention/value tensors for DMPC and DCPN within each model. All configurations use T=8 and G=2.
Representative results at each model's longest evaluated context are shown below (DMPC → DCPN, all ↓):
| Model | Length | Lost mass (%) | Zero-response (%) | AV NMSE |
|---|---|---|---|---|
| LLaMA-2-7B | 4K | 14.45 → 11.27 | 99.13 → 98.04 | .025123 → .018503 |
| LLaMA-3-8B | 8K | 14.06 → 11.03 | 99.55 → 99.06 | .027375 → .012566 |
| Qwen2.5-7B | 16K | 23.79 → 19.87 | 99.75 → 99.46 | .035427 → .010074 |
| Qwen3-8B | 16K | 18.30 → 15.00 | 99.74 → 99.49 | .047654 → .010703 |
Across all 17 configurations, DCPN consistently reduces lost attention mass, zero-response ratio, and AV NMSE. At 16K, AV NMSE decreases by 71.56% on Qwen2.5 and 77.54% on Qwen3.
These diagnostics support the transferability of weak-attention repair across model families and link the observed capability recovery to improved attention–value reconstruction.
Together, our results demonstrate improvements beyond LLaMA and single-fact retrieval, extending to 16K long-context benchmarks, multi-hop QA, and response generation. The observed recovery is substantial but remains incomplete relative to ANN.
We will incorporate the expanded evaluations, multi-hop QA results, and cross-family mechanism diagnostics into the revised manuscript.
Reply to Reviewer gXkk — W1. Additional SNN Attention Comparisons and Theoretical Analysis
We thank the reviewer for suggesting broader comparisons with existing spiking attention mechanisms and further theoretical analysis.
1. Additional comparisons with existing SNN methods.
Beyond DMPC, we evaluated LAS (T=16), SpikedAttention (T=40), FAS (T=16), and SpikeLLM (T=2, W4A4) on the same LLaMA models and long-context benchmarks.
LLaMA-2-7B:
LLaMA-3-8B:
DCPN at T=8 achieves the highest long-context scores among these evaluated SNN configurations. Compared with LAS T=16, DCPN improves 4K NeedleBench/RULER by 4.73/3.10 percentage points on LLaMA-2 and 5.78/3.57 points on LLaMA-3. It also improves LLaMA-3 NeedleBench 8K by 13.39 points.
These comparisons extend beyond the original DMPC baseline. Since the methods employ different timestep counts, neuron designs, and conversion procedures, they are cross-method performance comparisons rather than strictly matched-compute comparisons.
2. Theoretical interpretation of weak-attention distortion.
Consider one attention row a and its value matrix V. Let F(a) denote the spike-decoded attention response and e=F(a)−a. The resulting attention-output error satisfies
\|\Delta o\|_2^2=e^\top VV^\top e.$$ Thus, elementwise attention reconstruction error and downstream attention-output error are not equivalent: the latter also depends on the geometry of the value vectors. Let $Z$ denote attention entries mapped to zero, with erased mass $m_Z=\sum_{i\in Z}a_i$. Their removed contribution is $\sum_{i\in Z}a_iv_i$. For example, if $k$ weak entries each carry mass $m/k$ with similar value vectors, once $m/k$ falls below the zero-response threshold, their **aggregate removed contribution remains approximately $m\bar v$**, despite each individual probability being small. This illustrates why a fixed zero-response region can become consequential in longer contexts. The original response diagnostics support this mechanism: on LLaMA-2 at T=6, erased attention mass increases from **16.35% at 512 tokens to 47.52% at 4K**, while AV NMSE increases from **0.0739 to 0.3599**. DCPN addresses this representation problem by adjusting the final phase's threshold and readout to recover weak probabilities, while coordinating earlier-phase readouts to preserve the coverage of larger probabilities. The AV screening objective directly measures the resulting attention-output distortion. We will incorporate the expanded comparisons and this theoretical explanation into the revised manuscript.