When Weak Attention Goes Silent: Repairing Context-Amplified Attention Distortion in Spiking Language Models

Spiking neural networks (SNNs) offer sparse, event-driven inference for large language models (LLMs). Converting pretrained artificial neural networks (ANNs) to SNNs with few timesteps can preserve performance on mainstream language-modeling and question-answering benchmarks, yet we find that converted models increasingly fail to use relevant information as input length grows. Controlled interventions identify the few-spike encoding of post-Softmax attention as the dominant recoverable bottleneck: its zero-response region suppresses weak but collectively substantial attention probabilities. Uniformly rescaling the response to recover these probabilities compromises coverage of larger ones. We propose the Downstream-Calibrated Phase Neuron (DCPN) to repair both ends of the response at a fixed deployment budget. DCPN adjusts the final phase's threshold and readout to make weak probabilities accessible with appropriately small outputs, while coordinating earlier-phase readouts to preserve coverage of salient probabilities. It screens candidate responses by reconstructing attention-weighted value outputs, then confirms them using validation negative log-likelihood in the progressively converted model. DCPN changes only post-Softmax response parameters, preserving pretrained weights, timestep count, response-parameter storage, and few-spike inference structure. It improves conventional language-modeling and short-input task performance across evaluated settings. At eight timesteps, RULER and NeedleBench scores increase by 38.69 and 52.97 percentage points on LLaMA-2-7B at 4K, and by 24.44 and 38.07 points on LLaMA-3-8B at 8K, respectively, relative to the conversion baseline.