acceptodds
Under review as a conference paper at ICLR 2027

Enabling Downstream Adaptation of Spiking LLMs: From Zeroth-Order Learning to Cross-Resolution Transfer

Abstract

Spiking large language models (LLMs) communicate through spike trains and support event-driven computation, offering the potential for energy-efficient language processing on neuromorphic hardware. However, adapting these models to downstream tasks remains challenging. Backpropagation through time incurs memory costs that grow with time steps and compound the substantial memory demands of LLMs, making forward-only ZO-SGD an attractive alternative. To examine its effectiveness, we derive a convergence bound for spiking LLMs that accounts for conversion error, suggesting that its accumulation may limit fine-tuning gains under limited-resolution neuron settings. Our empirical results are consistent with this analysis, showing that ZO-SGD fine-tuning remains effective at higher neuron resolutions but yields substantially smaller gains under limited-resolution settings. Motivated by these findings, we propose a two-stage framework that combines task learning with knowledge transfer across neuron resolutions to obtain task-adapted spiking LLMs with limited-resolution neurons. The task learning stage applies ZO-SGD at a high-resolution neuron setting to acquire task-specific knowledge. The knowledge transfer stage uses the adapted high-resolution model as a frozen teacher for progressive block-wise representation alignment, matching the limited-resolution model's intermediate representations to the corresponding teacher outputs. This teacher-guided alignment is designed to compensate for representation mismatch across neuron resolutions while preserving the task knowledge acquired during high-resolution fine-tuning. Experimental results show that the proposed framework improves downstream task performance over direct ZO-SGD fine-tuning under limited-resolution neuron settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spiking Large Language Models, Zeroth-Order Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a67yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies zeroth-order adaptation of spiking LLMs at different neuron resolutions. It derives a bound in which accumulated conversion error contributes to the gap for the smoothed spiking objective, then proposes high-resolution ZO task learning followed by progressive block-wise teacher-student representation alignment at low resolution. Experiments on two conversion frameworks and several NLP tasks show gains over direct low-resolution ZO fine-tuning.

Critical strengths
  1. The paper identifies an empirically relevant failure mode—direct ZO adaptation degrades after reducing neuron resolution—and proposes a concrete high-to-low-resolution transfer procedure. The method is tested with two conversion frameworks rather than a single implementation.
  2. The final students improve over direct low-resolution ZO baselines in all three reported model/framework settings, and the block-wise-alignment ablation tests an important component of the proposed method.
Critical weaknesses
  1. The paper’s central memory-efficiency motivation is not directly validated. It argues that BPTT becomes costly as Tsp increases, but provides neither an SNN-BPTT baseline nor measurements of peak memory, or energy consumption across resolutions. The theory instead analyzes conversion error in the ZO-smoothed objective, not the cost or feasibility of BPTT. Moreover, the proposed second-stage block-wise alignment uses surrogate-gradient backpropagation, so the complete framework is not end-to-end forward-only. It is therefore unclear whether the method delivers an actual system-level efficiency advantage.
  2. The resolution study is narrow and its presentation obscures the end-to-end comparison. Table 1 does not state the exact (N, Tsp) configurations and reports only direct ZO baselines; the proposed two-stage method appears separately in Table 2. The paper also does not study one-step spiking settings, where the temporal BPTT motivation would be materially weaker, nor does it report a unified accuracy, memory, and runtime comparison. Finally, all experiments use only two older approximately 7B model families, leaving scaling behavior and the query/variance cost of ZO at larger scales unestablished.
Questions for authors
  1. Please compare with a feasible SNN BPTT/surrogate-gradient baseline and, where relevant, truncated or checkpointed variants. Report peak memory and energy consumption for the complete pipeline, including block-wise alignment, across the reported resolutions.

  2. The seed-based reconstruction in MeZO avoids storing the perturbation vector, but it does not remove the two-sided forward-query cost or estimator variance. What number of random directions and loss evaluations is used per update in practice? Please report sensitivity to the perturbation budget and random directions, together with peak memory and energy cost. A comparison with feasible BPTT/TBPTT or checkpointed surrogate-gradient baselines would clarify whether seed-based ZO provides a genuine end-to-end efficiency advantage for these Tsp settings.

  3. Please state the exact high- and limited-resolution configurations directly in Table 1 and provide an integrated accuracy / memory / infer-time comparison including the final method. Can the method be evaluated or at least discussed for one-step spiking representations and for larger/newer model families?

Reviewer_PQioOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Zeroth-Order method for converting large language models (LLMs) into spiking neural networks (SNNs), which avoids the excessive memory overhead in SNN-LLM training caused by the multi time-step scheme. Theoretically, the paper provides a convergence bound for ZO-SGD. Empirically, the authors validate the efficiency of the zero-order conversion on two models, LLaMA2-7B and OPT-6.7B, using two SNN architectures, spikeLLM and spikingLLM, and further evaluate it on multiple downstream LLM tasks.

Critical strengths
  • The theoretical analysis in this paper is fairly solid, providing guarantees for applying the ZO method to SNN conversion.
  • The experiments in this paper are fairly comprehensive. The model selection, the adaptation to different SNN methods, and the breadth of evaluation benchmarks are all quite thorough.
Critical weaknesses
  1. The experimental results of MeZO-S are clearly better than those of S-MeZO. Why are the MeZO-S results not included as a baseline in Table 2? The authors should supplement this part of the experiments to make the paper more convincing.
  2. The paper adopts a block-wise alignment method. Can the authors provide a theoretical analysis of this method? It appears that the paper does not offer a more detailed theoretical analysis specifically for the methods it proposes/uses.
  3. The motivation of the paper stems from the prohibitive memory overhead of BPTT under the multi time-step scheme. Then, can the authors provide both theoretical and empirical analyses of the memory overhead, comparing their method against BPTT?
Questions for authors

Please see the weaknesses above.

Reviewer_moAkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors leverage forward-only ZO-SGD to train spiking LLMs on downstream tasks, thereby avoiding the memory costs of using standard training methods such as BPTT. Empirical results show that ZO-SGD fine-tuning remains effective at higher neuron resolutions but yields substantially smaller gains under limited-resolution settings. The authors further propose a two-stage framework that combines task learning with knowledge transfer across neuron resolutions to obtain task-adapted spiking LLMs with limited-resolution neurons. Empirically its demonstrated that this teacher-guided alignment helps to compensate for representation mismatch across neuron resolutions while preserving the task knowledge acquired during high-resolution fine-tuning.

Critical strengths

Motivation of the paper is strong. With growing scale of LLMs it is essential to develop energy efficient alternatives that can enable edge based deployment. Spiking LLMs are a promising candidate in that aspect. Furthermore, leveraging Zeroth order SGD for training Spiking LLMs is also a promising direction.

Critical weaknesses
  1. The actual training time, memory improvements from using the proposed approach is not explicitly highlighted in the paper.
  2. The paper does not use any other baseline for training the resulting spiking LLM. Training is a one time process and proposed approach requires two stages. It will be interesting to see how the performance actually compares with other baselines like BPTT, etc.
  3. While the proposed approach is spiking, there is no discussion on actual spiking dynamics of the model. It will be interesting to know energy/power consumption of the model.
Questions for authors
  1. How does the proposed approach compare with training using BPTT, with varying number of time steps of operation?
  2. How does the average spiking rate, energy consumption, etc. compare with the different stages of training?
  3. Since, the authors use ZO-SGD, how many actual training steps are required to achieve convergence? How does that compare to BPTT?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.