- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies zeroth-order adaptation of spiking LLMs at different neuron resolutions. It derives a bound in which accumulated conversion error contributes to the gap for the smoothed spiking objective, then proposes high-resolution ZO task learning followed by progressive block-wise teacher-student representation alignment at low resolution. Experiments on two conversion frameworks and several NLP tasks show gains over direct low-resolution ZO fine-tuning.
Critical strengths
- The paper identifies an empirically relevant failure mode—direct ZO adaptation degrades after reducing neuron resolution—and proposes a concrete high-to-low-resolution transfer procedure. The method is tested with two conversion frameworks rather than a single implementation.
- The final students improve over direct low-resolution ZO baselines in all three reported model/framework settings, and the block-wise-alignment ablation tests an important component of the proposed method.
Critical weaknesses
- The paper’s central memory-efficiency motivation is not directly validated. It argues that BPTT becomes costly as Tsp increases, but provides neither an SNN-BPTT baseline nor measurements of peak memory, or energy consumption across resolutions. The theory instead analyzes conversion error in the ZO-smoothed objective, not the cost or feasibility of BPTT. Moreover, the proposed second-stage block-wise alignment uses surrogate-gradient backpropagation, so the complete framework is not end-to-end forward-only. It is therefore unclear whether the method delivers an actual system-level efficiency advantage.
- The resolution study is narrow and its presentation obscures the end-to-end comparison. Table 1 does not state the exact (N, Tsp) configurations and reports only direct ZO baselines; the proposed two-stage method appears separately in Table 2. The paper also does not study one-step spiking settings, where the temporal BPTT motivation would be materially weaker, nor does it report a unified accuracy, memory, and runtime comparison. Finally, all experiments use only two older approximately 7B model families, leaving scaling behavior and the query/variance cost of ZO at larger scales unestablished.
Questions for authors
-
Please compare with a feasible SNN BPTT/surrogate-gradient baseline and, where relevant, truncated or checkpointed variants. Report peak memory and energy consumption for the complete pipeline, including block-wise alignment, across the reported resolutions.
-
The seed-based reconstruction in MeZO avoids storing the perturbation vector, but it does not remove the two-sided forward-query cost or estimator variance. What number of random directions and loss evaluations is used per update in practice? Please report sensitivity to the perturbation budget and random directions, together with peak memory and energy cost. A comparison with feasible BPTT/TBPTT or checkpointed surrogate-gradient baselines would clarify whether seed-based ZO provides a genuine end-to-end efficiency advantage for these Tsp settings.
-
Please state the exact high- and limited-resolution configurations directly in Table 1 and provide an integrated accuracy / memory / infer-time comparison including the final method. Can the method be evaluated or at least discussed for one-step spiking representations and for larger/newer model families?