acceptodds
Withdrawn from ICLR 2027

sLoTh: An Efficient Framework for Continual Learning in Spiking Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Physical AI systems operate in dynamic environments where data arrives continuously, requiring models to adapt while preserving previously learned knowledge under strict memory and energy constraints. While parameter-efficient fine-tuning (PEFT) has shown promise for continual learning with vision transformers, conventional architectures rely on dense matrix multiplication and remain costly for real-world deployment. Spiking transformers provide energy-efficient event-driven computation, yet their continual learning capabilities remain largely unexplored, specifically from a PEFT perspective. We here introduce and study sLoTh, a parameter-efficient continual learning framework for pretrained spiking vision transformers. sLoTh freezes the backbone and restricts plasticity to scalable-efficient low-rank attention updates (seLoRA) and shared neuronal threshold modulation, enabling adaptation without replay buffers by updating fewer than 1% of model parameters. Experiments across CIFAR-100, Tiny-ImageNet, ImageNet-100, and ImageNet-R with up to 100 tasks demonstrate competitive rehearsal-free performance in class-incremental learning and online continual learning, while enabling 17 lower inference energy consumption than conventional dense vision transformers.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Transformer, Spiking Neurons, Class Incremental Learning, Online Continual Learning, Spiking Threshold

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cB6eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how pretrained spiking Transformers can learn new classes while retaining previous knowledge. The backbone is frozen, while task-shared LoRA modules and channel-wise firing thresholds are updated. Distillation, parameter anchors, and nearest-class-mean classification are used to mitigate forgetting. The online variant updates teacher snapshots using loss- or representation-based triggers. Experiments cover several datasets and spiking backbones, showing benefits of joint adaptation in some multi-epoch class-incremental settings, while single-module variants perform better in single-pass online learning.

Critical strengths

The paper addresses a meaningful continual-adaptation problem. With the same QKFormer backbone, sLoTh reaches 77.10% on 20-task ImageNet-R, versus 75.26% for SD-LoRA (Table 3). Joint adaptation achieves 81.23% on CIFAR-100 CIL, versus 72.63% for LoRA-only, supporting component synergy in this setting.

Critical weaknesses
  1. seLoRA uses a standard low-rank update; CL-LoRA already incorporates shared LoRA and distillation, while CATFormer studies threshold-based continual adaptation in spiking Transformers. Combining pretrained backbones, shared thresholds, and stabilization differs from these methods, but the existing components themselves should not be treated as new mechanisms. Please identify and validate the specific additions.

  2. The matched-backbone CIL comparison is useful, but OCL lacks a frozen-backbone baseline using the same NCM classifier. The contributions of pretrained features, prototype classification, and adaptation are therefore unclear. Please compare no adaptation, standard shared LoRA, threshold-only adaptation, and the full method using the same checkpoint, inference procedure, and data-access budget.

  3. On 100-task Tiny-ImageNet OCL, the full method achieves 42.90%, versus 65.12% for LoRA-only. The authors report this degradation, and the demonstrated CIL synergy should be recognized. Nevertheless, the evidence does not establish that both mechanisms are generally necessary for long task sequences. Please explain the difference under matched stabilization objectives and limit the claims accordingly.

  4. Prototypes are described as being computed after a task or after training, although OCL lacks true task boundaries. The order of class registration and trigger-loss computation is also unclear. Please explain how class registration, loss-based triggering, and prototype accumulation operate in a single pass, how feature drift is handled, and all data accesses needed to verify the online protocol.

  5. The loss-based and RT variants change both the backbone and the trainable blocks, preventing attribution to the detector. The TIL upper bound in Figure 3 also changes task information at inference, so its gap cannot be attributed solely to unavailable training boundaries. Please hold the backbone, adaptation modules, and inference over all seen classes fixed when comparing boundary information or detectors, and report evidence of detection quality.

Questions for authors

Please see the weaknesses above.

Reviewer_pTGXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a rehearsal-free continual learning approach for spiking vision transformers called sLoTh. The approach is a combination of many known design principles. Essentially, for a frozen backbone, sLoTh adapts low-rank attention parameters and firing thresholds for the spiking transformer using distillation and parameter regularization to limit forgetting. The authors evaluate their approach and its individual components in multiple continual learning settings and show good performance overall.

Critical strengths
  • The paper addresses a relevant class of methods, spiking transformers, that are often not considered in other continual learning works.
  • The authors evaluate their approach on multiple datasets and across different settings.
  • The results show competitive performance in several settings, particularly for the threshold-only and seLoRA-only variants in online continual learning. The authors put considerable effort into providing a broad range of comparisons.
Critical weaknesses
  • In my opinion, the paper did not clearly establish its core contribution. It combines several known ideas with relatively limited modifications. It is unclear to me what the main insight of this work is.
  • This is further exacerbated by the fact that the combination helps most in the class incremental setting while subparts of the approach perform significantly better in the online setting. The paper does not sufficiently explain these differences or provide guidance on which configuration to use.
  • The discussion of the results is mostly descriptive and often repeats the content of the tables. I found it somewhat difficult to follow overall and it would benefit from a clearer separation between comparisons with other methods, ablations, and efficiency experiments etc.
  • Common and well-known baselines such as SimpleCIL and RanPAC are also missing. This is needed in my opinion to help establish how much the proposed adaptation adds beyond using pretrained features with simple classifiers.
Questions for authors
  • What are the main insights a reader should take from this work? Particularly I would like to understand when does combining the threshold modulation and the low rank adaptation improve performance?
  • In a related question: Why does the full method perform worse than its individual components in the online continual learning setting?
Reviewer_hpLJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes sLoTh, a rehearsal-free framework for adapting pretrained spiking vision transformers through shared low-rank attention updates and channel-wise firing-threshold modulation. Training combines logit distillation with local and global parameter anchors, while inference uses class prototypes. Loss-based and representation-based triggers extend the framework to boundary-free online learning. Validation covers class-incremental, task-incremental, and online settings across image and event-based datasets and several backbones.

Critical strengths

The paper addresses an important problem: enabling continual adaptation while limiting forgetting and resource consumption. Its focus on parameter-efficient learning in spiking transformers is relevant to resource-constrained applications.

Critical weaknesses
  1. The principal OCL comparison does not establish algorithmic superiority. Table 4 labels the imported replay-based baselines as using ViT backbones. However, the cited S6MOD paper explicitly reports these results using ResNet-18 without pretraining and a streaming batch size of 10, whereas sLoTh uses pretrained spiking transformers and reports batch sizes of 128 for these datasets. The source also reports averages over ten runs, while Table 4 states three runs. These discrepancies substantially weaken the comparison: the accuracy differences cannot be attributed to the proposed continual-learning mechanism.

  2. The operational validity of rehearsal-free OCL is unclear. The manuscript does not fully define task-free class registration, classification-loss masking, or prototype maintenance under changing representations. These omissions affect whether the reported results satisfy the stated learning protocol and can be reproduced.

Questions for authors
  1. The boundary-free online learning procedure is insufficiently specified. The training objective uses cross-entropy over the current task’s classes, although task boundaries are unavailable in OCL. Moreover, class registration is described as occurring after a loss-based novelty trigger, leaving unclear how the loss is computed when a batch contains an unregistered label. The authors should specify the active classification space, the order of class registration and loss computation, and the alignment of teacher and student logits after classifier expansion.

  2. Although the backbone weights are frozen, threshold modulation and low-rank updates change the feature extractor. Consequently, prototypes computed for earlier classes may become incompatible with features produced by the updated model. The manuscript does not explain whether historical prototypes remain fixed, undergo correction, or are recomputed. For OCL, computing prototypes after completing a task also requires clarification because samples are supposed to be observed only once.

  3. Appendix A.2.2 defines seLoRA through the standard LoRA parameterization; the apparent distinction lies in sharing adapters across tasks and applying the proposed stabilization objective. The paper should therefore clarify whether seLoRA introduces a new adaptation mechanism or names a particular continual-training configuration. The seLoRA-only ablation should be explicitly identified as a shared-LoRA baseline where appropriate. A frozen-backbone nearest-class-mean baseline is also essential to establish how much performance comes from adaptation beyond pretrained representations and prototype inference. Pretraining datasets and checkpoints should be disclosed to make this comparison interpretable.

  4. Important reproducibility and reporting details remain missing. Table 8 omits the LoRA rank and initialization, exact adaptation locations, regularization coefficients, distillation temperature, cosine-classifier scale, detector parameters, and RT prototype construction procedure. Its multi-epoch settings also need to be distinguished explicitly from the single-pass OCL configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.