acceptodds
Withdrawn from ICLR 2027

Selective Backpropagation for Efficient Few-Shot Class-Incremental Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-Shot Class-Incremental Learning (FSCIL) requires models to continuously learn new classes from limited samples while retaining prior knowledge, under strict constraints on compute and memory. Existing approaches lie along a difficult trade-off: simple fine-tuning is computationally efficient but suffers from catastrophic forgetting, replay-based methods mitigate forgetting at the cost of substantial compute and memory, and exemplar-free methods often reduce forgetting by freezing most of the backbone, improving efficiency at the expense of adaptability. We propose Selective Backpropagation (SBP), a deterministic parameter budgeting framework that bridges this gap. SBP restricts gradient updates to a pre-allocated subset of network parameters, freezing past knowledge and preserving unbiased capacity for future learning, enabling rapid adaptation without costly mask optimization. We show that SBP achieves strong performance across standard FSCIL benchmarks while requiring training time close to that of naïve fine-tuning and substantially lower training time than prior SOTA methods. Crucially, our experiments expose a limitation of standard FSCIL evaluation: performance on short, distribution-consistent benchmarks does not necessarily predict behavior under distribution shift or over substantially longer learning horizons. We therefore evaluate FSCIL methods in cross-domain settings and over an 80-session ImageNet-1K stream. SBP remains strong across these regimes while maintaining low training cost, providing a favorable stability–plasticity–efficiency trade-off.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
class-incremental learning, continual learning, computer vision, few shot class incremental learning, few shot learning, lifelong learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cHNbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective Backpropagation (SBP), a parameter-budgeting framework for exemplar-free few-shot class-incremental learning. The method divides network parameters into three groups: previously assigned parameters that are frozen, currently assigned parameters that can be updated, and free parameters reserved for future tasks. The free parameters are randomly reinitialized at each training epoch to prevent the model from relying on them. The authors also combine SBP with prototype-based weight imprinting and a partial-distance classifier to support incremental learning. Experiments are conducted on CIFAR-100, MiniImageNet, and CUB-200, along with cross-domain FSCIL settings and a long-horizon ImageNet-1K benchmark.

Critical strengths
  1. The proposed parameter allocation strategy is simple and does not require additional optimization to learn importance masks. This makes the method relatively easy to implement and potentially useful for resource-constrained continual learning.
  2. The paper goes beyond standard FSCIL benchmarks by considering cross-domain and long-horizon learning scenarios. These experiments are useful for understanding how different methods perform under distribution shifts and over longer learning sequences.
  3. The authors provide ablation studies on parameter isolation, free-parameter reinitialization, and prototype imprinting. The results help show how these components contribute to the overall performance.
Critical weaknesses
  1. SBP randomly selects parameters for each incremental session, but the authors do not clearly explain why random selection is a good choice. Different parameters and layers may play different roles in learning new classes. It would be helpful to compare random selection with other strategies, such as importance-based or gradient-based parameter selection, to better justify this design.
  2. SBP uses a predefined parameter budget, with a larger portion assigned to the base session and a fixed portion to each incremental session. However, different tasks may require different amounts of capacity, especially under distribution shifts. It is unclear whether a fixed budget works well across different learning scenarios. The authors should further investigate how the allocation ratio affects performance and discuss what happens when the available parameters are exhausted.
  3. The relationship between parameter-level masks and feature-level masks in the partial-distance classifier is not very clear. The authors should explain how the feature dimensions are selected based on the assigned parameters, particularly when using different backbone architectures.
  4. The paper emphasizes that SBP achieves a good balance between stability and plasticity. However, most experiments only report overall classification accuracy across incremental sessions. This makes it difficult to tell whether the improvements come from better retention of old classes or better learning of new classes. It would be helpful to report base-class accuracy, novel-class accuracy, and forgetting measures separately across sessions.
  5. The cross-domain experiments are interesting, but the analysis is somewhat limited. A more detailed analysis of representation changes and performance on old and new classes would help explain why SBP performs better in these settings.
Questions for authors
  1. What is the main advantage of random parameter assignment over importance-based or gradient-based selection? Have the authors considered comparing these strategies under the same parameter budget?
  2. How sensitive is SBP to the predefined parameter allocation ratios? Would an adaptive allocation strategy work better when incremental tasks have different levels of complexity or distribution shifts?
  3. How are the feature-level masks in the partial-distance classifier determined from the parameter-level masks? Does this mapping work consistently across different backbone architectures?
  4. Can the authors provide separate results for old-class retention and new-class learning? This would help clarify whether the improvements mainly come from reducing forgetting or improving adaptation.
1 comment

We thank the reviewer for the constructive comments and questions. We denote the reviewer's weaknesses and questions as W1, W2, … and Q1, Q2, ….

W1,Q1: SBP deliberately avoids importance- or gradient-based selection, as used by prior masking-based methods such as SoftNet (Kang et al., 2024). SBP pre-allocates future capacity without letting current training implicitly shape it: free parameters remain unused and are reset at the beginning of each epoch, so later sessions start from unbiased capacity and earlier tasks do not depend on parameters needed later (L78-83). Empirically, random allocation outperforms the two importance-based baselines in our tables: SoftNet (learned importance scores) and WaRP (gradient-based importance in a rotated weight space). SBP improves final accuracy over SoftNet / WaRP by 3.14 / 3.16 points on CIFAR-100 (Table 1), 9.12 / 7.21 on MiniImageNet (Table 2), and 7.49 / 2.45 on MiniImageNet→CUB-200 (Table 3), while training 4.79× / 1.43× faster on CIFAR-100. Over 80 ImageNet-1K sessions, WaRP falls to 12.90% final accuracy versus 39.08% for SBP (Table 4).

W2,Q2: Our sensitivity study (Table 11) varies the base budget from 0.5 to 0.99. Performance is stable across the practically relevant range of 0.75 to 0.99, with average accuracy varying by only 2.38%. Thus, SBP does not require precise dataset-specific tuning. The same fixed schedule is used in the cross-domain experiments (Tables 3 and 9) without retuning.

Regarding exhaustion: like prior parameter-isolation methods (e.g., PackNet, SoftNet), SBP uses a fixed backbone, so parameter capacity will eventually saturate in never-ending scenarios. Once capacity runs out, the model behaves like frozen-backbone methods such as FACT and NC-FSCIL: all prior parameters stay frozen, and novel classes are still added through prototype imprinting. We state this limitation in the revised paper (Sec. 5). To test SBP over a much longer horizon, we constructed an 80-session ImageNet-1K benchmark (Table 4), where SBP retains 39.08% final accuracy, compared with 18.40% (OrCo), 14.50% (ADBS), and 12.90% (WaRP).

W3, Q3: SBP allocates capacity at the granularity of output channels: each output channel of a convolutional or linear layer, together with its incoming weights, is assigned to at most one session and frozen thereafter. In ResNet-18, the 512 output channels of the last convolution index the 512 feature dimensions after the residual sum and global average pooling (Appendix A.1). The feature-level mask for session j is the set of these channels assigned at session j, and the active mask is their union over all sessions up to the current one. The ViT follows the same procedure, with the feature-level mask given by the allocated output units of the layer that produces the final features. We add this in our revised paper (L141-143).

W4, Q4: We agree that overall accuracy alone does not separate these effects. Below are base-class and novel-class (all incremental classes seen so far) accuracy on CIFAR-100 (single run):

MethodBase S1Base S9Δ BaseNovel S9Overall S9
SBP82.779.8−2.926.658.5
SoftNet80.174.8−5.327.755.9
WaRP79.873.2−6.625.654.1
OrCo80.357.6−22.743.351.9

SBP loses only 2.9 points on base classes across all 8 incremental sessions, versus 5.3–22.7 for the baselines, while its novel-class accuracy is comparable to the other backbone-updating methods (SoftNet, WaRP). OrCo learns novel classes better but forgets substantially more. On this in-distribution benchmark, SBP's gains therefore come mainly from reduced forgetting.

W5: We thank the reviewer for this suggestion and agree that overall accuracy alone does not explain the cross-domain results. Below we report base-class (MiniImageNet) and novel-class (CUB-200, all seen so far) accuracy on MiniImageNet→CUB-200 (single run) for SBP and OrCo, a frozen-backbone method:

Session12357911
SBP base80.179.077.271.070.366.966.7
SBP novel–19.015.511.28.06.75.5
OrCo base69.968.868.668.267.366.866.2
OrCo novel–4.52.31.61.01.00.8

To measure representation change, we compute the mean cosine similarity between base-class backbone features after the base session and after the final session: 0.983 for SBP, and 1.00 for OrCo by construction. SBP's backbone continues to adapt while preserving most of the base representation, and learns substantially more CUB classes than OrCo, whose fixed features learn almost none (0.8%). The cost of this adaptation is a larger drop in base accuracy (80.1% to 66.7%, vs. 69.9% to 66.2% for OrCo), and learning 200 CUB classes from five shots remains challenging for both. We will include this analysis for all methods in the final version and revise Sec. 3.2 accordingly.

Reviewer_o2y2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SBP addresses exemplar-free few-shot class-incremental learning by randomly allocating parameters to tasks, freezing previously allocated parameters, and repeatedly reinitializing free parameters. It combines this scheme with prototype classification over progressively activated feature dimensions, and evaluates accuracy and training time on standard, cross-domain, and long-horizon benchmarks.

Critical strengths
  1. The method is simple and addresses a useful accuracy/efficiency trade-off without requiring mask learning or replay.

  2. I appreciate the cross-domain and 80-session evaluations, alongside the timing results and component ablations.

Critical weaknesses
  1. A major drawback, in my opinion, is that each task is allocated a fixed set of parameters, which means that if all parameters have been allocated, there would be no parameters available for new tasks, limiting its applicability in real scenarios.

  2. When new dimensions become active, old prototypes retain coordinates recorded before those dimensions were trained. How do these values remain compatible with current embeddings? The zero-mean argument in Appendix H does not establish this compatibility or prediction invariance.

  3. Could you explain how reliably optimization removes dependence on free parameters, and whether repeated resetting requires more training to converge? For the pretrained CUB backbone, does the same procedure discard useful pretrained information, and how is that handled?

  4. The ablation in Table 5 and the t-SNE results in Appendix I support the usefulness of resetting, but do not directly establish that predictions become insensitive to free parameters. Could you provide a direct test of this claim?

Questions for authors

Please refer to the Critical Weaknesses.

1 comment

We thank the reviewer for the constructive comments and questions. We denote the reviewer's weaknesses and questions as W1 and Q1, Q2, ….

W1: We agree that, similar to most prior masking-based CL methods, SBP requires a fixed backbone. Thus, parameter capacity will eventually saturate in never-ending scenarios. Once the backbone capacity runs out, the model behaves similarly to frozen-backbone methods: all prior parameters stay frozen in future sessions, and novel classes are still added through prototype imprinting. This is the regime that most FSCIL methods such as FACT, NC-FSCIL, Softnet, and ALICE adopt from the very first incremental session. We state this limitation explicitly in the revised paper (L499-505). However, in order to demonstrate that SBP remains effective under much longer settings which may be more applicable to real-world scenarios, we constructed the 80-session ImageNet-1K benchmark (Table 4). We show that SBP retains 39.08% final accuracy, compared with 18.40% (OrCo), 14.50% (ADBS), and 12.90% (WaRP). This demonstrates that, although SBP, like most other FSCIL methods, is limited when it comes to capacity saturation, it is better suited for real-world environments in which tasks may grow exceedingly larger.

W2: We agree with the reviewer, the zero-mean argument in Appendix H does not establish this compatibility. In our partial-distance formulation (Appendix A.1), the active mask is shared across all classes. A prototype stored at session tt is therefore compared, at later sessions, on dimensions activated after tt, whose coordinates were recorded while the corresponding parameters were still free. These coordinates are not guaranteed to be compatible with the current embedding; the mask only excludes dimensions that are still free. In practice, however, their effect on classification is limited. For base classes, they make up at most the 15% of feature dimensions allocated after the base session, and the distance is dominated by the dimensions allocated at tt, whose final-layer parameters remain frozen afterwards. The base-session budget study (Table 11) tests this directly, because the base budget sets the fraction of later-activated dimensions in base-class prototypes: about 1% at a 99% budget, 15% at 85%, and 50% at 50%. If the mismatch harmed old-class classification, forgetting would increase as the budget decreases. However, in our study, the drop from Session 1 to Session 9 is 25.8 points at 99%, 24.8 at 85%, and 22.7 at 50%. We clarify this in the revised paper (L737-744, 997-999).

W3: Because free parameters receives no gradients and is redrawn from the initialization distribution at every epoch, training minimizes an estimate of the expected loss over random draws of free parameters, rather than the loss at one fixed realization; any solution relying on a particular draw is penalized at the next redraw. We add this in our revised paper (L173-176). Reinitialization does not increase the training budget: SBP uses the same number of epochs as the tested baselines and has the shortest base-session training time among the compared FSCIL methods (Table 6). On CUB-200, the reviewer is correct that parameters in the free pool lose their ImageNet-pretrained values at the first reinitialization, while the 85% assigned in the base session retain them. To limit this loss, we also tried selecting the 15% free pool using parameter importance in the pretrained backbone, so that the least important pretrained weights are reinitialized. This had minimal effect on accuracy compared with random selection, so we use random selection for simplicity. Even with this loss, SBP achieves the best CUB-200 results among compared methods (Table 7), and we will discuss this trade-off explicitly.

W4: We can see the usefulness of free-epoch reinitialization in each incremental session. At each session, a subset of previously free parameters is assigned and trained, so their values change from those present when earlier classes were learned. If old-class predictions depended on these parameters, old-class accuracy would collapse at that point. This is exactly what happens without reinitialization: the w/o Free-Init variant, which freezes previous parameters identically to SBP, drops from 82.60% to 1.88% after the first incremental session (Table 5), and its base-class clusters collapse (Figure 2(a)). With reinitialization, old-class accuracy degrades only gradually, and base-class clusters remain compact and well separated (Figure 2(b)).

Reviewer_LpwUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose Selective Backpropagation (SBP), a deterministic parameter budgeting framework,by assigning parameters as previous, assigned, and free, and updating different parameters at different stages, FSCIL is completed. Not only did the experiment be conducted on the standard settings of this task, but also on cross domain scenarios and 80 sessions, achieving good results. However, there is insufficient research on paper related work and a lack of comparison with the latest work. In my view, the author's attitude towards this article is not serious enough, and they have not been able to organize and compare the latest related work and research results, so the persuasiveness of the relevant theories in the paper is very limited.

Critical strengths

Comprehensive Experimental Evaluation. The paper conducts extensive experiments across standard FSCIL benchmarks, cross-domain settings, and long-horizon learning scenarios, including 80 incremental sessions on ImageNet-1K. These evaluations demonstrate the effectiveness and computational efficiency of SBP under diverse learning conditions, providing strong empirical support for its practical applicability.

Critical weaknesses

Lack of recent related work review and comparison. Currently, the most recent referenced works include only two papers from 2025 [1][2]. However, some recent studies have not been reviewed. For example, Tri-WE (CVPR 2025) [3] allows the model to continue updating with limited incremental samples and preserves historical knowledge through weight-space ensembling. HyCal (CVPR 2026) [4] balances knowledge retention and new-class adaptation through training-free prototype calibration. [1] Shuo Li, Fang Liu, Licheng Jiao, Lingling Li, Puhua Chen, Xu Liu, and Wenping Ma. Promptbased concept learning for few-shot class-incremental learning. IEEE Transactions on Circuits and Systems for Video Technology, 35(5):4991–5005, 2025. [2] Linhao Li, Yongzhang Tan, Siyuan Yang, Hao Cheng, Yongfeng Dong, and Liang Yang. Adaptive decision boundary for few-shot class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 18359–18367, 2025. [3] Juntae Lee, Munawar Hayat, Sungrack Yun; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 15329-15338 [4] Eunju Lee, MiHyeon Kim, JuneHyoung Kwon, Yoonji Lee, JiHyun Kim, Soojin Jang, YoungBin Kim; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 29462-29471

Questions for authors

Q1:Freezing parameters, Parameter Stability is not equivalent to Representation Stability. How to ensure reliability on old categories? Q2:How to ensure that new tasks do not rely on the free parameter. Q3:In Table 11 of Appendix G, as the Base Budget increases, the AVG also increases. Does this indicate that the main performance advantage of the method depends on the powerful features already learned in the initial stage? Q4: Is there a significant difference in the results of training ResNet-18 from scratch for 60 categories under full supervision in the basic stage? The ultimate accuracy advantage of SBP may partly come from better initial representation?

1 comment

We thank the reviewer for the constructive comments and questions. We denote the reviewer's weaknesses and questions as W1 and Q1, Q2, ….

W1: To ensure a fair comparison, we reproduced and evaluated all baselines under the same experimental settings and ResNet-18 backbone as SBP (Sec. 3, Appendix A). We therefore restricted our comparison to methods that (i) have publicly available code, and (ii) were developed with backbones similar to ours (i.e., ResNet). Neither suggested work currently meets both criteria:

Tri-WE (CVPR 2025): No official implementation of their work is made publicly available. HyCal (CVPR 2026): official code is available, but HyCal is a training-free calibration method built on frozen CLIP embeddings with image–text feature fusion, and its implementation supports only CLIP backbones. As discussed in Sec. 4, such approaches rely heavily on massive external pre-training datasets. This departs from the strict, learning-from-scratch FSCIL setting studied in this paper.

We agree both works are still relevant to FSCIL, and have cited them in our revised paper (L456-458, 471).

Q1: We agree that freezing the previous parameters does not guarantee representation stability, since newly assigned parameters in earlier layers can change the inputs to frozen later layers. Our ablation demonstrates this: the w/o Free-Init variant (Table 5) freezes θ_prev identically to SBP, yet its old-class representation collapses: accuracy falls from 82.60% to 1.88% after one session, and the base-class clusters in Figure 2(a) collapse with it. With free-parameter reinitialization, the same clusters remain compact and well separated (Figure 2(b)). SBP converts parameter stability into representation stability in three ways: (1) the only parameters updated in later sessions come from θ_free. These were reinitialized at every epoch of earlier sessions, so old-class representations were trained to be insensitive to them (Sec. 2.2, Appendix H); batch normalization running statistics are frozen after base training, following common practice [1]. Otherwise, each incremental session would re-estimate them from only 25 images of 5 new classes, shifting the normalization of every layer, including fully frozen channels, toward the new classes. [1] Haeyong Kang, Jaehong Yoon, Sung Ju Hwang, and Chang D. Yoo. “Continual Learning: Forget-Free Winning Subnetworks for Video Representations.” IEEE TPAMI, 47(3):2140–2156, 2024.

Q2: Free parameters receive no gradients and are reinitialized at the start of every training epoch, so the model never sees a consistent mapping through them. Any solution that depended on particular free-parameter values would break at the next reinitialization. The optimizer therefore has to learn representations that rely only on previous and assigned parameters (Appendix H). Without reinitialization, free parameters act as a fixed signal that the assigned parameters co-adapt to. When those parameters are later assigned and updated, that dependency breaks. This causes the collapse in Table 5 (w/o Free-Init) and Figure 2(a). With reinitialization, every incremental session assigns and trains previously free parameters. Yet old-class accuracy degrades only gradually, and base-class clusters remain well separated (Figure 2(b); see also Q1).

Q3, Q4: A larger base budget gives more trainable capacity to the 60 data-rich base classes, so it yields a stronger initial representation, as it would for any FSCIL method. However, Table 11 shows that the budget mainly changes the starting point, not how well SBP retains it:

  • The gain in average accuracy is driven by Session 1. From 50% to 95%, Session 1 accuracy rises by 8.40 points and average accuracy by 7.06 points, while the drop from Session 1 to Session 9 stays within 22.7–25.8 points. At every budget, final accuracy retains 68.5–70.0% of Session 1 accuracy.
  • From 75% to 99%, average accuracy varies by only 2.38 points and is not monotonic: 99% is below 95%, and final accuracy at 90% is below 85%.
  • SBP's incremental performance does not depend on a strong base representation. At 50%, its Session 1 accuracy (74.87%) is 4.3–5.7 points below FACT, NC-FSCIL, OrCo, Comp, and ADBS (Table 1), yet its final accuracy (52.14%) is higher than all of them (45.67–51.95%). Regarding Q4, base accuracy does differ across methods (Table 1, Session 1), and a better initial representation benefits every method. The 50% setting in Table 11 shows, however, that SBP's advantage in later sessions does not come primarily from it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.