- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper theoretically investigates how data selection affects learning and forgetting in continual learning. Using a linear teacher-student model, the authors analyze two scoring rules, feature novelty and residual error, under different levels of task similarity. The results suggest that feature novelty consistently improves knowledge retention, whereas the effectiveness of residual-error-based selection depends on the relationship between feature and readout similarities. The authors further extend their analysis to hard selection and provide empirical evaluations on CIFAR-100, CUB-200, and AwA2.
Critical strengths
The paper provides a theoretical analysis that connects data selection strategies with task similarity in continual learning.
The finding that residual-error-based selection can either improve or worsen knowledge retention depending on task similarity is interesting.
Critical weaknesses
1. Unclear theoretical novelty over prior work. Previous studies, such as Lee et al. [1] and Hiratani [2], have already investigated the effects of feature and readout similarities on knowledge transfer and forgetting. Furthermore, the benefit of preserving directions associated with previous tasks has been explored through gradient projection methods [3, 4]. Although this paper extends these insights to data selection, it remains unclear whether the main theoretical findings represent fundamentally new insights or relatively straightforward extensions of existing results. The authors should clarify the theoretical novelty of their work.
2. Limited scope of data selection strategies. The analysis focuses exclusively on feature novelty and residual error, although various other data selection strategies have been studied in the literature. It remains unclear whether the theoretical conclusions extend to other scoring rules, such as gradient-based or diversity-based selection. A broader discussion of the applicability of the proposed analysis would strengthen the paper.
3. Restrictive theoretical assumptions. The theoretical analysis assumes fixed feature representations, an infinite-sample limit, and a strongly overparameterized linear model. However, feature representations typically evolve during continual learning, which can substantially affect forgetting. It is unclear whether the main theoretical conclusions, particularly the consistent advantage of feature novelty, remain valid under such conditions. The authors should clarify the implications of these assumptions for practical deep continual learning.
[1] Lee, S., Goldt, S., and Saxe, A. (2021). "Continual Learning in the Teacher-Student Setup: Impact of Task Similarity." International Conference on Machine Learning (ICML).
[2] Hiratani, N. (2024). "Disentangling and Mitigating the Impact of Task Similarity for Continual Learning." Advances in Neural Information Processing Systems (NeurIPS).
[3] Farajtabar, M., Azizan, N., Mott, A., and Li, A. (2020). "Orthogonal Gradient Descent for Continual Learning." International Conference on Artificial Intelligence and Statistics (AISTATS).
[4] Saha, G., Garg, I., and Roy, K. (2021). "Gradient Projection Memory for Continual Learning." International Conference on Learning Representations (ICLR).
Questions for authors
Please refer to the weaknesses.
3 comments
Response to Reviewer 3 (Part 2/2)
W3. Assumptions and evolving representations.
We agree that feature evolution can change forgetting and the effect of selection in end to end deep learning. Nevertheless, our numerical experiments, including newly added end-to-end CIFAR-100 training, qualitatively recover the predicted retention/forgetting behaviors from fixed-feature linear setting.
Appendix A.7 uses the empirical second moment to examine finite-sample effects, and the real-data experiments evaluate selection with finite datasets. Appendix D.5 trains a two-layer MLP sequentially on frozen pretrained features and recovers the predicted score ordering with comparable task-2 accuracy.
We have also added end-to-end CIFAR-100 training of both the backbone and readout (Appendix D.6, Fig. 13). The retention ordering persists, with similar task-2 accuracy at the 70% budget; at 30%, novelty's retention advantage comes with lower task-2 accuracy. Diagnostics confirm that both the representation and readout change, with novelty showing the smallest mean representation drift.
The completed five-task end-to-end CIFAR-100 experiment also recovers the simulation ordering: novelty random residual for both task-1 retention and task-average retention, with a small cost in task-5 accuracy. These results are reported in Appendix D.6, Fig. 14.
These experiments support the qualitative predictions under evolving representations. Analytically, extending the framework would require tracking how selection changes both the representation and subsequent updates; this remains beyond the present derivation.
We hope these clarifications and additional experiments address your concerns and strengthen the evidence for our main findings. If so, we would be grateful if you would reflect these improvements in your updated assessment and rating. Thank you again for your constructive feedback.
Rebuttal Follow-up
Dear Reviewer,
We wanted to kindly follow up on our rebuttal and the revised manuscript. We would be grateful to hear whether our revisions address your concerns, or if there are any remaining questions we could clarify before the discussion period ends.
Thank you again very much for your time and consideration!
Response to Reviewer 3 (Part 1/2)
We thank the reviewer for highlighting the similarity-dependent effects of residual scoring. Our main theoretical contribution is to explain how sample scoring shifts learning between shared and task-specific directions, and when this shift improves or worsens the stability-plasticity trade-off in continual learning. In the revision, we show how the framework applies to other scoring rules. We also find that the predicted retention ordering persists as representations evolve during end-to-end CIFAR-100 training. Revisions are highlighted in blue in the updated manuscript.
W1. Theoretical contribution and relation to prior work.
Lee et al. (2021) and Hiratani (2024) establish how feature and readout similarity affect transfer and forgetting. We build on these insights to address a further question: for a given task relationship, how does prioritizing particular current-task samples change retention at the same current-task loss.
Our model distinguishes shared and task-specific latent directions, so samples differ in their contributions to learning along directions that interfere with the previous task. We derive how a scoring rule changes the latent second moment and, through it, the learning gains in these two blocks. This yields a criterion for whether reweighting improves or worsens retention relative to uniform sampling at matched task-2 loss (Section 4, Appendix A.6).
In particular, within the stated approximation regime, residual scoring changes from harmful to beneficial at ρb=ρa/2, where ρa and ρb describe feature and readout similarity, respectively. Thus the analysis predicts when emphasizing high-error examples worsens the learning-retention trade-off. We also connect proportional reweighting and hard selection through their induced second moments, deriving their common retention ordering and different effect sizes under the stated approximations (Section 5).
The connection to gradient projection is useful but does not establish this result by itself. A high-novelty sample can still have components in the past-task input subspace, so selecting it does not eliminate interfering updates. Following the reviewer's suggestion, we added Appendix B.4 to make the distinction explicit: projection constrains update directions, whereas novelty scoring changes which samples are emphasized. In our linear setting, at the start of task-2 training, the squared Frobenius norm of the projected per-sample gradient equals the residual score multiplied by the novelty score. Novelty is therefore the input-space factor determining how large a gradient remains after projection.
W2. Other scoring rules and applicability of the framework.
Following the reviewer's suggestion, we investigated two additional scoring rules, predictive uncertainty and squared-gradient scoring, in newly added Appendix B, “Connections to other scoring rules and gradient projection,”.
Novelty and residual error are representative scores for which we derive explicit retention predictions. The second-moment formulation applies more broadly: for a fixed nonnegative score σ(s), where s is the latent sample, proportional reweighting induces
C=E[σ(s)]E[σ(s)ss⊤].
Substituting this matrix into the update equation gives the trained weights and both task losses. A new score therefore requires characterizing its induced C; it need not have the same retention behavior as novelty or residual scoring. In particular, the squared-gradient score equals the residual score multiplied by the squared input norm. We use this as a representative gradient-based score, rather than implementing full gradient-based algorithms such as Gradient-based Sample Selection (GSS) or Online Coreset Selection (OCS), which require additional machinery such as replay buffers (Aljundi et al., 2019; Yoon et al., 2021). We clarified this broader applicability in the main text (lines 255–256).
Diversity-based selection generally depends on the selected set rather than on a fixed score for each example. Within our linear squared-loss setting, its effect can also be studied through the selected data's second moment C. Deriving C and retention predictions for these methods remains future work.