- Rating
- 3
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces an interesting approach of predicting the continual learning capability of a particular model architecture, that is to use a proxy defined as the logit shift of the old task after the new task is trained over. To approximate this proxy, the authors proposed a theoretical framework that factors in several intuitive measures like gradient, the model complexity and task-gradient alignment, followed by an regression to make this form concrete based on real data. A good amount of model architectures and datasets are selected to perform the experiments that verify the effectiveness of the method.
Critical strengths
Overall the proposed method seems to demonstrate its superiority based on the extensive data points in the experiments. The proposed formulation is also easy to implement, despite being a bit adhoc.
Critical weaknesses
I do not find major flaws, but my below questions could be answered before we call the theory in this paper really sound.
Questions for authors
-
Consider a scenario of multi-class classification with softmax prediction, and the logics for all classes shift by the same constant after the new task. In this case, the prediction distribution on the old task doesn't change, but ADS changed likely. Put it more generally, why the logit and its derived signal could be a good surrogate for transferability measurement, apart from the theoretical convenience?
-
ADS afaik is an upperbound of the logit shift. But upperbound doesn't automatically guarantee the monotonity. For example it's possible that A < B, B_i > B_j but A_i < A_j. A further analysis of this, at least a guarantee even if not 100% accurate, will make the theory more justifiable.
2 comments
Looking forward to the response from Reviewer rzwt
Dear Reviewer rzwt,
Thanks for your valuable feedback. As the author–reviewer discussion period will end in approximately three days , we kindly ask you to review our rebuttal and let us know whether your concerns have been addressed.
If you have any remaining questions, we would be happy to provide further clarification.
Best regards, The Authors
Point 1
Point 2
Reference
[1] Etash Kumar Guha and Vihan Lakshman. On the diminishing returns of width for continual learning. In International Conference on Machine Learning, pp. 16706–16730. PMLR, 2024.
[2] Joonkyu Kim, Yejin Kim, and Jy yong Sohn. Measuring representational shifts in continual learning: A linear transformation perspective. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=Kz1zCJRr1r.