- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a projection-based method for continual learning in language models. It uses loss statistics from reference data to estimate the importance of input–output direction pairs in each weight matrix, then removes components along protected directions from optimizer updates when learning new tasks. The paper presents the definition of these directions as a flexible design choice, but this flexibility makes its main contribution less clear. Moreover, it does not directly validate whether the importance scores accurately identify the directions that should be protected, and it provides insufficient detail on key parameter settings and how the protected set changes across tasks.
Critical strengths
1、Finer-grained protection. SCLL selects input–output direction pairs rather than protecting all output directions associated with a given input direction. This provides a reasonable approach to retaining prior capabilities while learning new tasks.
2、Evaluation across different settings. The paper evaluates new-task learning and retention of prior capabilities on the TRACE and Long CL task streams. It also examines performance on related datasets that were not used to construct the protection.
Critical weaknesses
1、Insufficient validation of the importance scores. The paper makes a plausible attempt to address limitations of existing methods, but provides limited empirical evidence for its central mechanism. In particular, it proposes an approximate importance score without directly validating it. It remains unclear whether selecting higher-scoring directions leads to better training outcomes under the same protection budget.
2、Underreported settings and limited analysis of the protected set. Key method parameters are not clearly specified or ablated. For example, the main text does not report the importance-coverage threshold . The appendix indicates that SCLL and the strong baseline NSCL use different values of , but neither choice is accompanied by a corresponding ablation study. In Long CL, the proportion of directions selected for protection decreases more noticeably, yet the paper does not analyze which matrices or input–output direction pairs enter or leave the protected set.
3、Limited model diversity and scale. The experiments are restricted to Qwen3.5 models in the 0.8B–2B range. It remains unclear whether the findings extend to other model families, such as Llama, or to larger models, such as a 7B-parameter model.
4、Unclear effect of the factor-aggregation approximation. The paper aggregates the and matrices by averaging them across tasks. However, the mean of Kronecker products is generally not equal to the Kronecker product of the means. How large is the resulting estimation error? Can the final importance scores faithfully represent the expected squared change after a parameter update?
Questions for authors
1、How closely does the ranking induced by agree with the actual squared change in reference loss when the corresponding direction pair is perturbed? When the number or proportion of protected directions is held fixed, does selecting pairs by this score yield a better trade-off between learning new tasks and retaining prior capabilities than alternative selection rules?
2、What evidence supports the scalability of SCLL to larger models or other model families? Beyond the asymptotic storage complexity, could the authors report measured runtimes and peak memory usage for reference-statistics collection, eigendecomposition, and projection during training, and explain how these costs scale with model width?
3、How were the values of chosen for SCLL and NSCL, respectively? How sensitive are the results to these choices?