- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a method for training a model continually, using constrained optimization to prevent parameter updates from interfering with what was learned on past tasks. Instead of relying only on diagonal curvature, the method uses the Hessian matrix to capture interactions between parameters. A main contribution of the paper is an efficient way to maintain and update a rank-k memory of the average Hessian.
Critical strengths
- An efficient way to maintain and update a rank-k memory of the average Hessian. The Lanczos algorithm estimates each task’s dominant curvature directions from Hessian–vector products, and a factored update merges them into the memory at O(nk) cost.
Critical weaknesses
- The experimental evaluation is limited in both baselines and datasets. The ablation on the rank k is limited, and there is no ablation on the regularization coefficient λ. Reporting the actual training cost would also be informative, in particular the wall-clock time of the Lanczos estimation.
Questions for authors
None
Thank you author for your invaluable feedback. We have taken your concerns into account and have updated the paper, with more robust experiments and have modified our claims accordingly. Thank you.