- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether mean gradient rank-1 Fisher surrogates faithfully represent gradient second moments. It revisits a linear model justification and shows that a dominant eigenvalue can coexist with substantial Frobenius reconstruction error. Validation combines simulations, held out gradient measurements on diffusion models, and continual learning retention experiments. The results identify limits to the geometric justification while distinguishing approximation quality from the practical benefit of using rank-1 regularization.
Critical strengths
-
The theoretical analysis makes a useful distinction between a dominant eigenvalue and a small relative Frobenius error. The corrected gradient and explicit linear–Gaussian counterexample show why the motivating argument does not establish accurate rank-1 reconstruction in this norm.
-
The empirical study separates fitting and evaluation gradients, addressing the bias from scoring on the fitted sample. The held-out MNIST check still favors rank-1 over diagonal, which provides a useful favorable case alongside the negative findings.
Critical weaknesses
-
I found the theoretical correction is useful, but its broader significance for continual learning is less clear. Showing that rank-1 can be a poor matrix approximation does not establish when this matters for forgetting. I would like a clearer account of how the analysis should change our understanding or choice of continual-learning methods.
-
Both relative Frobenius errors in B.6 are close to one and barely differ. The authors acknowledge that this restricted range prevents a strong conclusion about the fidelity–forgetting relationship, so Table 10 is inconclusive about the practical consequences of the geometry. The relation between its reconstruction score and the trace-matched training penalties also needs to be stated explicitly.
-
Rank-1+GD has higher mean math accuracy than Diagonal+GD in Table 11, but the paired difference changes sign across seeds and rank-1 has higher later-task loss. The authors already share the replay cache and use an early-loss rule to choose a common trace budget. That is a useful control, but it does not match final adaptation quality. A budget sweep could show whether the retention advantage remains at comparable final later-task loss.
Questions for authors
- When should the geometric diagnostic change the choice of a continual learning regularizer?
- Can you sweep the implemented trace budget, keep the existing replay cache fixed, and compare full-test math accuracy at overlapping levels of final held-out later-task loss across seeds?