acceptodds
Under review as a conference paper at ICLR 2027

Rank-1 Fisher for Continual Learning: When Is Its Geometry Faithful

Abstract

Mean-gradient rank-1 empirical-Fisher surrogates offer a memory-efficient approach to continual learning, but their practical utility does not establish that one retained direction faithfully represents loss-gradient second moments on unseen data. We examine this geometric interpretation theoretically and empirically. The motivating linear-model proof omits an input outer product: the gradient is proportional to , rather than . Under the corresponding linear–Gaussian model, we derive the exact population second-moment operator and show that, although vanishes with dimension for a scaled-identity residual, the optimal rank-1 relative Frobenius error approaches . Simulations further show that same-sample evaluation can spuriously favor rank-1 at low , whereas this advantage disappears on disjoint test samples. Held-out evaluations on three frozen masked diffusion language models spanning 219M to 8B parameters reveal no uniform rank-1 advantage: at the largest calibration size, rank-1 slightly outperforms diagonal on SMDM-219M, but underperforms on SMDM-1.14B and in 14 of 15 LLaDA-8B settings. An exact error decomposition gives a diagnostic answer: held-out faithfulness jointly requires rank-1 capacity, transfer of the mean-gradient direction, and transfer of its fitted scale. A source-setting check on small MNIST UNets provides a favorable local boundary, but the overall results vary across objectives, scales, layers, and corruption levels. Mean-gradient rank-1 Fisher therefore need not provide a faithful held-out reconstruction, although this does not rule out its consolidation utility. Code is available.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, catastrophic forgetting, diffusion theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oyi5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether mean gradient rank-1 Fisher surrogates faithfully represent gradient second moments. It revisits a linear model justification and shows that a dominant eigenvalue can coexist with substantial Frobenius reconstruction error. Validation combines simulations, held out gradient measurements on diffusion models, and continual learning retention experiments. The results identify limits to the geometric justification while distinguishing approximation quality from the practical benefit of using rank-1 regularization.

Critical strengths
  1. The theoretical analysis makes a useful distinction between a dominant eigenvalue and a small relative Frobenius error. The corrected gradient and explicit linear–Gaussian counterexample show why the motivating argument does not establish accurate rank-1 reconstruction in this norm.

  2. The empirical study separates fitting and evaluation gradients, addressing the bias from scoring on the fitted sample. The held-out MNIST check still favors rank-1 over diagonal, which provides a useful favorable case alongside the negative findings.

Critical weaknesses
  1. I found the theoretical correction is useful, but its broader significance for continual learning is less clear. Showing that rank-1 can be a poor matrix approximation does not establish when this matters for forgetting. I would like a clearer account of how the analysis should change our understanding or choice of continual-learning methods.

  2. Both relative Frobenius errors in B.6 are close to one and barely differ. The authors acknowledge that this restricted range prevents a strong conclusion about the fidelity–forgetting relationship, so Table 10 is inconclusive about the practical consequences of the geometry. The relation between its reconstruction score and the trace-matched training penalties also needs to be stated explicitly.

  3. Rank-1+GD has higher mean math accuracy than Diagonal+GD in Table 11, but the paired difference changes sign across seeds and rank-1 has higher later-task loss. The authors already share the replay cache and use an early-loss rule to choose a common trace budget. That is a useful control, but it does not match final adaptation quality. A budget sweep could show whether the retention advantage remains at comparable final later-task loss.

Questions for authors
  1. When should the geometric diagnostic change the choice of a continual learning regularizer?
  2. Can you sweep the implemented trace budget, keep the existing replay cache fixed, and compare full-test math accuracy at overlapping levels of final held-out later-task loss across seeds?
Reviewer_YufeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Wang et al. (ICLR 2026) propose rank-1 EWC based on the idea that gradients in diffusion models become nearly aligned at high noise levels. This paper revisits this claim and identifies a missing term in the original derivation. The authors show that a large eigengap does not necessarily mean rank-1 is a good approximation. They also point out that evaluating on the same samples used for fitting can incorrectly favor rank-1. Experiments on three masked diffusion language models show mixed results, with rank-1 not consistently outperforming the diagonal approximation.

Critical strengths
  1. The mathematical correction is valid and clearly explained. The counterexample shows that a large gap between the two largest eigenvalues does not necessarily mean rank-1 is a good approximation, since the smaller eigenvalues can still contribute substantially when taken together.

  2. The paper identifies an important issue with evaluation and is transparent about its experiments. When the number of samples is small relative to the number of parameters, evaluating on the same samples used for fitting can incorrectly favor rank-1. The authors clearly demonstrate this with a Gaussian noise simulation and use held-out samples in their subsequent evaluations. They also report outliers and explain which experimental runs were invalidated.

Critical weaknesses
  1. The main experiments do not sufficiently cover the setting studied by Wang et al. This paper only tests the smallest image setting, MNIST, where rank-1 has lower held-out approximation error in 26 out of 27 cases. Most of the negative results come from diffusion language models, which the authors themselves acknowledge do not satisfy the assumptions used by Wang et al. The experiments therefore mainly show that the explanation does not directly generalize to diffusion language models, but do not establish how much the theoretical issue matters in the original image diffusion setting.

  2. The Fisher approximation metric is not clearly connected to actual forgetting. The paper mainly uses relative Frobenius error to evaluate the Fisher approximation, but it is unclear how well this metric reflects forgetting in continual learning. The held-out Fisher is estimated from relatively few samples, and in some settings, both approximations have errors close to 1. However, rank-1 has lower error along actual parameter updates such as 0.761 vs. 0.997. Since these experiments only use three seeds, more runs would help determine whether this trend is consistent.

Questions for authors
  1. Could the authors run the same held-out evaluation on larger image diffusion settings such as CIFAR-10 or ImageNet, especially at high noise levels? If rank-1 also performs poorly there, the critique would be more convincing.

  2. Tables 9 and 11 show trends favoring rank-1 in forgetting or knowledge retention. Could the authors run more seeds to check whether these differences are statistically significant?

  3. In Appendix B.4 and B.5, the authors reuse experimental artifacts from another concurrent ICLR 2027 submission (Anonymous Authors, 2027). Could the authors clarify how the two papers are related and whether there is any overlap in their experimental contributions?

Reviewer_QM3hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits the geometric justification for the mean-gradient rank-1 empirical-Fisher surrogate used in rank-1 EWC for continual diffusion models (Wang et al., 2026). It corrects an omitted xxTxx^{T} factor in the source linear-denoiser gradient and shows that in the resulting linear–Gaussian model, λ2/λ1=2/(d+2)→0\lambda_2/\lambda_1 = 2/(d+2)\to 0 yet the optimal rank-1 relative Frobenius error approaches 2/3\sqrt{2/3}. It then shows that same-sample evaluation spuriously favors rank-1 at low n/dn/d, compares rank-1 and diagonal surrogates on disjoint gradient splits from three masked diffusion language models (SMDM-219M, SMDM-1.14B, LLaDA-8B) via a scale–direction–capacity error decomposition, and adds controls on MNIST UNets, dense slices, update fidelity, and retention.

Critical strengths
  1. The linear–Gaussian correction cleanly separates spectral dominance from matrix reconstruction: even when the mean gradient aligns with the top eigendirection and λ2/λ1=2/(d+2)→0\lambda_2/\lambda_1 = 2/(d+2)\to 0, the optimal rank-1 relative Frobenius error still approaches 2/3\sqrt{2/3}.

  2. The split-sample protocol and isotropic null identify a genuine evaluation bias, and reporting outcomes where rank-1 still wins (26/27 MNIST cells and all three update-weighted seeds) alongside audited artifacts keeps the empirical study balanced.

Critical weaknesses
  1. The paper's answer to "when" is mostly post hoc. Beyond the linear–Gaussian counterexample, the analysis relies on Eq. (14), which requires the test Fisher FtestF_{test} to compute both the optimal rescaling atest∗a^*_{\mathrm{test}} and the oracle rank-1 error. It therefore diagnoses failure after evaluation rather than providing a criterion checkable at calibration time, and concluding that faithfulness requires capacity, direction transfer, and scale transfer largely restates the reconstruction objective.

  2. The link between Frobenius fidelity and continual learning is weak. The main benchmark covers one checkpoint per model and mostly normalization slices with mixed results (rank-1 wins 6/15, 3/15, and 1/15 cells in Table 2 vs. 26/27 on MNIST). More importantly, the paper's own controls decouple reconstruction error from downstream performance: in Appendix B.6 both surrogates have error ≈1\approx 1 across all cells and fidelity is uncorrelated with forgetting (ρ=−0.046\rho=-0.046), yet Rank-1+GD forgets less in 14/18 cells and slightly outperforms Diagonal+GD on GSM8K (52.31% vs. 49.23%). Moving between Frobenius error, update error, and retention without a unified takeaway makes the practical conclusion unclear.

Questions for authors
  1. Appendix A mentions eRtope_{R_{\mathrm{top}}}, the error of a rank-1 fit to the leading eigenpair of F^cal\hat{F}_{\mathrm{cal}}. How does its held-out error compare with RμR_\mu and the diagonal surrogate—is the bottleneck the mean-gradient estimator or rank-1 calibration fitting itself?

  2. Does any calibration-only statistic (e.g., split-half stability of μ\mu or scale) predict whether rank-1 beats diagonal on F^test\hat{F}_{\mathrm{test}}, and does Frobenius fidelity correlate with forgetting on slices where the two reconstruction errors actually differ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.