- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper examines how accurately a cached curvature matrix can approximate a nearby matrix when estimating the norm of a curvature-preconditioned vector. Since this quantity involves inverse squares, non-commutativity plays an important role.
Under a spectral locality assumption, Theorem 4.1 establishes a tight bound independent of the matrix dimension and condition number, and characterizes when this bound remains finite. Theorem 5.1 further derives an exact worst-case bound for unrestricted spectral coupling, while a corollary extends the analysis to approximate locality.
The authors evaluate the proposed bounds through a screening procedure for logistic regression using synthetic data, random Fourier features, and frozen MLP and Pythia-70M features. Cached curvature estimates provide bounds on , with PCG applied only to cases that cannot be resolved directly. Computational efficiency is assessed using HVP counts and wall-clock time, with comparisons against global screening, Chebyshev-based bounds, and direct Cholesky factorization.
Critical strengths
-
Tight theoretical bounds. The paper establishes matching upper and lower bounds, including an asymptotically tight construction under spectral locality and an exact characterization for unrestricted coupling. I also checked the main formulas numerically and found no apparent errors, which supports the theoretical validity of the results.
-
Clear acknowledgment of limitations. The authors report that direct Cholesky is faster in the evaluated settings, include a relevant Chebyshev-based baseline, and acknowledge that their experiments are limited to fixed-feature models. I appreciate this transparency, although these findings also raise questions about the practical significance of the proposed approach.
Critical weaknesses
-
Limited novelty and empirical improvement over existing bounds. The sharp locality threshold can already be obtained from a Chebyshev expansion, as the authors acknowledge (pp. 2–4). Related decay results are also well established, and Appendix J notes that Janas et al. (2018) recover the sharp scalar-chain norm. The main theoretical contribution therefore appears to be a constant-factor improvement (less than 2×) and a generalization to arbitrary spectral configurations. The unrestricted result similarly improves the constant rather than the asymptotic dependence.
The empirical gains are modest. Compared with the Chebyshev baseline, HVP counts decrease by only 2.1–5.5% (Table 5), with very small differences in runtime (Table 6). The local method is also slower than global screening on dense features (Tables 14–15). The reported 3.1× HVP reduction is relative to a baseline that does not exploit structure, while direct Cholesky remains faster in the evaluated settings. Overall, I find the improvement over existing methods too limited to establish a substantial contribution.
-
Insufficient evidence of practical relevance. Reusing cached curvature information could be useful for methods such as Shampoo/SOAP, K-FAC, or influence-function estimation. However, the experiments are restricted to fixed-feature logistic regression, with dimensions up to 1024 and Pythia-70M as the largest backbone. All experiments run on a single CPU thread in float64 (Appendix K). It remains unclear whether the proposed bounds offer meaningful advantages in realistic large-scale settings.
Moreover, the screening task is defined specifically for this paper, without demonstrating benefits for actual training or data selection. The experiments therefore do not establish when the method would be preferable to existing approaches.
The presentation also needs improvement. Several appendices contain extensive numerical results with limited interpretation or material whose relevance to the main contribution is unclear. Appendix O retains revision artifacts and indicates that some raw records and plotting scripts needed for reproduction are missing. These issues make the empirical results difficult to assess and reproduce.
Questions for authors
-
Beyond the less than 2× improvement in the constant, are there settings where the new locality bound meaningfully changes screening decisions or reduces runtime compared with the Chebyshev bound? If possible, please quantify this using existing logs, including how often the bound is finite, the magnitude of the weighted spectral tails, and which bound determines the final interval.
-
Could the authors evaluate the proposed method in a more realistic setting where curvature reuse matters, such as delayed Shampoo/SOAP or K-FAC updates, or influence-function estimation with EK-FAC? Comparisons with Chebyshev and a simple periodic-refresh baseline would be helpful. It would also be useful to know whether spectral locality holds empirically in these settings.
-
What practical benefit does the proposed screening task offer over random selection or using all examples?
-
Could the authors provide instructions to reproduce the main tables from a clean environment and clarify which results depend on historical raw data that are no longer available?