- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to store training gradients compactly for repeated influence queries. It shows that PCA of half-whitened gradients is optimal for its normalized worst-case error objective, then approximates this with an EK-FAC subspace, PCA correction, and one-bit quantization. GPT-2 experiments validate the method against retraining outcomes; OLMo 2 experiments from 1B to 32B measure agreement with uncompressed EK-FAC rankings. The main result is substantially better attribution quality per byte than the tested compression baselines.
Critical strengths
-
The storage-quality tradeoff is convincing. On GPT-2, EOGP at 96 KB exceeds the tested compression baselines at 1,536 KB in LDS, and the removal experiments support the same trend (Figure 2). This gives the compression gains a meaningful validation beyond matching another estimator.
-
The design is well motivated and the ablations help explain the gains. Figure 5 separates PCA correction from quantization, while Appendices F and G test shared EK-FAC curvature and one-bit baselines. These controls make the contribution more convincing than a storage-only comparison.
Critical weaknesses
-
Large-model validation remains indirect. The OLMo 2 results use 1,000 candidates and 100 queries and measure agreement with EK-FAC, which also supplies EOGP’s curvature approximation (Section 4.2; Appendix D.2). Appendix F is a useful control, but the GPT-2 correlation between fidelity and LDS does not establish that the same relation holds for SFT models. This limits how strongly I interpret the large-model attribution claims.
-
Per-example storage hides a substantial fixed cost. Appendix E.1 reports about 481 GB of PCA correction matrices for the 32B configuration. This is disclosed, but the amortization examples use stores of 100 million or one billion examples, and query timing uses a batch of 100. The practical advantage for smaller stores and individual queries is less clear.
Questions for authors
-
Assessment-relevant validation: Do you have a small removal-and-refinetuning check on OLMo 2 1B comparing EOGP with the strongest baseline? Even a limited check would help determine whether the EK-FAC fidelity gains translate to a measurable change in model behavior. This is a request for additional validation, rather than a clarification of an existing result.
-
Clarification: At the reported 32B configuration, what is the total-storage break-even point against LoGra, including both methods’ shared artifacts? What is the latency for one query, including projection loading and gradient computation? These numbers would clarify when EOGP’s storage advantage is useful in practice.
3 comments
Thank you for the additional experiments. The OLMo 2 1B LDS results directly address my main concern about relying on EK-FAC fidelity. The total-storage comparison and latency breakdown also clarify the practical tradeoff, although 31 seconds per query remains a limitation when matrices are not cached.
Based on the reported additions, my concerns are largely addressed. I maintain my weak acceptance recommendation.
We sincerely thank the reviewer for carefully reading our response and for letting us know that the concerns are largely addressed. The feedback was truly valuable for improving the paper, and we appreciate the time and care put into evaluating it.
We sincerely thank the reviewer for the thoughtful review and for the time devoted to evaluating our paper. We also appreciate that the reviewer highlighted that the storage-quality tradeoff of our method is convincing and that the design is well motivated.
We have uploaded a revised version of the paper. To make the new results easy to find, we collected the new experiments in Appendix H at the end (pages 22 to 25). In a future revision, we will move them to the appropriate places in the paper.
[W1, Q1] Ground-truth LDS evaluation on OLMo 2 1B
On OLMo 2 1B, EOGP achieves higher LDS than the compression baselines at the same per-example storage, and at 28.9 KB and above it achieves higher LDS than K-FAC influence functions. Figure 8 (page 23), which we added to the revision, shows the LDS of each method across storage budgets on pools of 1,000 training examples randomly sampled from MS MARCO [1] and from TyDiQA [2]. Both EOGP and EOGP-R achieve higher LDS than all compression baselines. The difference from the best of these baselines is statistically significant from 7.4 KB for EOGP on MS MARCO and for both EOGP and EOGP-R on TyDiQA.
[W2, Q2] Total-storage break-even against LoGra at 32B and single-query latency
Total-storage break-even point. The total storage of EOGP and LoGra becomes equal at about 283k documents when both methods compress gradients to the same dimension. For a clearer comparison, in Figure 9 (page 23) of the updated paper we further measured EK-FAC fidelity against total storage at the size of the OLMo 2 SFT data. EOGP consistently outperforms the baselines in total storage as well. In this view EOGP-R is often better than EOGP, which shows that EOGP-R serves well as an alternative when the storage for the shared correction matrices is not available.
Single-query latency. A single query takes 31.1 seconds on OLMo 2 32B with k = 2,048 on one B200 GPU. This time includes both loading and computation, and the table below shows each component. Most of the time is spent loading the correction matrices from disk into RAM, so once the matrices are held in RAM most of this cost disappears. We added this table as Table 7 on page 25.
[1] Bajaj, Payal, et al. "Ms marco: A human generated machine reading comprehension dataset." arXiv preprint 2016.
[2] Clark, Jonathan H., et al. "Tydiqa: A benchmark for information-seeking question answering in typologically diverse languages." TACL 2020.