acceptodds
Under review as a conference paper at ICLR 2027

Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection

14 views

Abstract

Influence functions estimate how individual training examples affect the behavior of large language models (LLMs). Analyzing how training data influence different behaviors of an LLM involves repeated influence computation. Reusing stored training gradients reduces the computational cost, but storing full gradients is prohibitively expensive at LLM scale. We study how to compress these gradients while preserving influence estimates for future queries that are unknown at storage time. Through a worst-case analysis, we characterize the optimal fixed-dimensional linear representation and propose eigenbasis-corrected one-bit gradient projection (EOGP) to approximate it at scale. Specifically, EOGP uses EK-FAC to reduce gradient dimensionality, then applies PCA within the retained subspace to learn compression directions from the training gradients. We then apply one-bit quantization to the resulting coordinates, allowing more coordinates to be retained within a fixed storage budget. On GPT-2, EOGP predicts retraining outcomes more accurately than the evaluated compression baselines while using one-sixteenth of their per-example storage. On OLMo 2 SFT models from 1B to 32B parameters, EOGP remains competitive with the baselines allocated over 100 times as much storage per example. Code is available at https://anonymous.4open.science/r/EOGP_anonymous-70FA.

Primary area:
interpretability and explainable AI
Keywords:
influence functions, training data attribution, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5tokOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to store training gradients compactly for repeated influence queries. It shows that PCA of half-whitened gradients is optimal for its normalized worst-case error objective, then approximates this with an EK-FAC subspace, PCA correction, and one-bit quantization. GPT-2 experiments validate the method against retraining outcomes; OLMo 2 experiments from 1B to 32B measure agreement with uncompressed EK-FAC rankings. The main result is substantially better attribution quality per byte than the tested compression baselines.

Critical strengths
  1. The storage-quality tradeoff is convincing. On GPT-2, EOGP at 96 KB exceeds the tested compression baselines at 1,536 KB in LDS, and the removal experiments support the same trend (Figure 2). This gives the compression gains a meaningful validation beyond matching another estimator.

  2. The design is well motivated and the ablations help explain the gains. Figure 5 separates PCA correction from quantization, while Appendices F and G test shared EK-FAC curvature and one-bit baselines. These controls make the contribution more convincing than a storage-only comparison.

Critical weaknesses
  1. Large-model validation remains indirect. The OLMo 2 results use 1,000 candidates and 100 queries and measure agreement with EK-FAC, which also supplies EOGP’s curvature approximation (Section 4.2; Appendix D.2). Appendix F is a useful control, but the GPT-2 correlation between fidelity and LDS does not establish that the same relation holds for SFT models. This limits how strongly I interpret the large-model attribution claims.

  2. Per-example storage hides a substantial fixed cost. Appendix E.1 reports about 481 GB of PCA correction matrices for the 32B configuration. This is disclosed, but the amortization examples use stores of 100 million or one billion examples, and query timing uses a batch of 100. The practical advantage for smaller stores and individual queries is less clear.

Questions for authors
  1. Assessment-relevant validation: Do you have a small removal-and-refinetuning check on OLMo 2 1B comparing EOGP with the strongest baseline? Even a limited check would help determine whether the EK-FAC fidelity gains translate to a measurable change in model behavior. This is a request for additional validation, rather than a clarification of an existing result.

  2. Clarification: At the reported 32B configuration, what is the total-storage break-even point against LoGra, including both methods’ shared artifacts? What is the latency for one query, including projection loading and gradient computation? These numbers would clarify when EOGP’s storage advantage is useful in practice.

3 comments

We sincerely thank the reviewer for the thoughtful review and for the time devoted to evaluating our paper. We also appreciate that the reviewer highlighted that the storage-quality tradeoff of our method is convincing and that the design is well motivated.

We have uploaded a revised version of the paper. To make the new results easy to find, we collected the new experiments in Appendix H at the end (pages 22 to 25). In a future revision, we will move them to the appropriate places in the paper.

[W1, Q1] Ground-truth LDS evaluation on OLMo 2 1B

On OLMo 2 1B, EOGP achieves higher LDS than the compression baselines at the same per-example storage, and at 28.9 KB and above it achieves higher LDS than K-FAC influence functions. Figure 8 (page 23), which we added to the revision, shows the LDS of each method across storage budgets on pools of 1,000 training examples randomly sampled from MS MARCO [1] and from TyDiQA [2]. Both EOGP and EOGP-R achieve higher LDS than all compression baselines. The difference from the best of these baselines is statistically significant from 7.4 KB for EOGP on MS MARCO and for both EOGP and EOGP-R on TyDiQA.

[W2, Q2] Total-storage break-even against LoGra at 32B and single-query latency

  • Total-storage break-even point. The total storage of EOGP and LoGra becomes equal at about 283k documents when both methods compress gradients to the same dimension. For a clearer comparison, in Figure 9 (page 23) of the updated paper we further measured EK-FAC fidelity against total storage at the size of the OLMo 2 SFT data. EOGP consistently outperforms the baselines in total storage as well. In this view EOGP-R is often better than EOGP, which shows that EOGP-R serves well as an alternative when the storage for the shared correction matrices is not available.

  • Single-query latency. A single query takes 31.1 seconds on OLMo 2 32B with k = 2,048 on one B200 GPU. This time includes both loading and computation, and the table below shows each component. Most of the time is spent loading the correction matrices from disk into RAM, so once the matrices are held in RAM most of this cost disappears. We added this table as Table 7 on page 25.

ComponentTime (s)
Loading, disk to RAM22.07
Loading, RAM to GPU8.69
Query gradient0.17
Eigenbasis projection0.09
Correction matrices and scoring0.11
Total31.13

[1] Bajaj, Payal, et al. "Ms marco: A human generated machine reading comprehension dataset." arXiv preprint 2016.
[2] Clark, Jonathan H., et al. "Tydiqa: A benchmark for information-seeking question answering in typologically diverse languages." TACL 2020.

Thank you for the additional experiments. The OLMo 2 1B LDS results directly address my main concern about relying on EK-FAC fidelity. The total-storage comparison and latency breakdown also clarify the practical tradeoff, although 31 seconds per query remains a limitation when matrices are not cached.

Based on the reported additions, my concerns are largely addressed. I maintain my weak acceptance recommendation.

We sincerely thank the reviewer for carefully reading our response and for letting us know that the concerns are largely addressed. The feedback was truly valuable for improving the paper, and we appreciate the time and care put into evaluating it.

Reviewer_F9jgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EOGP, a technique of compressing gradients used to estimate influence functions. It combines one-bit gradient projection with an eigenbasis correction meant to preserve information relevant to influence estimates to reduce storage and computational costs. EOGP-R is also introduced for improved compression-quality tradeoffs.

The evaluation assesses gradient reconstruction quality and influence estimation with comparisons to existing compression approaches. The results suggest that this approach can reduce the costs of storing and processing gradients while preserving influential attribution quality. However, the evidence in favor of scaling to large language models is indirect compared to smaller-scale evaluation: agreement with EK-FAC is presented as a proxy in situations where validation by retraining would be expensive.

Critical strengths
  1. Evaluation beyond compression ratios. The paper considers the effect of compression on influence-estimation quality rather than focusing only on storage savings, judging compression by what it does to attribution quality, not by storage or reconstruction error alone. This makes the proposed approach more practically interesting, assuming the limitations of the large-model validation are acknowledged. On retraining-based LDS with bootstrap confidence intervals, EOGP at 96 KB per example beats every compression baseline given 1,536 KB. It also leads on counterfactual removal at matched budgets, averaged over five retraining seeds (Fig. 2; p.17). This is the only ground-truth evidence in the paper, and it is the main reason I do not rate it lower.
  2. The problem is well posed and practically important. An uncompressed gradient of the 32B model takes 62.4 GB per example, against 115.6 KB for EOGP (p.9). Framing the goal as the worst-case error in influence scores for unknown queries, rather than as gradient reconstruction, gives a clear design rule (whiten, then compress) and explains why curvature should enter before compression. I found this formulation the most reusable part of the paper.
Critical weaknesses
  1. The scaling claim rests on a proxy that is close to circular, and the method is never tested on a downstream task. Retraining-based evaluation is done only on GPT-2 small (Radford et al., 2019), a much older and smaller model than the ones the paper targets, and all LLM-scale results come from a single model family, OLMo 2 (OLMo et al., 2024). At that scale the paper scores methods by agreement with EK-FAC influence (George et al., 2018; Grosse et al., 2023), because retraining "is computationally prohibitive at this scale" (p.7). That is understandable, but EK-FAC uses the same eigenbasis, eigenvalues and damping that EOGP truncates (Eq. 31), so the metric may simply reward closeness to the method's own reference. The evidence that the proxy tracks attribution quality is a correlation across 28 configurations on GPT-2 (Fig. 3, p.18), plus App. F, where the EK-FAC-whitened baselines actually score below their original versions (Fig. 6). Neither tests the proxy at LLM scale, and Fig. 4 has no error bars, with only 1,000 candidates and 100 queries (p.18).

    The method is also never used for the applications that motivate it. The introduction names selecting data for fine-tuning, and finding training examples behind undesired behavior, as the main uses of influence functions (p.1). The related work discusses LESS (Xia et al., 2024) and QLESS (Ananta et al., 2025), which select instruction data with projected gradients (p.9). Yet, apart from the GPT-2 retraining used to score attribution (LDS and counterfactual removal), no model is trained on data chosen with EOGP, and no downstream task performance is reported at any scale. The conclusion nevertheless states that EOGP makes "reusable gradient stores more practical at LLM scale" (p.9). Since the title and abstract promise scaling to LLMs, the lack of both ground-truth and downstream evidence at that scale is the point on which my rating mainly rests.

  2. The theory-motivated PCA step is not shown to help on ground truth, and it is expensive. On GPT-2, EOGP stored in FP16 is close to the baselines at 12–96 KB (Fig. 7), so most of the low-budget gain there comes from 1-bit storage fitting 16×16\times more coordinates; no PCA ablation is reported with LDS. The PCA step does help on OLMo 2 (Fig. 5), but only under the EK-FAC agreement metric. On 32B, the PCA correction matrices take 481 GB of shared storage (App. E.1), equal to the per-example payload of about 4.2 million examples, while EOGP-R, which needs no such matrices, has higher Spearman correlation than EOGP at almost every budget (Fig. 4). The theory also adds less than the framing suggests: Prop. 1 is the Eckart–Young theorem (Eckart & Young, 1936) after a change of variables, as the authors note (p.13), and neither proposition covers the EK-FAC approximation or 1-bit storage. Because the PCA-in-eigenbasis construction is the claimed novelty, I would like to see its benefit on a ground-truth metric.

References:

  • Ananta, M., Adilazuarda, M. F., Zuhri, Z. M. K., Purwarianti, A., and Aji, A. F. QLESS: A quantized approach for data valuation and selection in large language model fine-tuning. arXiv:2502.01703, 2025.
  • Eckart, C. and Young, G. The approximation of one matrix by another of lower rank. Psychometrika, 1936.
  • George, T., Laurent, C., Bouthillier, X., Ballas, N., and Vincent, P. Fast approximate natural gradient descent in a Kronecker-factored eigenbasis. NeurIPS, 2018.
  • Grosse, R., et al. Studying large language model generalization with influence functions. arXiv:2308.03296, 2023.
  • OLMo, T., et al. 2 OLMo 2 Furious. arXiv:2501.00656, 2024.
  • Radford, A., et al. Language models are unsupervised multitask learners. OpenAI blog, 2019.
  • Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. LESS: Selecting influential data for targeted instruction tuning. ICML, 2024.
Questions for authors
  1. Requested experiment: could you report LDS on a small set of retraining runs for OLMo 2 1B (OLMo et al., 2024) or a model from another family? Even a small experiment of this kind would show whether the OLMo results reflect real attribution quality.
  2. Requested experiment: could you run one downstream application with the compressed stores? For example, select fine-tuning data on OLMo 2 1B in the LESS setup (Xia et al., 2024), fine-tune on the selected subset, and report target-task performance for EOGP, the compression baselines and the uncompressed reference at matched storage. This would show whether better agreement with EK-FAC (George et al., 2018; Grosse et al., 2023) actually leads to better selected data. Together with the previous question, this is the result most likely to change my score.
  3. Clarification: shared artifacts are counted separately from the per-example budget (Sec. 4.4, App. E.1). How much shared storage do LoGra (Choe et al., 2025), GraSS (Hu et al., 2025), LoRIF (Li et al., 2026) and EOGP-R need at the same budgets, and how do the methods compare at equal total storage (Eq. 40) for a store the size of the SFT dataset?
  4. Clarification: the PCA fitting set and the per-module kk and mm are given only for OLMo 2 (App. C.2). What are they for GPT-2 (Radford et al., 2019)? Since the Gram-matrix PCA yields at most ∣C∣|C| directions (App. B.2), how are directions chosen when k>∣C∣k > |C|?
  5. Clarification: on 32B, the Spearman correlation of LoGra and GraSS falls as storage grows (Fig. 4), and on GPT-2 their 1-bit LDS declines and even turns negative (Fig. 7). Was LoGra's fixed damping of 0.1 (p.17) tuned per budget, and what damping do GraSS and LoRIF use? Under-tuned baselines would make the gaps look larger than they are.
  6. Requested experiment: how do FP16 truncated EK-FAC, TrackStar (Chang et al., 2025) and Arnoldi-style projection (Schioppa et al., 2022) compare at matched budgets?

References:

  • Chang, T., Rajagopal, D., Bolukbasi, T., Dixon, L., and Tenney, I. Scalable influence and fact tracing for large language model pretraining. ICLR, 2025.
  • Choe, S., et al. What is your data worth to GPT? LLM-scale data valuation with influence functions. NeurIPS, 2025.
  • George, T., Laurent, C., Bouthillier, X., Ballas, N., and Vincent, P. Fast approximate natural gradient descent in a Kronecker-factored eigenbasis. NeurIPS, 2018.
  • Grosse, R., et al. Studying large language model generalization with influence functions. arXiv:2308.03296, 2023.
  • Hu, P., Melkonian, J., Tang, W., Zhao, H., and Ma, J. GraSS: Scalable data attribution with gradient sparsification and sparse projection. NeurIPS, 2025.
  • Li, S., Le, H., Xu, J., and Salzmann, M. LoRIF: Low-rank influence functions for scalable training data attribution. arXiv:2601.21929, 2026.
  • OLMo, T., et al. 2 OLMo 2 Furious. arXiv:2501.00656, 2024.
  • Radford, A., et al. Language models are unsupervised multitask learners. OpenAI blog, 2019.
  • Schioppa, A., Zablotskaia, P., Vilar, D., and Sokolov, A. Scaling up influence functions. AAAI, 2022.
  • Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. LESS: Selecting influential data for targeted instruction tuning. ICML, 2024.
4 comments

We sincerely thank the reviewer for the thoughtful review and for the time devoted to evaluating our paper. We also appreciate that the reviewer highlighted that our problem is well posed and practically important, and that our formulation gives a clear design rule for compression.

We have uploaded a revised version of the paper. To make the new results easy to find, we collected the new experiments in Appendix H at the end (pages 22 to 25). In a future revision, we will move them to the appropriate places in the paper.

[W1, W2, Q1] Ground-truth LDS evaluation on OLMo 2 1B and benefit of the PCA step

On OLMo 2 1B, EOGP achieves higher LDS than the compression baselines at the same per-example storage, and at 28.9 KB and above it achieves higher LDS than K-FAC influence functions. Figure 8 (page 23), which we added to the revision, shows the LDS of each method across storage budgets on pools of 1,000 training examples randomly sampled from MS MARCO [1] and from TyDiQA [2].

Both EOGP and EOGP-R achieve higher LDS than all compression baselines. The difference from the best of these baselines is statistically significant from 7.4 KB for EOGP on MS MARCO and for both EOGP and EOGP-R on TyDiQA. EOGP also achieves higher LDS than EOGP-R, which replaces the PCA step with a random projection, at every storage budget on MS MARCO and at 7.4 KB and above on TyDiQA.

[W1, Q2] LESS-style data selection on OLMo 2 1B at matched storage

EOGP achieves the highest target-task performance among the compression methods on four of the five tasks in LESS-style data selection on OLMo 2 1B.

We followed the data pool, target tasks, and LoRA recipe of LESS and added ARC-Easy and ARC-Challenge as two further tasks. Each method scores 20,000 examples sampled from the LESS pool, and we fine-tune on the top 1% with three seeds. All compression methods use 14.6 KB per example, and EK-FAC is the uncompressed reference. The following table shows the results, and we added it as Table 5 on page 24.

MethodMMLU [3]TyDiQA [2]BBH [4]ARC-Easy [5]ARC-Challenge [5]
Base42.1911.7729.5473.5042.32
Full pool42.8428.0728.9875.5544.71
Random41.91 ± 0.1424.08 ± 0.5329.23 ± 0.8174.10 ± 0.2042.61 ± 0.13
EK-FAC (no compression)41.96 ± 0.2029.83 ± 0.3830.15 ± 0.3775.73 ± 0.0343.71 ± 0.20
Gradient-compression methods
LoGra (PCA)41.06 ± 0.1526.21 ± 0.1928.86 ± 0.4274.97 ± 0.1342.72 ± 0.27
LoGra (Random)41.56 ± 0.0626.91 ± 0.6829.69 ± 0.5374.67 ± 0.0643.17 ± 0.23
GraSS41.74 ± 0.0625.06 ± 0.2729.51 ± 0.4274.85 ± 0.2242.63 ± 0.05
LoRIF41.59 ± 0.0327.19 ± 0.6131.45 ± 0.2375.08 ± 0.0343.20 ± 0.20
EOGP42.09 ± 0.0831.94 ± 0.1729.66 ± 0.3975.27 ± 0.1243.94 ± 0.26

On MMLU and TyDiQA, EOGP is significantly higher than each of the four compression baselines (p < 0.05). In addition, fine-tuning on the 1% selected with EOGP achieves performance comparable to or higher than fine-tuning on the full pool. Given these results, we believe that EOGP can be used effectively for data selection, where the stored gradients of the pool can be reused to select data for many target tasks without recomputing them.

[W1] Confidence intervals in Figure 4

We added 95% bootstrap intervals over queries to Figure 4 (page 8). At every model size and storage budget, the interval of the difference between EOGP and the best baseline at matched storage excludes zero for both NDCG@20 and Spearman correlation.

[W2] Whether the low-budget gain on GPT-2 comes mainly from one-bit storage

One-bit storage does contribute to the gain at small budgets, but the baselines cannot obtain this gain reliably. Figure 7 (page 22) shows that EOGP achieves higher LDS than every baseline stored in one bit at the same storage budget. In addition, the one-bit LDS of LoGra and GraSS collapses as the budget grows and falls below their FP16 LDS, whereas EOGP shows no such drop.

One-bit quantization keeps the sign of each coordinate but replaces every magnitude with one common scale, so it distorts a vector little only when its coordinate magnitudes are already similar. We expect that half-whitening flattens the coordinate magnitudes of EOGP, which keeps this distortion small, as we discuss in Appendix G.

We verified this on GPT-2 with the one-bit reconstruction error, which is the squared difference between the original and one-bit coordinates divided by the squared norm of the original. Half-whitening lowers this error of the stored coordinates from 0.73 to 0.46, which confirms that it reduces the distortion from one-bit quantization. We added this result to Appendix H (page 25).

[W2, Q3] Shared storage of each method and comparison at equal total storage

  • Cost. The shared storage of EOGP is considerably larger than that of the baselines, and most of it is the PCA correction matrices. The table below shows the shared storage at a matched per-example storage budget. However, we would like to emphasize that this cost is amortized over the number of stored examples. EOGP-R stores no correction matrices, and we proposed it for the case where shared storage is a burden. We added this table as Table 6 on page 25.
Shared storage (GB)1B7B13B32B
EOGP121.1243.3304.9490.1
EOGP-R0.92.84.39.1
LoGra (Random, PCA)0.040.120.190.40
GraSS0.010.010.020.03
LoRIF0.491.21.73.3
  • Benefit. Even in terms of total storage, the EK-FAC fidelity of EOGP exceeds that of all baselines. We added the results for a store of about 0.9M examples, the size of the OLMo 2 SFT data, as Figure 9 on page 23.

[Q4, Q5] GPT-2 settings of EOGP and damping of all methods

  • GPT-2 setting clarification. For GPT-2, the PCA was fit on the stored dataset, all 4,656 WikiText-2 examples. We set m to about four times k, with an upper limit of 262,656. k was 2,048, 4,096, 16,384, 65,536, and 262,144 and m was 8,192, 16,290, 65,884, 262,656, and 262,656 for the 12 KB, 24 KB, 96 KB, 384 KB, and 1.5 MB budgets. When k exceeds the fitting set size, the remaining dimensions use a random projection.
  • Damping of all methods. All methods use their official settings. EOGP uses the official setting of EK-FAC [6], 0.1 times the mean eigenvalue. The official setting of LoGra is also 0.1 times the mean eigenvalue, GraSS uses 1e-5 times the mean eigenvalue, and LoRIF uses 0.1 times the mean of the top 266 eigenvalues.

We added these clarifications to Appendices C.2 and C.3 (pages 16 and 17).

[Q6] Comparison with FP16 truncated EK-FAC, TrackStar, and Arnoldi-style projection

EOGP achieves higher EK-FAC fidelity than FP16 truncated EK-FAC and TrackStar at matched per-example storage across all model sizes. The only exception is the Spearman correlation on the 32B model at the three smallest budgets, where truncated EK-FAC is slightly higher. We added these results as Figure 10 on page 24. Arnoldi-style projection cannot be compared at matched storage because its projection basis becomes too large at LLM scale, and LoGra excludes it for the same reason [7].

[1] Bajaj, Payal, et al. "Ms marco: A human generated machine reading comprehension dataset." arXiv preprint 2016.
[2] Clark, Jonathan H., et al. "Tydiqa: A benchmark for information-seeking question answering in typologically diverse languages." TACL 2020.
[3] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." ICLR 2021.
[4] Suzgun, Mirac, et al. "Challenging big-bench tasks and whether chain-of-thought can solve them." Findings of ACL 2023.
[5] Clark, Peter, et al. "Think you have solved question answering? try arc, the ai2 reasoning challenge." arXiv preprint 2018.
[6] Grosse, Roger, et al. "Studying large language model generalization with influence functions." arXiv preprint 2023.
[7] Choe et al. What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions. NeurIPS 2025.

Thank you for the detailed response and the new experiments. They address the two points on which my rating mainly rested.

I am therefore raising my rating from 2 to 3. I retain from a higher score because the new evidence is still narrow. The LDS pools have 1,000 examples on a single 1B model from one family. The selection gains are clearest on TyDiQA, while on MMLU every selected subset, EOGP's included, stays slightly below the base model. I would encourage you to move Appendix H into the main text as planned, and to state there the measured storage trade-off between EOGP and EOGP-R (490.1 vs 9.1 GB of shared storage on 32B).

We are sincerely grateful to the reviewer for considering our response and for the positive feedback. We have uploaded a revised version of the paper. As the reviewer suggested, we first moved the LDS evaluation on OLMo 2 1B and the data selection result (Section 4.1) and the shared storage of EOGP and EOGP-R (Section 4.4) from Appendix H into the main text. We are still considering which of the remaining results in Appendix H should also be moved into the main text, and we will revise the paper accordingly.

Once again, we sincerely thank the reviewer for the effort devoted to evaluating our paper and for the truly helpful feedback.

Reviewer_uBTxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The influence functions require the model's gradients on the training data to estimate the importance of the datapoints. However, the authors point out that gradient of a datapoint on an 8B-parameter model requires 16 GB in half precision. They propose Eigenbasis Optimal Gradient Projection (EOGP), a scalable method for storing and retrieving influence-function representations for LLMs. They show that, if the goal is to preserve influence scores rather than raw gradients, the optimal low-dimensional representation is obtained by performing PCA on half-whitened gradients H−1/2H^{-1/2} where H is the curvature matrix. Given the large number of parameters in LLMs, it is not possible to compute this. Hence, they use EK-FAC to obtain a scalable approximation of the curvature eigensystem and project gradients into this space. Although EK-FAC provides a scalable curvature-aware basis, the resulting directions are not optimized for representing the distribution of half-whitened gradients. The authors therefore apply a second PCA stage within the projected space to obtain a more compact and informative representation before quantization. The resulting representations are then compressed using one-bit quantization, yielding substantially improved influence-attribution accuracy over prior compressed influence-function methods at comparable storage budgets. In their experiments, the authors demonstrate that EOGP retrieves influential training examples more effectively than competing baselines. They further perform ablation studies showing that the additional PCA refinement improves the quality of the learned representation and that, under a fixed storage budget, one-bit quantization yields better retrieval performance than FP16 by enabling more projected dimensions to be retained.

Critical strengths
  1. The paper derives a principled objective for influence-function compression rather than treating it as a generic dimensionality-reduction problem. By showing that influence preservation naturally leads to PCA on half-whitened gradients, it provides a clear theoretical foundation for constructing low-dimensional influence representations.
  2. The evaluation is fairly comprehensive. They evaluate this method against all the baselines on OLMO 2 (1B to 32B) and GPT-2. The ablations carefully validate the proposed design choices. They isolate the contribution of the PCA refinement stage and demonstrate that it improves retrieval quality over directly retaining EK-FAC directions. They also show that under a fixed memory budget, allocating storage toward retaining more influence-relevant dimensions can be more beneficial than storing fewer dimensions at higher numerical precision, highlighting an effective tradeoff between compression and retrieval quality. 3.In the appendix, the authors show that EOGP's learned projections are substantially more robust to aggressive quantization than those of competing methods. When EOGP representations are compressed from FP16 to 1-bit, the resulting influence scores remain highly correlated with their FP16 counterparts, achieving Pearson correlations of approximately 0.89-0.91. In contrast, LoGra and GraSS exhibit a much larger degradation under the same quantization setting.
  3. The paper addresses a practically important bottleneck in training data attribution. Rather than focusing solely on improving influence estimation, it studies how influence information can be stored and retrieved efficiently at scale, making attribution more feasible for large models and datasets.
Critical weaknesses
  1. Given that all experiments are conducted under a fixed storage budget, it would be valuable to include an additional ablation with unconstrained-storage setting where all methods are allocated the same projection dimensionality rather than the same storage budget. These experiments would help disentangle the contributions of EOGP's EK-FAC+PCA projection basis from those of quantization.
Questions for authors
  1. Table 1 of LoGra suggests that EK-FAC-based methods can incur substantial additional preprocessing costs relative to simpler projections. EOGP performs EK-FAC followed by an additional PCA. In section 4.4 and in the appendix table 3, the authors reports store-construction times comparable to LoGra, but it is unclear whether these measurements include the one-time costs of estimating EK-FAC statistics. They only report fitting the PCA basis.
3 comments

We sincerely thank the reviewer for the thoughtful review and for the time devoted to evaluating our paper. We also appreciate that the reviewer highlighted that our paper derives a principled objective for influence-function compression and addresses a practically important bottleneck in training data attribution.

We have uploaded a revised version of the paper. To make the new results easy to find, we collected the new experiments in Appendix H at the end (pages 22 to 25). In a future revision, we will move them to the appropriate places in the paper.

[W1] Ablation at the same projection dimensionality to separate the EK-FAC+PCA basis from one-bit quantization

At the same projection dimensionality, EOGP generally achieves higher LDS and EK-FAC fidelity than the baselines, even without one-bit quantization. Figure 7 (page 22) shows this comparison for LDS on GPT-2, and Figure 5 (right, page 8) shows it for EK-FAC fidelity.

These figures also show that one-bit quantization substantially lowers both the LDS and the EK-FAC fidelity of LoGra and GraSS as the budget grows, whereas EOGP shows no such drop. One-bit quantization keeps the sign of each coordinate but replaces every magnitude with one common scale, so it distorts a vector little only when its coordinate magnitudes are already similar. We expect that half-whitening flattens the coordinate magnitudes of EOGP, which keeps this distortion small, as we discuss in Appendix G.

We verified this on GPT-2 with the one-bit reconstruction error, which is the squared difference between the original and one-bit coordinates divided by the squared norm of the original. Half-whitening lowers this error of the stored coordinates from 0.73 to 0.46, which confirms that it reduces the distortion from one-bit quantization. We added this result to Appendix H (page 25).

[Q1] One-time cost of EK-FAC fitting and whether Table 3 includes it

EK-FAC fitting takes from 0.2 hours for the 1B model to 23.1 hours for the 32B model on a single B200 GPU, as summarized in the table below. Table 3 reports the PCA fitting time and the per-example store-construction time, and it does not include the EK-FAC fitting time. EK-FAC fitting can be shortened with multiple GPUs because it accumulates covariances over documents, and each document is processed independently. Among the baselines, LoGra (PCA) also requires the EK-FAC eigenbasis for its initialization and pays the cost of fitting it, whereas LoGra (Random) does not. We added this result to Appendix H (page 24).

Model1B7B13B32B
EK-FAC fitting time (h)0.20.92.423.1
Reviewer_uBTxOpenReview ↗

Thank you for your response

I thank the reviewers for the response and the additional experiments. The additional experiments such as counterfactual retraining also demonstrate the method is effective at training data attribution under reduced storage budget. Please note that Figure 5 and Figure 7 compare the method and the baselines by increasing the storage but not by providing the same reduced dimensions as I had requested. Overall, I wish to retain my original score.

We are sincerely grateful to the reviewer for considering our response and for the positive feedback on the additional experiments. The review was truly helpful, and we appreciate the effort devoted to evaluating our paper.

Comments on the paper

1 comment

Dear Reviewers,

We sincerely appreciate the thoughtful comments and all the time that went into reviewing our paper. As the discussion period comes to a close, we hope our responses have helped address the concerns raised. If there are any remaining questions or additional comments, please feel free to leave them at any time. It would be a great help to us, and we would be more than happy to clarify anything further.

Warm regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.