- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents dattri-LLM, a library for training data attribution (TDA) on large language models (LLMs). It features a FLOP-aware routing mechanism that chooses the cheaper representation of per-example gradients (factorized or materialized), captures gradients from existing training loops through autograd hooks, and provides interfaces for new attribution methods and training-time callbacks. The library is benchmarked against competing libraries on Qwen models from 0.5B to 110B parameters, where the authors report an average speedup in attribution throughput. The authors also compare attribution fidelity against trajectory-specific leave-one-out retraining (TSLOO) on three smaller models and reproduce an online data-selection method.
Critical strengths
- The paper proposes a useful library that unifies many TDA methods (gradient similarity, curvature-based influence, and trajectory-based methods) and shows that gradient-based attribution is feasible on LLMs up to 110B parameters, which is valuable for the community.
- The paper has very detailed reporting. The appendix gives per-configuration throughput, batch sizes, and GPU memory usage, including infeasible cases and cases where the library does not win, making the results easier to check.
Critical weaknesses
-
The throughput gain claim ( over existing libraries) is hard to interpret:
- Section 4.1 compares the attribution throughput of entire libraries without a component-level ablation or sufficient explanation of the sources of the speedup, so it remains unclear what exactly in
dattri-LLMcontributes to this gain. The FLOP-aware routing mechanism is presented alongside the speedup in the Introduction, but its only ablation (Appendix B.4, Figure 7) is run in a different setting and does not really explain the reported cross-library gain. - The paper emphasizes the ability to scale to larger LLMs, but the headline average overstates the gain at these scales. As I calculated from Table 4 in the appendix using the arithmetic mean of speedup ratios against the fastest completed competitor for each method, the throughput gain falls notably with model scale: at 0.5B, at 7B, at 14B, at 32B, and at 72B and 110B (where only GradDot has a completed competing implementation). A decline is expected, but the paper should report the gain by model scale and attribution method, rather than only as a single average over 20 configurations of methods and LLMs.
- Section 4.1 compares the attribution throughput of entire libraries without a component-level ablation or sufficient explanation of the sources of the speedup, so it remains unclear what exactly in
-
Missing code and limited evidence for compatibility:
- No code is provided, although an anonymized repository could facilitate evaluation during review. Since the library itself is the paper's main contribution, I think this matters more here than for a typical submission.
- Without the code, I cannot fully verify the central claims of extensibility and compatibility. The paper claims, "We demonstrate this integration with HuggingFace Trainer, TRL, and OLMo, and evaluate capture under DDP and FSDP", but I think this is not fully addressed. Section 3.3 and Appendix A.5 are useful in describing the design and capture mechanism, but they do not provide experimental validation across all these settings. As far as I can tell, only OLMo is actually run in an experiment, while HuggingFace Trainer and TRL are shown only through integration code snippets (Figure 6). Similarly, FSDP is used in the throughput and memory benchmarks, but no experiment explicitly evaluates gradient capture under DDP.
Questions for authors
-
The paper claims that the library "offers superior attribution fidelity-cost trade-offs across a range of models with different model families and scales", but attribution fidelity is measured only up to 7B parameters. I understand that measuring fidelity on larger LLMs is expensive, but can you report it on at least one larger model (e.g., 14B or 32B)?
-
Also related to Q1, the fidelity-cost trade-off study in Figure 3 compares only against Bergson. This is understandable for the trajectory-based methods, but the figure also includes EK-FAC, which Kronfluence and LogIX implement (Table 4). Is there a particular reason not to include them in Figure 3?
-
The paper states that LogIX does not support FSDP (Table 1, Appendix B.1) and benchmarks it with one full model copy per GPU, leading to out-of-memory (OOM) failures (Figure 2). However, the LogIX paper describes its hooks as integrating with FSDP [1], and the LogIX repository's README also lists FSDP as supported (https://github.com/logix-project/logix). Can you clarify whether this refers to a specific limitation of the evaluated LogIX implementation or configuration, or if I am missing something?
[1] Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy, Roger Grosse, and Eric P. Xing. What is your data worth to GPT? LLM-scale data valuation with influence functions. In Advances in Neural Information Processing Systems, volume 38, pp. 145944-145985, 2025.
8 comments
5. [Q1] Fidelity evaluation on models larger than 7B.
The purpose of Section 4.2 is to examine how the library’s configurable options control the fidelity–efficiency trade-off across model scales. Our evaluation covers GPT-2, Qwen2.5-1.5B, and OLMo-3-7B and shows a consistent pattern: stronger compression reduces computational cost while sacrificing attribution fidelity. This consistency across model families and scales supports the expectation that the same qualitative trade-off will persist at larger scales, although its exact magnitude may vary. We therefore believe the current evidence is sufficient to demonstrate that our users can adjust the representation size to balance fidelity and efficiency.
Moreover, fidelity evaluation becomes increasingly computationally prohibitive as model size grows. The main cost of extending the fidelity experiment is constructing the retraining ground truth. At each model scale, our protocol requires at least 50 separate leave-one-out training runs. Extending this protocol to 14B or 32B is therefore substantially more expensive than running attribution alone.
This evaluation bottleneck is also reflected in prior work. SOURCE’s language-model experiments extend only to GPT-2, and the authors explicitly restrict their evaluation to tasks requiring less than 20 minutes per training or fine-tuning run (Bae et al., 2024, Appendix E). Similarly, LoGra scales attribution computation to Llama3-8B to demonstrate efficiency and qualitative attribution at that scale, but also restricts retraining-based fidelity evaluation to GPT-2 and smaller models due to the cost of repeated retraining (Choe et al., 2025, Sections 4.1–4.2). DVEmb uses Pythia-410M for its main efficiency and training-dynamics experiments, but limits fidelity evaluation to small MNIST models (Wang et al., 2025, Section 5 and Appendix E). Grosse et al. (2023) apply EK-FAC influence estimation to models with up to 52B parameters but validate approximation fidelity only up to an 810M-parameter language model. These precedents illustrate the substantial additional computational burden of validating attribution fidelity beyond computing attribution scores alone.
6. [Q2] Why Figure 3 does not include Kronfluence and LogIX EK-FAC.
We selected Bergson because it provides the trajectory-based baselines MAGIC, SOURCE, and TrackStar alongside EK-FAC, which allows Figure 3 to compare these methods within a common baseline implementation. Including more EK-FAC implementations from other libraries would provide limited additional insight into fidelity, as implementations of the same method are expected to produce similar results when their configurations are aligned. Moreover, cross-library EK-FAC efficiency is evaluated separately in Section 4.1 and Appendix B.3, including comparisons with Kronfluence and LogIX. We therefore expect these additional implementations to have limited impact on the reported Pareto frontier and our conclusions.
References
Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate unrolling. In Advances in Neural Information Processing Systems, volume 37, pp. 66647–66686, 2024.
Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy, Roger Grosse, and Eric P. Xing. What is your data worth to GPT? LLM-scale data valuation with influence functions. In Advances in Neural Information Processing Systems, volume 38, pp. 145944–145985, 2025.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023.
Jiachen T. Wang, Dawn Song, James Zou, Prateek Mittal, and Ruoxi Jia. Capturing the temporal dependence of training data influence. In International Conference on Learning Representations, volume 2025, pp. 11236–11273, 2025.
I thank the authors for addressing my concerns related to the code release, and especially for identifying the issue with LogIX and updating the results in a timely manner. Below, I provide a few follow-up points.
1. Mixed effects of memory and speed at larger scales
From Table 4, at 72B and 110B, LogIX runs at half the per-device batch size of dattri-LLM. With this difference, it is hard to tell whether the large gains in this setting (3.5–8.9$\times$) come from memory efficiency (fitting a larger batch) or from truly faster computation (processing the same batch in less time).
Could you also report dattri-LLM's throughput at LogIX's batch size at these scales (batch 4 at 72B, batch 2 at 110B), so that the two effects can be separated?
2. Contribution of FLOP-aware routing to throughput
The paper provides complementary evidence for these factors.
I understand, but I was not asking whether these factors help individually. My question is how much the FLOP-aware routing actually contributes to the reported gain. From my perspective, the Introduction strongly presents FLOP-aware routing as the main feature of efficiency, together with the 1.4–8.9$\times$ throughput gain.
Please either clarify which key features account for this gain, or revise the text to reflect the routing's actual contribution more appropriately.
First, Appendix B.3 holds the batch size fixed across libraries. Under these conditions, dattri-LLM has the lowest runtime in nine of twelve configurations, including every curvature-based configuration.
While reviewing Appendix B.3, I noticed that the LogIX numbers in Table 9 are identical to the original submission (e.g., 68 s, 89 s and 143 s with one query), while the same fix changed LogIX's throughput substantially in Table 4. Were the LogIX rows in Table 9 re-run with the corrected setup, or am I missing something?
Second, Appendix B.4 isolates routing within the same implementation by comparing adaptive execution with each fixed representation.
Third, Appendix B.6 isolates the optional weight-gradient-free capture mode used outside training: standard capture takes 7–19% longer in those paired comparisons.
Moreover, while these ablations are useful for understanding each mechanism on its own, they are run only on one small model (Pythia-410M) on single GPUs (A40, L40S), unlike the setting of Table 4. Because of this, they do not show how much the FLOP-aware routing contributes to the 1.4–8.9$\times$ gain.
Could you report dattri-LLM's throughput in the Table 4 setting with routing fixed to a single representation, at least at one small and one large scale?
I will update my assessment once these points are addressed.
We thank the reviewer for the detailed assessment, particularly for identifying the issue with our LogIX configuration. We have corrected that comparison, released the anonymous implementation at https://anonymous.4open.science/r/dattri-llm-4CFC/, and revised the presentation of the throughput results. We address the concerns individually below.
1. [Q3] LogIX’s FSDP support and the fairness of the benchmark.
The reviewer is correct: our original statement that “LogIX does not support FSDP” was inaccurate. In our initial configuration, FSDP wrapping preceded LogIX instrumentation and caused failures. We subsequently confirmed that LogIX works with FSDP when instrumentation and LoRA projection insertion occur before FSDP wrapping. The failure was therefore due to the order of operations in the setup, and we have corrected the statement.
We have corrected the comparison and updated the affected scaling and throughput benchmarks. Table 1 now acknowledges distributed execution, Appendix B.1 specifies that all four libraries use FSDP in the four-H200 throughput comparison, and Table 4 reports completed LogIX runs through 110B for all three methods.
During this process, we also identified another benchmarking issue: following its documentation, we previously evaluated LogIX using its attribution-from-storage workflow, while all other libraries used on-the-fly attribution. This introduced additional disk I/O overhead for LogIX. In the updated benchmark, we aligned LogIX’s training-gradient workflow with the on-the-fly attribution used by the other libraries. This change improves LogIX’s runtime, but our library still retains a substantial efficiency advantage.
2. [W1(a)] What contributes to the throughput gains?
The cross-library result measures the combined effects of gradient representation, operation routing, capture implementation, curvature processing, and the batch size that fits within the resource budget. We agree that the routing ablation alone does not explain the entire gain.
The paper provides complementary evidence for these factors. First, Appendix B.3 holds the batch size fixed across libraries. Under these conditions, dattri-LLM has the lowest runtime in nine of twelve configurations, including every curvature-based configuration. For projected K-FAC and EK-FAC, its throughput advantage over LogIX is 1.6–2.5×, showing that the improvement is not solely due to larger feasible batches.
Second, Appendix B.4 isolates routing within the same implementation by comparing adaptive execution with each fixed representation. Adaptive routing provides a 1.16–1.50× speedup over always-materialized execution. Third, Appendix B.6 isolates the optional weight-gradient-free capture mode used outside training: standard capture takes 7–19% longer in those paired comparisons. Finally, the memory measurements in Section 4.1 and Tables 5–6 explain how representation and curvature-storage choices affect the feasible workload and batch size.
These ablations isolate the effects of individual mechanisms under controlled conditions. The overall performance gain reflects the combined effects of multiple factors and should not be attributed solely to routing.
3. [W1(b)] Reporting gains by model scale and attribution method. We agree that an aggregate average is insufficient to describe performance at individual scales. Section 4.1 now reports method-specific ranges and discusses the largest models. Table 4 gives throughput and per-device batch size for every library, method, and scale.
The updated results also change the large-model comparison because LogIX now completes the curvature-based methods through 110B. Against the fastest completed competitor, GradDot’s gain is approximately 1.4–1.8× across the evaluated scales. For K-FAC and EK-FAC, the gains over LogIX are approximately 1.8–3.6× through 32B, increasing to 3.5–4.8× at 72B and 6.6–8.9× at 110B. The updated average is approximately 2.9× over the 24 method–scale configurations.
4. [W2] Code availability and evidence for integration and distributed capture.
The anonymous repository is now available at https://anonymous.4open.science/r/dattri-llm-4CFC/. It includes the library implementation, experimental scripts, unit tests, and runnable examples.
The released
examples/trainers/directory provides executable HuggingFace Trainer, TRL SFT, and TRL GRPO integrations. For distributed capture,tests/gradient/test_live_sharded.pyexplicitly exercises both DDP and FSDP and compares the materialized per-sample gradients with single-process autograd references. It also checks that training shards cover the dataset and each rank receives the required queries. These released checks provide concrete validation procedures beyond the integration snippets.