acceptodds
Under review as a conference paper at ICLR 2027

dattri-LLM: A Unified and Efficient Library for Training Data Attribution at LLM Scale

15 views

Abstract

Training data attribution (TDA) estimates the contribution of individual training examples to model outputs. Most scalable TDA methods rely on per-example gradients, whose computation and use at LLM scale pose challenges in efficiency, compatibility, and extensibility. We introduce dattri-LLM, a TDA library that makes gradient-based attribution more practical at scale. For efficiency, dattri-LLM uses compact gradient representations and dynamically routes gradient operations based on a cost model. For compatibility, its capture mechanism collects per-example gradients from existing training loops that call backward(), without requiring changes to the loop or its configuration. This includes distributed training with DDP and FSDP and pipelines built with HuggingFace Transformers, TRL, and OLMo. For extensibility, dattri-LLM exposes reusable gradient operations and training-time callbacks for implementing attribution methods and applications. These interfaces support a variety of attribution methods, including gradient similarity, curvature-based influence, and trajectory-based methods, as well as applications that act on gradients during training, such as online data selection. On the same hardware and workload, dattri-LLM achieves 3.2x the throughput of the fastest competing library on average, scales multiple attribution methods to 110B-parameter models across four H200 GPUs, and offers superior attribution fidelity-cost trade-offs across a range of models with different model families and scales.

Primary area:
interpretability and explainable AI
Keywords:
data attribution, data-centric AI

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_NuYBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents dattri-LLM, a library for training data attribution (TDA) on large language models (LLMs). It features a FLOP-aware routing mechanism that chooses the cheaper representation of per-example gradients (factorized or materialized), captures gradients from existing training loops through autograd hooks, and provides interfaces for new attribution methods and training-time callbacks. The library is benchmarked against competing libraries on Qwen models from 0.5B to 110B parameters, where the authors report an average 3.2×3.2\times speedup in attribution throughput. The authors also compare attribution fidelity against trajectory-specific leave-one-out retraining (TSLOO) on three smaller models and reproduce an online data-selection method.

Critical strengths
  1. The paper proposes a useful library that unifies many TDA methods (gradient similarity, curvature-based influence, and trajectory-based methods) and shows that gradient-based attribution is feasible on LLMs up to 110B parameters, which is valuable for the community.
  2. The paper has very detailed reporting. The appendix gives per-configuration throughput, batch sizes, and GPU memory usage, including infeasible cases and cases where the library does not win, making the results easier to check.
Critical weaknesses
  1. The throughput gain claim (3.2×3.2\times over existing libraries) is hard to interpret:

    • Section 4.1 compares the attribution throughput of entire libraries without a component-level ablation or sufficient explanation of the sources of the speedup, so it remains unclear what exactly in dattri-LLM contributes to this gain. The FLOP-aware routing mechanism is presented alongside the 3.2×3.2\times speedup in the Introduction, but its only ablation (Appendix B.4, Figure 7) is run in a different setting and does not really explain the reported cross-library gain.
    • The paper emphasizes the ability to scale to larger LLMs, but the headline 3.2×3.2\times average overstates the gain at these scales. As I calculated from Table 4 in the appendix using the arithmetic mean of speedup ratios against the fastest completed competitor for each method, the throughput gain falls notably with model scale: 5.5×5.5\times at 0.5B, 3.0×3.0\times at 7B, 2.6×2.6\times at 14B, 1.9×1.9\times at 32B, and 1.4×1.4\times at 72B and 110B (where only GradDot has a completed competing implementation). A decline is expected, but the paper should report the gain by model scale and attribution method, rather than only as a single average over 20 configurations of methods and LLMs.
  2. Missing code and limited evidence for compatibility:

    • No code is provided, although an anonymized repository could facilitate evaluation during review. Since the library itself is the paper's main contribution, I think this matters more here than for a typical submission.
    • Without the code, I cannot fully verify the central claims of extensibility and compatibility. The paper claims, "We demonstrate this integration with HuggingFace Trainer, TRL, and OLMo, and evaluate capture under DDP and FSDP", but I think this is not fully addressed. Section 3.3 and Appendix A.5 are useful in describing the design and capture mechanism, but they do not provide experimental validation across all these settings. As far as I can tell, only OLMo is actually run in an experiment, while HuggingFace Trainer and TRL are shown only through integration code snippets (Figure 6). Similarly, FSDP is used in the throughput and memory benchmarks, but no experiment explicitly evaluates gradient capture under DDP.
Questions for authors
  1. The paper claims that the library "offers superior attribution fidelity-cost trade-offs across a range of models with different model families and scales", but attribution fidelity is measured only up to 7B parameters. I understand that measuring fidelity on larger LLMs is expensive, but can you report it on at least one larger model (e.g., 14B or 32B)?

  2. Also related to Q1, the fidelity-cost trade-off study in Figure 3 compares only against Bergson. This is understandable for the trajectory-based methods, but the figure also includes EK-FAC, which Kronfluence and LogIX implement (Table 4). Is there a particular reason not to include them in Figure 3?

  3. The paper states that LogIX does not support FSDP (Table 1, Appendix B.1) and benchmarks it with one full model copy per GPU, leading to out-of-memory (OOM) failures (Figure 2). However, the LogIX paper describes its hooks as integrating with FSDP [1], and the LogIX repository's README also lists FSDP as supported (https://github.com/logix-project/logix). Can you clarify whether this refers to a specific limitation of the evaluated LogIX implementation or configuration, or if I am missing something?


[1] Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy, Roger Grosse, and Eric P. Xing. What is your data worth to GPT? LLM-scale data valuation with influence functions. In Advances in Neural Information Processing Systems, volume 38, pp. 145944-145985, 2025.

8 comments

We thank the reviewer for the detailed assessment, particularly for identifying the issue with our LogIX configuration. We have corrected that comparison, released the anonymous implementation at https://anonymous.4open.science/r/dattri-llm-4CFC/, and revised the presentation of the throughput results. We address the concerns individually below.

1. [Q3] LogIX’s FSDP support and the fairness of the benchmark.

The reviewer is correct: our original statement that “LogIX does not support FSDP” was inaccurate. In our initial configuration, FSDP wrapping preceded LogIX instrumentation and caused failures. We subsequently confirmed that LogIX works with FSDP when instrumentation and LoRA projection insertion occur before FSDP wrapping. The failure was therefore due to the order of operations in the setup, and we have corrected the statement.

We have corrected the comparison and updated the affected scaling and throughput benchmarks. Table 1 now acknowledges distributed execution, Appendix B.1 specifies that all four libraries use FSDP in the four-H200 throughput comparison, and Table 4 reports completed LogIX runs through 110B for all three methods.

During this process, we also identified another benchmarking issue: following its documentation, we previously evaluated LogIX using its attribution-from-storage workflow, while all other libraries used on-the-fly attribution. This introduced additional disk I/O overhead for LogIX. In the updated benchmark, we aligned LogIX’s training-gradient workflow with the on-the-fly attribution used by the other libraries. This change improves LogIX’s runtime, but our library still retains a substantial efficiency advantage.

2. [W1(a)] What contributes to the throughput gains?

The cross-library result measures the combined effects of gradient representation, operation routing, capture implementation, curvature processing, and the batch size that fits within the resource budget. We agree that the routing ablation alone does not explain the entire gain.

The paper provides complementary evidence for these factors. First, Appendix B.3 holds the batch size fixed across libraries. Under these conditions, dattri-LLM has the lowest runtime in nine of twelve configurations, including every curvature-based configuration. For projected K-FAC and EK-FAC, its throughput advantage over LogIX is 1.6–2.5×, showing that the improvement is not solely due to larger feasible batches.

Second, Appendix B.4 isolates routing within the same implementation by comparing adaptive execution with each fixed representation. Adaptive routing provides a 1.16–1.50× speedup over always-materialized execution. Third, Appendix B.6 isolates the optional weight-gradient-free capture mode used outside training: standard capture takes 7–19% longer in those paired comparisons. Finally, the memory measurements in Section 4.1 and Tables 5–6 explain how representation and curvature-storage choices affect the feasible workload and batch size.

These ablations isolate the effects of individual mechanisms under controlled conditions. The overall performance gain reflects the combined effects of multiple factors and should not be attributed solely to routing.

3. [W1(b)] Reporting gains by model scale and attribution method. We agree that an aggregate average is insufficient to describe performance at individual scales. Section 4.1 now reports method-specific ranges and discusses the largest models. Table 4 gives throughput and per-device batch size for every library, method, and scale.

The updated results also change the large-model comparison because LogIX now completes the curvature-based methods through 110B. Against the fastest completed competitor, GradDot’s gain is approximately 1.4–1.8× across the evaluated scales. For K-FAC and EK-FAC, the gains over LogIX are approximately 1.8–3.6× through 32B, increasing to 3.5–4.8× at 72B and 6.6–8.9× at 110B. The updated average is approximately 2.9× over the 24 method–scale configurations.

4. [W2] Code availability and evidence for integration and distributed capture.

The anonymous repository is now available at https://anonymous.4open.science/r/dattri-llm-4CFC/. It includes the library implementation, experimental scripts, unit tests, and runnable examples.

The released examples/trainers/ directory provides executable HuggingFace Trainer, TRL SFT, and TRL GRPO integrations. For distributed capture, tests/gradient/test_live_sharded.py explicitly exercises both DDP and FSDP and compares the materialized per-sample gradients with single-process autograd references. It also checks that training shards cover the dataset and each rank receives the required queries. These released checks provide concrete validation procedures beyond the integration snippets.

5. [Q1] Fidelity evaluation on models larger than 7B.

The purpose of Section 4.2 is to examine how the library’s configurable options control the fidelity–efficiency trade-off across model scales. Our evaluation covers GPT-2, Qwen2.5-1.5B, and OLMo-3-7B and shows a consistent pattern: stronger compression reduces computational cost while sacrificing attribution fidelity. This consistency across model families and scales supports the expectation that the same qualitative trade-off will persist at larger scales, although its exact magnitude may vary. We therefore believe the current evidence is sufficient to demonstrate that our users can adjust the representation size to balance fidelity and efficiency.

Moreover, fidelity evaluation becomes increasingly computationally prohibitive as model size grows. The main cost of extending the fidelity experiment is constructing the retraining ground truth. At each model scale, our protocol requires at least 50 separate leave-one-out training runs. Extending this protocol to 14B or 32B is therefore substantially more expensive than running attribution alone.

This evaluation bottleneck is also reflected in prior work. SOURCE’s language-model experiments extend only to GPT-2, and the authors explicitly restrict their evaluation to tasks requiring less than 20 minutes per training or fine-tuning run (Bae et al., 2024, Appendix E). Similarly, LoGra scales attribution computation to Llama3-8B to demonstrate efficiency and qualitative attribution at that scale, but also restricts retraining-based fidelity evaluation to GPT-2 and smaller models due to the cost of repeated retraining (Choe et al., 2025, Sections 4.1–4.2). DVEmb uses Pythia-410M for its main efficiency and training-dynamics experiments, but limits fidelity evaluation to small MNIST models (Wang et al., 2025, Section 5 and Appendix E). Grosse et al. (2023) apply EK-FAC influence estimation to models with up to 52B parameters but validate approximation fidelity only up to an 810M-parameter language model. These precedents illustrate the substantial additional computational burden of validating attribution fidelity beyond computing attribution scores alone.

6. [Q2] Why Figure 3 does not include Kronfluence and LogIX EK-FAC.

We selected Bergson because it provides the trajectory-based baselines MAGIC, SOURCE, and TrackStar alongside EK-FAC, which allows Figure 3 to compare these methods within a common baseline implementation. Including more EK-FAC implementations from other libraries would provide limited additional insight into fidelity, as implementations of the same method are expected to produce similar results when their configurations are aligned. Moreover, cross-library EK-FAC efficiency is evaluated separately in Section 4.1 and Appendix B.3, including comparisons with Kronfluence and LogIX. We therefore expect these additional implementations to have limited impact on the reported Pareto frontier and our conclusions.

References

Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate unrolling. In Advances in Neural Information Processing Systems, volume 37, pp. 66647–66686, 2024.

Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy, Roger Grosse, and Eric P. Xing. What is your data worth to GPT? LLM-scale data valuation with influence functions. In Advances in Neural Information Processing Systems, volume 38, pp. 145944–145985, 2025.

Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023.

Jiachen T. Wang, Dawn Song, James Zou, Prateek Mittal, and Ruoxi Jia. Capturing the temporal dependence of training data influence. In International Conference on Learning Representations, volume 2025, pp. 11236–11273, 2025.

I thank the authors for addressing my concerns related to the code release, and especially for identifying the issue with LogIX and updating the results in a timely manner. Below, I provide a few follow-up points.


1. Mixed effects of memory and speed at larger scales

From Table 4, at 72B and 110B, LogIX runs at half the per-device batch size of dattri-LLM. With this difference, it is hard to tell whether the large gains in this setting (3.5–8.9$\times$) come from memory efficiency (fitting a larger batch) or from truly faster computation (processing the same batch in less time).

Could you also report dattri-LLM's throughput at LogIX's batch size at these scales (batch 4 at 72B, batch 2 at 110B), so that the two effects can be separated?


2. Contribution of FLOP-aware routing to throughput

The paper provides complementary evidence for these factors.

I understand, but I was not asking whether these factors help individually. My question is how much the FLOP-aware routing actually contributes to the reported gain. From my perspective, the Introduction strongly presents FLOP-aware routing as the main feature of efficiency, together with the 1.4–8.9$\times$ throughput gain.

Please either clarify which key features account for this gain, or revise the text to reflect the routing's actual contribution more appropriately.

First, Appendix B.3 holds the batch size fixed across libraries. Under these conditions, dattri-LLM has the lowest runtime in nine of twelve configurations, including every curvature-based configuration.

While reviewing Appendix B.3, I noticed that the LogIX numbers in Table 9 are identical to the original submission (e.g., 68 s, 89 s and 143 s with one query), while the same fix changed LogIX's throughput substantially in Table 4. Were the LogIX rows in Table 9 re-run with the corrected setup, or am I missing something?

Second, Appendix B.4 isolates routing within the same implementation by comparing adaptive execution with each fixed representation.

Third, Appendix B.6 isolates the optional weight-gradient-free capture mode used outside training: standard capture takes 7–19% longer in those paired comparisons.

Moreover, while these ablations are useful for understanding each mechanism on its own, they are run only on one small model (Pythia-410M) on single GPUs (A40, L40S), unlike the setting of Table 4. Because of this, they do not show how much the FLOP-aware routing contributes to the 1.4–8.9$\times$ gain.

Could you report dattri-LLM's throughput in the Table 4 setting with routing fixed to a single representation, at least at one small and one large scale?


I will update my assessment once these points are addressed.

Reviewer_htXkOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a (to-be?) open-source library for training data attribution and data selection for LLMs. It includes experiments with models of sized up to 110B parameters and showcases strong efficiency than existing open-source libraries. The key contributions are,

  1. Ease-of-integration with various training libraries such as HF, TRL and OLMo. I believe it could be extended to other libraries by the open-source community.
  2. Interfaces to add new data attribution methods. The paper includes an illustration a recent method, Dr. Post-Training.
  3. Cost-aware routing between two gradient representations, factorized and materialized. I believe this is a valuable contribution of the library.

Experiments include a scaling analysis on Qwen (0.5-110B) where dattri-LLM performs better tradeoff in throughput and peak memory usage compared to other libraries. They also compare the implementation of various attribution methods in dattri-LLM to the their original implementations.

Overall, I think the library is well structured and likely extensible to future attribution methods and trainer libraries.

Critical strengths
  • Paper empirically shows that dattri-LLM is an efficient implementation of existing data attribution methods and has the potential to be easily extended with any newer data attribution methods.
  • The gradient capture and routing method seems novel (FLOP-aware routing between factorized and materialized representations). Paper could benefit from more analysis of this method. Figure 7 in Appendix provided a comparison but it shows that FLOP-aware routing is just as effective as factorized-only. Maybe I am missing something?
Critical weaknesses
  • Probably not a critical weakness but I didn't see any discussion about costs of gradient capture to storage. Is this done asynchronously? and how does it affect the overall throughput? Also, its not immediately clear how the gradient capture in dattri-LLM is better than the one in LogIX or Bergson libraries.
  • I am not sure I fully understand how the FLOP-aware routing is learned. For a given model, training dataset and attribution method, is it a one-time cost to learn the router? More details here can help.
Questions for authors

See my questions in the strengths and weaknesses sections above. Some additional comments,

  • Is dattri-LLM easily extensible to modern architectures? For instance, MoE and linear-attention?
  • The paper doesn't clearly establish the bounds for model sizes or training dataset sizes currently supported by dattri-LLM. What is a practical limit that a user of the library should be aware of?
4 comments

We thank the reviewer for the helpful questions about capture overhead, routing, architecture compatibility, and practical limits. We clarify these points below and have expanded the implementation details in the revised paper. We have also anonymously released our code at https://anonymous.4open.science/r/dattri-llm-4CFC/ for your reference.

1. [W1] Gradient capture, storage, and throughput overhead.

Gradient storage is optional. In the on-the-fly workflow used for the main throughput benchmark, gradients are captured and consumed during attribution without writing a training-gradient cache to disk. The reported throughput therefore only includes gradient capture, curvature estimation where required, and scoring.

For users who wish to store gradients, the offload callback supports asynchronous writes through async_write. As described in Appendix A.7, a background thread waits for a producer-recorded CUDA event, transfers records to host memory on a separate stream, and writes them in submission order. A bounded queue, holding two record groups by default, applies backpressure if the writer falls behind. This permits transfers and writes to overlap computation while bounding buffered device memory. However, the overhead usually depends on several factors, such as I/O volume and storage bandwidth. For reference, in a small-scale experiment with full-factor attribution on Pythia-410M, asynchronous writes reduced per-step runtime by 12%, from 7.4 s to 6.5 s.

2. [W1] Advantages of the capture mechanism over existing libraries.

The main distinction of our proposed capture is how it integrates with an evolving training run and what applications can do with the captured gradients. LogIX’s supplied HuggingFace integration replaces the optimizer and scheduler with no-op implementations for checkpoint-based extraction (logix/huggingface/patch.py). Bergson provides training-time capture through a HuggingFace Trainer integration that augments the dataset and collator with sample indices (bergson/huggingface.py). These are useful capabilities, but they differ from our framework-independent capture interface: dattri-LLM attaches to existing PyTorch training loops without requiring a specific training framework or changes to the underlying training logic, and supports both offline attribution and online applications such as data selection.

dattri-LLM captures inputs and output gradients through hooks during existing backward calls, while keeping the training loop, optimizer, and data pipeline unchanged. Content-based sample hashes identify the captured examples without adding dataset indices. The same mechanism supports HuggingFace Trainer, TRL, and OLMo integrations, with runnable examples in examples/trainers/ and an OLMo-core experiment in Section 4.2.

The captured representations are also available to callbacks within the training step. This permits applications to act on gradients before the optimizer update, as demonstrated by online data selection in Section 5. Sections 3.3–3.4 and Appendices A.5–A.6 describe these interfaces in more detail.

3. [W2] How the FLOP-aware router is obtained and its setup cost.

The router is analytical, not learned. It requires neither a router-training stage nor a profiling stage. For each supported layer and operation, it compares operation-count estimates derived from layer dimensions, sequence lengths, comparison-batch sizes, and any projection dimensions, then selects the cheaper factorized or materialized route.

For example, Section 3.2 and Appendix A.4 give the costs of computing a cross-Gram matrix through either representation. The decision is evaluated from the current operation’s shapes, so there is no one-time learning cost tied to a particular model, training dataset, or attribution method. Both routes represent the same mathematical operation and introduce no additional approximation in exact arithmetic.

4. [Q1] MoE and linear-attention architectures.

Yes. The factorized capture mechanism applies to linear layers within these architectures, and Section 3.2 now briefly notes this support. The released implementation also handles two practical complications: MoE routing can flatten and reorder tokens across experts, and fused implementations can bypass ordinary module hooks. We track routed tokens to restore their sample identities and provide capture-time overrides that expose the relevant operations.

The repository includes tests/gradient/test_fused_experts.py, which compares expert and router gradients against per-sample autograd on small Mixtral and GPT-OSS models, and tests/gradient/test_linear_attention.py, which checks captured gradients and output equivalence on models including Qwen3-Next and Kimi-Linear, alongside recurrent/state-space architectures. These checks verify the correctness of gradient capture for the supported layers and implementations. Although all linear modules are supported in theory, implementation-specific adaptations are still required in practice, and we are actively extending support to additional layer implementations.

5. [Q2] Practical model and dataset limits. Our demonstrated scale is 110B parameters on four H200 GPUs, using the configurations in Section 4.1. Table 6 reports memory at batch size one and provides a practical reference for hardware requirements. Feasibility mainly depends on whether the model fits across the available devices, and other factors like sequence length, batch size, projection, and attribution method also affect this limit.

Dataset size generally does not impose a hard limit on attribution, but larger datasets increase computational costs and, when gradient storage is enabled, storage requirements. In our library, for additional flexibility, the disk, memory, and tiered residency options described in Appendix A.7 further allow users to control where intermediate representations are stored and balance host-memory usage against disk storage.

thanks for answering my questions, would be nice to see these clarifications in the next draft. I am happy to increase my score.

We thank the reviewer for the constructive feedback and for raising their score. We have incorporated the clarifications into the revised manuscript. As promised, we will also publicly release the code by the time of the conference.

Reviewer_KPPGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors discuss their new software package, dattri-LLM for training data attribution (TDA) and claim that their package makes gradient-based attributions practical at scale. Their software library has three characteristics - it supports factorized and materialized representations of per-example gradients (with options to reduce dimensionality in both cases) with a routing model to choose the most efficient option per layer and operation, it captures gradients from existing training loops with autograd hooks and works with both DDP and FSDP paradigms of training, and it has callback options during gradient capture allowing applications to inspect or modify training state (this can inspire new applications). The library itself has an architecture stack with hooks and utilities forming the bottom layer, gradient attribution later, amd an API layer for implementing existing attribution methods and supporting new methods.

The authors show comparisons with existing packages for LLM-based TDA methods, and compare throughout and memory scaling with models ranging up to 110B in size (across 4 H200 GPUs). They also demonstrate the attribution fidelity comparing to a LOO retraining (Wang et al, 2025b) for models up to 7B in size. Overall the authors claim that dattri-llm is 3.2x faster than the fastest existing libraries on an average. As an application of integrating this into other workflows the authors demonstrate attribution-guided data selection and full-parameter and LoTA settings.

Critical strengths
  • The authors have clearly thought out the architecture and have incorporated substantial support for large-scale TDA (gradient based). In terms of the characteristics, I liked the two gradient representations supported with optional dimensionality reduction and routing between them, and also the fact that this can work seamlessly with existing training pipelines. The hierarchical architecture is also clearly presented. Results demonstrate the claims made by the authors positively.

  • The ease-of-use with new methods (like the demonstrated attribution-guided data selection) is a nice feature that I think will be quite helpful for practitioners.

Critical weaknesses
  • I could not spot any code. The authors can attach it as zip file to preserve anonymity. Code and some notebook examples (even with a tiny model) will be helpful for evaluation.
Questions for authors
  1. Please clarify the point listed in weaknesses.
  2. Can you list what the scientific contributions are (besides the library itself) somewhere? Is section 5 a completely new contribution not reported before?
  3. What would it take to bring in methods that go beyond gradient approaches? The authors can comment on it somewhere.
  4. Minor: The second sentence in AI use statement needs to be revised. It looks mangled.
3 comments

We thank the reviewer for the constructive feedback. We have released the anonymous code and added an explicit contribution statement.

1. [W1, Q1] Code availability and runnable examples.

The source code is now available at https://anonymous.4open.science/r/dattri-llm-4CFC/. The repository includes the library implementation, experimental scripts, unit tests, and runnable examples. In particular, instructions for reproducing the paper’s experiments are provided in experiments/REPRODUCE.md.

2. [Q2] Scientific contributions and what Section 5 infers.

We have added an explicit contribution list to the introduction. Our contributions are: (i) a unified systems design for LLM-scale attribution, combining exact gradient representations, analytical FLOP-aware routing, non-invasive training integration, and reusable interfaces for methods and applications; and (ii) empirical validation of its efficiency, scalability to 110B-parameter models, fidelity–cost trade-offs, and downstream applicability.

To answer the question about Section 5 directly: it reproduces the existing data selection methods and results of Hu et al. (2026). We do not claim the underlying selection method or its effectiveness as a new finding. The contribution of this experiment is demonstrating that one can easily implement a gradient-based data selection method through a generic callback attached to an otherwise unmodified training loop, and the implementation reproduces the results. Section 5 states this explicitly.

3. [Q3] Extending beyond gradient-based attribution.

The library deliberately focuses on gradient-based attribution, whose methods share costly operations such as per-example gradient capture, projection, preconditioning, and pairwise scoring. Most efficient training data attribution methods fall in this category, and our Section 4.2 further illustrates the useful fidelity–efficiency trade-offs available within this family.

Supporting retraining-based approaches, such as Data Shapley (Ghorbani & Zou, 2019), would additionally require case-by-case implementation, as these operations fall outside the current gradient-stream abstraction. Moreover, such methods are already supported by general-purpose attribution and valuation libraries, including pyDVL (TransferLab Team, 2024), OpenDataVal (Jiang et al., 2023) or dattri (Deng et al., 2024). Our focus is the complementary infrastructure needed for scalable gradient-based attribution at LLM scale.

4. [Q4] AI-use statement.

The original AI-use statement was filled in directly using the provided template, which reads awkward. We have corrected the malformed sentence. Thank you for pointing this out.

References

Junwei Deng, Ting-Wei Li, Shiyuan Zhang, Shixuan Liu, Yijun Pan, Hao Huang, Xinhe Wang, Pingbang Hu, Xingjian Zhang, and Jiaqi W. Ma. dattri: A library for efficient data attribution. In Advances in Neural Information Processing Systems, volume 37, pp. 136763–136781, 2024.

Amirata Ghorbani and James Zou. Data Shapley: Equitable valuation of data for machine learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2242–2251. PMLR, 2019.

Pingbang Hu, Xueshen Liu, Z. Morley Mao, and Jiaqi W. Ma. Dr. post-training: A data regularization perspective on LLM post-training. arXiv preprint arXiv:2605.07063, 2026.

Kevin Fu Jiang, Weixin Liang, James Zou, and Yongchan Kwon. OpenDataVal: A unified benchmark for data valuation. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 36, pp. 28624–28647, 2023.

TransferLab Team. pyDVL: The Python data valuation library. https://pydvl.org/stable/, 2024. Version 0.9.2.

Reviewer_KPPGOpenReview ↗

thanks

Thanks, I saw the authors responses to my comments as well as the comments of other co-reviewers.

The only further question I have is if the code developed by the authors will be released openly by the time of the conference.

We thank the reviewer for the careful review. Yes, as promised in our reproducibility statement, we will publicly release the code by the time of the conference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.