acceptodds
Under review as a conference paper at ICLR 2027

How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines

Abstract

Trajectory-based data attribution methods estimate the influence of training samples on model outputs by tracing through the training trajectory. They are increasingly used in applications such as data selection, data valuation, and model diagnosis, but there is a lack of comprehensive error analysis of these methods, raising concerns about method faithfulness and hindering reliable deployment. In this work, we provide the first systematic analysis of error sources in trajectory-based data attribution, together with remedies to mitigate them and practical guidelines for downstream use. We organize the error sources into three categories: config-level, algorithm-level, and system-level. Building on this taxonomy, we make three contributions. First, we identify optimizer mismatch as the dominant config-level error: most existing methods derive trajectory-based attribution under the assumption of SGD, even when the model is trained with the widely used AdamW optimizer. To mitigate this error, we propose AdamW-influence, which faithfully accounts for AdamW's optimization dynamics and improves the Spearman correlation between estimated and ground-truth influence by 10% to over 300% across settings spanning MLP, CNN, GPT-2, and Llama 3.2-1B. Second, after correcting optimizer mismatch, we show that the remaining algorithm-level error due to the first-order Taylor approximation is primarily governed by two factors: the learning rate and the remaining trajectory length from a sample's training step to the final model. The estimated influence is more faithful at smaller learning rates and over shorter trajectories. We further derive a closed-form proxy for per-sample approximation error that can be evaluated along the original training trajectory without retraining. Third, we translate these insights into practical guidelines for downstream attribution-guided data selection. Our empirical results indicate that the attribution faithfulness plays a critical role in the data selection effectiveness: the proposed AdamW-influence significantly outperforms the SGD-based counterpart; while online data selection, where the estimated influence accounts for only a short training trajectory to guide each local update, often outperforms offline data selection, where the estimated influence accounts for the full training trajectory of a reference run. Together, our results provide a systematic framework for diagnosing trajectory-based attribution error and using these methods more reliably in downstream applications.

Primary area:
interpretability and explainable AI
Keywords:
data attribution, data selection

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_GEKUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Trajectory-based data attribution methods approximate the influence of a training sample at a specific iteration on final model outputs by tracing through the training trajectory from that iteration. This paper decomposes error sources in trajectory-based data attribution into 3 categories: system-level, config-level, and algorithm-level. The config-level error is mainly due to optimizer mismatch, where previous works leverage the SGD assumption. To deal with this, this paper proposes AdamW-influence, which faithfully accounts for AdamW's optimization dynamics and thus improves the Spearman correlation. In addition, this paper shows that the algorithm-level error is due to first-order Taylor approximation governed by the learning rate and the remaining trajectory length. This paper also derives a proxy for per-sample error. Then, these insights are translated into practical guidelines for data selection in both offline and online settings.

Critical strengths
  • This paper introduces AdamW-influence, which faithfully accounts for AdamW's optimization dynamics during estimating data attribution and thus improves the Spearman correlation between estimated attributions and the ground-truth values. The improvement in correlation is up to 300%.
  • This paper illustrates that algorithm-level error mainly arises from the first-order Taylor expansion that AdamW-influence leverages to estimate attribution. The magnitude of this update-estimation error is governed by both learning rate and trajectory length. In addition, a closed-form error proxy is introduced to approximate per-sample error along the original training trajectory without retraining.
  • The proposed offline and online AdamW-influence outperform SGD-influence significantly.
Critical weaknesses
  • The proposed AdamW-influence approach can approximate data attribution for models trained with the AdamW optimizer but may not extend to other optimization methods.
  • The method requires a substantial computational cost, particularly for large pretrained models. To reduce this cost, SGD-influence uses efficient random projection, while AdamW-influence employs an ensemble of random masking. This masking strategy raises concerns because the estimation error might accumulate with first-order Taylor approximations and random masking. In this paper, the authors apply only 256 dimensions per mask for the Alpaca + Llama 3.2-1B model, which ignores the contribution of massive unselected model parameters.
Questions for authors

Please address my concerns in Weaknesses and:

  1. Could the authors provide a detail description of how online AdamW-influence work?
  2. What is the performance of online AdamW-influence compared to GREATS [1]?

[1] Jiachen T Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. Greats: Online selection of high-quality data for llm training in every iteration. Advances in Neural Information Processing Systems, 37:131197–131223, 2024

4 comments

We sincerely thank the reviewer for the positive feedback, and for recognizing the contributions of AdamW-influence, our analysis of approximation errors, and the improvements in downstream data selection.

W1: AdamW-influence on other optimizers

We thank the reviewer for raising this concern. Our key message is that attribution methods should faithfully match the optimizer used in the actual training process, rather than that AdamW-influence is a universal solution. As demonstrated in Appendix F.1 (Table 4), when models are trained with SGD, SGD-influence substantially outperforms AdamW-influence, further validating the importance of optimizer alignment. We focus on AdamW because it is widely adopted in modern deep learning and presents non-trivial challenges in deriving how sample perturbations propagate through its moment states.

W2: Random masking’s information loss

We thank the reviewer for raising this concern. We would like to clarify why random masking is necessary for AdamW-influence and how we mitigate the resulting information loss.

First, random masking is necessary to preserve AdamW's coordinate-wise optimization dynamics. Unlike SGD-influence, which supports random projection, AdamW-influence explicitly propagates perturbations through coordinate-wise adaptive scaling and optimizer moment states. Random projection mixes parameter coordinates and therefore breaks these dynamics. In contrast, random masking preserves the original coordinates, allowing faithful propagation within the retained dimensions (Appendix D.1).

Second, random masking substantially reduces computational cost. Retaining SS out of pp parameters reduces the complexity from O(3Tbp2)O(3Tbp^2) to O(3TbS2)O(3TbS^2), providing a reduction in computation.

Third, we use an ensemble of independent masks to mitigate information loss and variance. Although each mask retains only 256 dimensions for Llama 3.2-1B, different masks capture complementary subsets of parameters, and their attribution estimates are aggregated. We acknowledge that masking introduces additional approximation error, but our fidelity evaluation against TSLOO ground truth already measures its combined effect with first-order approximation.

Q1: How does online AdamW-influence work?

We thank the reviewer for the question. Online AdamW-influence performs data selection at each training step by sampling a candidate batch, estimating each candidate's influence on validation loss over a K-step look-ahead horizon, and selecting the most beneficial examples for the model update. The influence is estimated by propagating sample perturbations through AdamW's parameter and optimizer-state dynamics. Here, K controls how many subsequent training steps are considered, balancing longer-term influence against accumulated approximation error. We have added a more detailed description of this procedure and the experimental setup in Section 5.2.

Q2: GREATS comparison

We thank the reviewer for the question. Under our unified framework, GREATS corresponds to the immediate-effect setting. We therefore evaluate a GREATS-like baseline to isolate the benefit of look-ahead. This baseline uses K=0 without implementing the greedy selection procedure of GREATS. Online AdamW-influence with a tuned look-ahead horizon achieves better performance on both MNIST+MLP and Tulu3+Llama 3.2-1B.

MethodMNIST+MLP (Error Rate ↓)Tulu3+Llama3.2 (Perplexity ↓)
Standard0.1045 ± 0.00172.2466 ± 0.0461
LESS (same bs as online)0.1082 ± 0.00152.0677 ± 0.0302
Offline AdamW-influence (same bs as online)0.1057 ± 0.00201.9741 ± 0.0251
Online SGD-influence (tuned K)0.0952 ± 0.0017 (K=10)2.0763 ± 0.0349 (K=2)
Online AdamW-influence (K=0, GREATS-like)0.0965 ± 0.00161.8554 ± 0.0163
Online AdamW-influence (tuned K)0.0935 ± 0.0011 (K=10)1.8470 ± 0.0230 (K=2)

Thank you for providing additional experimental results and clarifications. Most of my concerns have been adequately addressed; thus, I maintain my rating leaning toward acceptance.

We thank the reviewer for the positive feedback and suggestions!

Reviewer_8eMROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

In this paper the authors perform a systematic analysis of error modes in trajectory-based data attribution, and propose approaches to mitigate those errors thereby improving the faithfulness of trajectory-based data attribution. They categorize errors as: config-level (mostly due to optimizer mismatch -- approaches’ theoretical grounding in SGD while other optimizers, such as AdamW, are actually used in practice), algorithm-level (error arising from using e.g. first-order Taylor expansions to approximate) and system-level (errors introduced by floating-point precision or GPU non-determinism).

To address this, they propose an alternative AdamW-aware estimator (AdamW-influence) and show that it improves Spearman correlation with leave-one-out retraining compared to the SGD-based approach, across architectures (MLP, CNN, GPT-2, Llama 3.2-1B). They argue that the remaining algorithm-level error is governed by learning rate and trajectory length, and derive a per-sample error proxy that can be computed without retraining. They bring all of this together into a practical framework for using trajectory-based attribution for data selection that combines AdamW-influence with K-step lookahead, with experiments on MNIST (MLP and CNN, test error) and Llama 3.2-1B SFT on subsets of Alpaca and Tulu3 (perplexity on SAMSum and tydiqa), finding that online selection with a small-to-moderate K outperforms offline selection and that the best K shrinks as a function of learning rate.

Critical strengths
  • The paper makes seemingly practical empirical and theoretical advances towards faithful, yet computationally tractable, approaches to data attribution. Understanding the relationship between data, optimization and end-task performance on real data is both theoretically interesting, and has practical implications for effective data selection.
  • Although a bit dense, the paper is reasonably well-written/presented and easy to follow.
  • I like the unified conceptual framework for data selection, and anchoring of prior work within that conceptual understanding: The paper not only presents new findings, but also provides a useful unifying overview of the area.
Critical weaknesses
  • Experimental results are lacking appropriate baselines. With respect to fidelity (Table 1), AdamW-influence is only compared to SGD-Influence, but that’s a relatively weak baseline; there exist other optimizer-aware approaches such as SOURCE (Bae et al. 2024) and Ding et al. (2026) (extends In-Run Data Shapley (Wang et al. 2025) to Adam). Some of these are described in related work in the appendix, but as a reader it’s not clear to me empirically which I should use and why. With respect to data selection, GREATS is described as equivalent to K=0 in the proposed framework, but not run and compared. I’d be fine with just K=0 (not including greedy selection / the full GREATS approach), but without it it’s not clear that that the paper’s claim to use lookahead is supported. I’d also like to see a control for the impact of different batch size vs. different examples for the online vs. offline approaches. It’s not clear how much of the reported difference is due to the change in batch size, especially since online vs offline is where we see the most notable difference (and is also where the batch size is different.)
  • Lack of practical experiments at scale: While there are findings applying the proposed approach for data selection, the settings are not realistic (number of examples, realistic trajectory lengths) and performance is only evaluated in terms of perplexity, which may not correspond to actual end-task performance.
  • I like the idea of the error taxonomy, but as is noted in the intro (and is also evident in the analysis of MAGIC compared to AdamW-influence), the “algorithm-level” error arises from a combination of linearization and approximating curvature, as well as dimensionality reduction. This is particularly the case for larger scale applications (e.g. LLMs). The MAGIC vs AdamW-influence analysis is nice and shows the error attributable to curvature, but it would be good to similarly see error attributable to the other components in the appropriate setting. I’d like to understand the relative roles of all three components in contributing to error, in particular because Sec. 4 / Proposition 2 seem to attribute the entire algorithm-level term to linearization, but Fig. 2 and Fig. 4 measure the total update error including curvature approximation and masking. Without separating it out, I can’t tell how critical the lr / trajectory length actually are to overall error, versus just linearization.

Smaller writing/presentation things:

  • Not enough description of empirical experimental setup in the paper to properly assess/understand results (Figure 5). In my opinion, 5.2 is a pretty critical section and a description of the basic experimental setup is lacking and should be moved up into the main paper from appendix.
  • Related work relegated to appendix. I like the unifying framework aspect of the paper, and would like to see the authors lean into that more. For example, the comparison/anchoring in MAGIC is informative and helpful! Putting related work in the appendix does a disservice to that potential strength.
  • Include a little more info on the computational overhead of AdamW-influence within the paper. Literally the last sentence of the analysis in the appendix (specifically the 3x amount) should be included in the main paper. This is relevant for understanding tradeoffs of the proposed approach.
Questions for authors
  • Will you release the code?
5 comments

We sincerely thank the reviewer for the thoughtful feedback, and for recognizing both the theoretical and practical contributions of our work, as well as the value of our unified framework for data selection.

W1 (1): Fidelity comparison with SOURCE/In-Run Data Shapley Adam

We thank the reviewer for this suggestion. To address this concern, we extend our fidelity evaluation to include two additional optimizer-aware baselines: SOURCE [1] and In-Run Adam [2]. As shown in the table below, AdamW-influence achieves the highest Spearman correlation against TSLOO ground truth across all six settings, covering both MNIST+MLP and Alpaca+Llama 3.2-1B.

These improvements can be explained by differences in how the methods model training dynamics. SOURCE treats the optimizer preconditioner as fixed during differentiation (Appendix I). In-Run Adam accounts for Adam's adaptive updates but relies on local utility estimates without propagating perturbations through subsequent training steps. In contrast, AdamW-influence jointly propagates perturbations through model parameters and both optimizer moment states across the training trajectory, providing a more faithful approximation of the TSLOO target. The results have also been updated on PDF in Appendix F.1.

SettingLRAdamW-influenceSGD-influenceSOURCE(optimizer-aware)In-Run Adam
MNIST+MLP1e-30.205±0.0110.075±0.0090.134±0.0080.062±0.009
1e-40.294±0.0140.242±0.0160.249±0.0150.223±0.014
1e-50.786±0.0130.715±0.0120.734±0.0130.715±0.014
Alpaca+Llama 3.26e-60.276±0.0470.156±0.0190.220±0.0380.240±0.045
2e-60.342±0.0360.197±0.0070.272±0.0370.331±0.043
5e-70.665±0.0130.330±0.0170.515±0.0220.575±0.020

[1] Bae, J., Lin, W., Lorraine, J., & Grosse, R. (2024). Training data attribution via approximate unrolling. Advances in Neural Information Processing Systems, 37, 66647-66686. [2] Ding, M., Zhang, Z., Wang, D., & Hu, L. (2026). In-Run Data Shapley for Adam Optimizer. arXiv preprint arXiv:2602.00329.

W1 (2): Data selection with K=0 as baseline and align the training batch size

First, we include K=0 (GREATS-like, without greedy selection) to isolate the benefit of look-ahead. Across both MNIST+MLP and Tulu3+Llama 3.2-1B, online AdamW-influence with a tuned K outperforms K=0, providing empirical evidence that incorporating future training dynamics can improve data selection.

Second, we control for the difference in batch size between online and offline methods. Specifically, we rerun LESS and offline AdamW-influence using the same training batch size as the online methods. Even with matched batch sizes, online AdamW-influence consistently outperforms both offline baselines across the two settings. These results indicate that the observed improvements cannot be attributed to batch-size differences.

MethodMNIST+MLP (Error Rate ↓)Tulu3+Llama3.2 (Perplexity ↓)
Standard0.1045 ± 0.00172.2466 ± 0.0461
LESS (same bs as online)0.1082 ± 0.00152.0677 ± 0.0302
Offline AdamW-influence (same bs as online)0.1057 ± 0.00201.9741 ± 0.0251
Online SGD-influence (tuned K)0.0952 ± 0.0017 (K=10)2.0763 ± 0.0349 (K=2)
Online AdamW-influence (K=0, GREATS-like)0.0965 ± 0.00161.8554 ± 0.0163
Online AdamW-influence (tuned K)0.0935 ± 0.0011 (K=10)1.8470 ± 0.0230 (K=2)

W2: realistic settings and actual end-task performance

We thank the reviewer for raising these concerns.

Regarding training scale and trajectory length of data selection, both our MNIST classification and Llama 3.2-1B SFT experiments train the models to convergence, which reflects a realistic trajectory length. We acknowledge that the number of training examples is constrained by our computational budget.

Regarding evaluation metrics, our primary goal is to understand errors in data attribution methods. We therefore use perplexity, a monotonic transformation of the cross-entropy loss used in our attribution formulation, to avoid introducing additional confounding factors arising from discrepancies between the attribution objective and evaluation metric. Nevertheless, we also report downstream task metrics for reference.

To further assess downstream task performance, we additionally evaluate the fine-tuned Llama 3.2-1B models on TyDiQA (F1) and SAMSum (ROUGE-L), using the full test sets over 10 random seeds. As shown in the additional results, online AdamW-influence achieves the best F1 on TyDiQA and competitive ROUGE-L on SAMSum. Overall, these downstream results are broadly consistent with our perplexity findings.

MethodTyDiQA F1 (%) ↑SAMSum ROUGE-L (%) ↑
Standard32.55 ± 0.5121.98 ± 0.02
LESS32.68 ± 0.4422.10 ± 0.04
Online SGD-influence33.39 ± 0.3922.08 ± 0.05
Offline AdamW-influence34.53 ± 0.4522.28 ± 0.03
Online AdamW-influence36.79 ± 1.0222.24 ± 0.04

W3: The confounders in algorithm-level term

We thank the reviewer for this suggestion. In the revised PDF we separate the three contributions (new Appendix F.3, referenced at the end of Sec. 4.1). For MNIST+MLP and Llama-3.2-1B, we run the AdamW-influence recurrence without masking (all parameters for the MLP; all linear layers except the output head for Llama), once with the empirical Fisher and once with exact Hessian–vector products, keeping everything else fixed. This splits the blue term into (i) random masking, (ii) empirical Fisher vs. exact Hessian, and (iii) the first-order linearization error that remains with the exact Hessian.

The curvature approximation accounts for only a small share of the error. Masking adds a moderate, mostly early-training contribution on Llama. In both settings, the dominant component is the linearization error that persists with the exact Hessian. We have also clarified in Sec. 4 that Fig. 2 and Fig. 4 measure the total update-estimation error, and that Appendix F.3 shows this error is dominated by linearization.

W4: Description of experimental setup in sec 5.2

We have expanded Section 5.2 to include a concise description of the experimental procedure, covering the online data-selection process, evaluated models and datasets, look-ahead horizon selection, and evaluation metrics.

W5: More related work and discussion in main body

We thank the reviewer for this suggestion. We have added a new paragraph at the end of Section 3.2 to bring these connections into the main paper. Specifically, we summarize our empirical comparisons with optimizer-aware baselines (SOURCE and In-Run Adam), explain their methodological differences, and highlight the equivalence between AdamW-influence and MAGIC under the exact Hessian and full parameter space, alongside the computational advantages of our approach. We also provide pointers to related works (Appendix A) and F.1–F.2 for further detailed results.

W6: Explicit computational cost claim in main body

We thank the reviewer for this suggestion. We have added an explicit computational cost comparison immediately after Proposition 1 in Section 3.1.

Q1: Code release

Yes, we will publicly release our code. Most of the implementation code has already been included in the supplementary materials.

Reviewer_nBGKOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper systematically investigates the faithfulness of trajectory-based data attribution by identifying three major error sources: configuration-level, algorithm-level, and system-level errors. To address optimizer mismatch, the authors propose AdamW-influence, which explicitly accounts for the dynamics of AdamW's moment states. The paper further analyzes approximation errors and introduces an error proxy to characterize the effects of learning rate and propagation horizon. Finally, it applies these insights to data selection, demonstrating improvements over existing baselines.

Critical strengths
  1. The paper provides a systematic analysis of error sources in trajectory-based data attribution, distinguishing configuration-level, algorithm-level, and system-level errors.

  2. It proposes AdamW-influence, which explicitly propagates perturbations through both model parameters and optimizer moment states, improving attribution faithfulness while maintaining linear complexity in the number of training steps.

  3. The paper theoretically analyzes approximation errors and introduces an error proxy to characterize the effects of learning rate and propagation horizon.

Critical weaknesses

None

Questions for authors

Could the authors provide more theoretical insights into why the proposed AdamW-influence is expected to outperform other single-pass trajectory-based attribution methods (in Related Work), particularly those designed for modern optimizers?

3 comments

We sincerely thank the reviewer for the positive and constructive feedback, and for recognizing the contributions of our systematic error analysis, AdamW-influence, and theoretical characterization of approximation errors.

Q1: Insights into why AdamW-influence is expected to outperform others

We thank the reviewer for this question. The key advantage of AdamW-influence is that it explicitly models how sample perturbations propagate through both the optimizer states and subsequent training steps. We clarify its theoretical differences from three representative methods below.

AdamW-influence vs. SOURCE (Adam-aware extension, detailed explanation in Appendix I). SOURCE incorporates adaptive optimization through a preconditioning matrix but treats this matrix as fixed during differentiation. Consequently, it does not capture how removing a training sample changes AdamW's first- and second-moment states and, in turn, influences subsequent parameter updates.

AdamW-influence vs. In-Run Adam [1]. In-Run Adam accounts for Adam's adaptive updates but estimates sample utility locally, without propagating perturbations through subsequent training steps. AdamW-influence instead tracks these perturbations along the remaining training trajectory, capturing their effects on the final model.

AdamW-influence vs. TracIn. TracIn estimates influence by accumulating gradient similarities across training checkpoints. AdamW-influence instead accounts for how a sample's effect evolves along the remaining training trajectory up to the final model, capturing temporal dependencies and optimizer dynamics omitted by TracIn.

[1] Ding, M., Zhang, Z., Wang, D., & Hu, L. (2026). In-Run Data Shapley for Adam Optimizer. arXiv preprint arXiv:2602.00329.

Thank the author for the clarification, which fully addresses my concerns.

We sincerely thank the reviewer for the positive feedback and are glad that our response has addressed the concerns!

Reviewer_3b15OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines errors in trajectory-based data attribution against trajectory-specific leave-one-out (TSLOO) ground truth. It proposes a three-level error taxonomy, introduces AdamW-influence to address optimizer mismatch, analyzes first-order approximation errors, and develops a KK-step look-ahead framework for data selection. Overall, it links attribution fidelity to downstream selection performance, highlighting the practical implications of reliable data attribution.

Critical strengths
  1. The paper addresses an important and relatively underexplored problem concerning the reliability of trajectory-based data attribution.

  2. The error taxonomy provides a useful perspective for understanding different sources of attribution inaccuracies.

  3. AdamW-influence is technically motivated and provides a principled extension to account for AdamW's optimizer dynamics.

  4. The paper evaluates both attribution fidelity and downstream data-selection performance, making the findings practically relevan

Overall, I consider this a promising and technically solid work, with meaningful contributions to understanding and improving trajectory-based data attribution. Nevertheless, I have concerns about the empirical generality and practical scalability of the proposed framework.

Critical weaknesses
  1. The novelty and generality of the proposed error taxonomy could be better established, as the current analysis primarily focuses on optimizer mismatch and first-order approximation error.

  2. Despite maintaining the same asymptotic complexity as SGD-influence, AdamW-influence introduces approximately three times the computational and memory overhead. Its reliance on random masking and multiple ensembles also raises concerns about scalability and information loss.

  3. The proposed error proxy is validated only on MLP and CNN models trained on MNIST. Further evaluation on language models is needed to establish its generality.

  4. The empirical support for the proposed (K)-step data-selection guidelines is somewhat limited. In particular, the relationship between the optimal horizon and learning rate should be investigated across more diverse architectures and datasets.

  5. The fidelity and data-selection experiments use relatively constrained training settings. More extensive evaluations on larger models and realistic data-selection scenarios would strengthen the paper's claims.

Questions for authors

My main concerns remain regarding computational scalability, the generality of the error proxy, and the empirical validation of the proposed data-selection guidelines. Further experiments and clarification of these aspects would strengthen the work.

Please response weakness point by point.

2 comments

We thank the reviewer for the thoughtful feedback, as well as for recognizing the importance of studying trajectory-based data attribution reliability, the usefulness of our error taxonomy, the technical contributions of AdamW-influence, and the practical relevance of our downstream evaluations. We address the concerns in detail below.

W1: Novelty and generality of error taxonomy

We thank the reviewer for raising this point. Our taxonomy is motivated by the origins of errors in trajectory-based data attribution, which also call for different actions. Specifically, (1) config-level errors arise from mismatches between the attribution formulation and the actual training procedure (e.g., optimizer, learning-rate schedule, and batch composition), and can be addressed by aligning the attribution formulation with the training configuration. (2) Algorithm-level errors arise from computational approximations made for tractability, even when the training configuration is faithfully modeled, and require improving or controlling these approximations. (3) System-level errors arise from numerical precision and hardware nondeterminism, and are largely irreducible by attribution algorithm design.

We focus on optimizer mismatch and first-order approximation error for two reasons. First, they are empirically significant: our quantitative decomposition shows that optimizer mismatch contributes substantially to attribution error, while parameter-update approximation dominates the remaining error after correcting optimizer mismatch. Second, both present nontrivial technical challenges that remain underexplored. Unlike other configuration factors that can be easily aligned by replaying recorded training information, accounting for AdamW requires propagating perturbations through its coupled momentum states and adaptive scaling. First-order approximation errors accumulate along the training trajectory, which can not be easily mitigated by trivial cost-performance trade-off.

W2: Complexity and reliance on random masking / ensemble of AdamW-influence

We thank the reviewer for the comment. We would like to clarify the computational overhead and the role of random masking and ensembles.

First, dimensionality reduction is not unique to AdamW-influence. Efficient implementations of SGD-influence [1] also rely on random projection to reduce computational and memory costs. Due to AdamW's coordinate-wise adaptive scaling, we instead adopt random masking to preserve the optimizer dynamics (Appendix D.1). While AdamW-influence incurs a 3X overhead in working memory and backward-recurrence computation relative to SGD-influence, this does not imply a 3X overhead over full model training (see the second point). Moreover, per-sample embedding storage remains identical, and trajectory caching introduces only a small additional overhead (Table 2).

Second, random masking substantially reduces the absolute computational cost. With SS retained coordinates out of pp parameters, the computational complexity decreases from O(3Tbp2)O(3Tbp^2) to O(3TbS2)O(3TbS^2), yielding a reduction proportional to (S/p)2(S/p)^2. Although the relative 3× factor remains compared to equally compressed SGD-influence, the additional overhead is relatively modest compared with the cost of full model training.

Third, ensembling provides a flexible performance-cost trade-off rather than a necessary requirement. A single mask is sufficient to run AdamW-influence, while multiple masks can mitigate information loss and variance at small retained dimensions. Since masks can be processed sequentially on the same recorded training trajectory, increasing the ensemble size does NOT require additional model training or multiply peak memory usage. It only increases attribution computation time that is also relatively modest compared with full training.

[1] Wang, J. T., Song, D., Zou, J. Y., Mittal, P., & Jia, R. (2025, May). Capturing the temporal dependence of training data influence. In International conference on learning representations (Vol. 2025, pp. 11236-11273).

W3: generality of error proxy

We thank the reviewer for this suggestion. To address this concern, we extend the error proxy evaluation to Llama 3.2-1B, also considering two learning rates. As shown in the updated Figure 4(e–f), the proposed error proxy achieves Spearman correlations of 0.92 and 0.94, respectively. These results demonstrate that the error proxy remains highly effective in capturing relative approximation errors on language models. A brief table is shown below, and an updated Figure 4 is reflected in the revised PDF.

SettingLearning rateSpearman ρ
MLP + MNIST1e-30.85
MLP + MNIST1e-40.87
CNN + MNIST1e-30.83
CNN + MNIST1e-40.89
Llama-3.2-1B + Alpaca2e-60.92
Llama-3.2-1B + Alpaca5e-70.94

W4: Empirical support for the proposed (K)-step

We thank the reviewer for this valuable suggestion. To further examine the generality of our findings, we extend the evaluation of the look-ahead horizon K to Llama 3.2-1B fine-tuned on Tulu3, averaging results over 10 random seeds.

At the standard SFT learning rate, we observe a similar trade-off: evaluation perplexity first decreases and then increases as K grows, with the best performance achieved at K=2. This supports our analysis that a larger horizon captures more future influence but also accumulates greater approximation error.

SettingMetric ↓K=1K=2K=5K=10
Tulu3 + Llama 3.2 (10 seeds)Final Eval PPL1.8686 ± 0.02031.8470 ± 0.02301.8816 ± 0.02491.8792 ± 0.0224

Regarding the relationship between the optimal K and learning rate, our MNIST experiments demonstrate that the preferred horizon increases as the learning rate decreases. Extending this sweep to LLM fine-tuning is computationally expensive, and SFT performance is particularly sensitive to learning-rate choices. We therefore focus on evaluating the horizon trade-off under a practical SFT configuration, while acknowledging that a broader learning-rate sweep would further strengthen the generality of our conclusions.

W5: Extensive evaluations on larger models

We thank the reviewer for this suggestion. While we acknowledge that larger-scale evaluations would further strengthen the empirical evidence, we would like to highlight two considerations.

First, both our attribution fidelity and downstream data-selection experiments already extend to LLM-scale models (Llama 3.2-1B), with data selection evaluated on instruction-tuning datasets including Alpaca and Tulu3. These experiments complement our evaluations on smaller architectures and demonstrate the applicability of our framework beyond conventional classification tasks.

Second, evaluating attribution fidelity requires computationally expensive ground-truth TSLOO calculations, where each training sample requires a separate counterfactual retraining run. This computational burden grows substantially with model size, making comprehensive fidelity evaluation on larger LLMs prohibitively expensive. The fidelity test in the data attribution literature is commonly comparable to or smaller than our study [1] [2].

[1] Bae, J., Lin, W., Lorraine, J., & Grosse, R. (2024). Training data attribution via approximate unrolling. Advances in Neural Information Processing Systems, 37, 66647-66686.

[2] Wang, J. T., Song, D., Zou, J. Y., Mittal, P., & Jia, R. (2025, May). Capturing the temporal dependence of training data influence. In International conference on learning representations (Vol. 2025, pp. 11236-11273).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.