- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper extends the In-Run Data Shapley attribution method from SGD to Adam. Under SGD, In-Run Data Shapley credits a training example, at each step it appears in, with the dot product between its gradient and the validation loss gradient. To apply the same per-step Shapley decomposition under Adam, the authors linearize the Adam update at the batch gradient with the previous step's moment estimates held fixed. This yields an approximate Adam Shapley value that, up to a momentum-dependent correction, is the same dot product as under SGD with the validation loss gradient divided coordinate-wise by Adam's running gradient scale. It can therefore still be obtained without explicitly computing per-example gradients. Experimentally, the authors report that the SGD score is a poor proxy on Adam runs, and that their score gives better attributions.
Critical strengths
The approach is a sound way to obtain an approximate in-run Shapley value under Adam. The problem is relevant given that Adam is a common optimizer choice in deep learning practice and the SGD-based score is not designed for it.
Critical weaknesses
The operation the method arrives at is what LESS [1, Section 3.1] and TrackStar [2, Section 3] already do: dividing gradients coordinate-wise by the square root of the optimizer’s second-moment estimate before the dot product. Neither work is cited. That an Adam-aware score matches or beats the SGD score on Adam-trained models was also shown by LESS [1, Table 9]. The momentum-dependent term the submission adds on top of LESS’s operation is never ablated.
Contribution 1 is overstated. It is unsurprising that SGD-based attributions are not good for Adam-trained models: attribution is a property of the training algorithm used. Also, the experiment supporting this claim (Figure 1) lacks the basic control of comparing attributions across seeds under the same optimizer. Those can be weakly correlated too, so the near-zero correlation reported may just be noise.
Contribution 4 is overstated too. The near-perfect fidelity only shows that the proposed linearization is faithful to the per-step Adam Shapley values computed without it. It does not show that the method produces good attributions, which the paper never evaluates the standard way, against retraining-based attributions, except on a toy task.
What seems novel is the derivation through a linearization, which also lets the score be obtained without explicitly computing per-example gradients.
[1] Xia et al. LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML, 2024.
[2] Chang et al. Scalable Influence and Fact Tracing for Large Language Model Pretraining. ICLR, 2025.
Questions for authors
- Which values of Adam's beta_1 and beta_2 were used in each experiment?
- Figure 1 shows about thirty points, while Appendix B says 10,000 samples were tracked to produce the figure. What does each point represent?