- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies cross-epoch interactions in training data attribution, focusing on how replacing a training example affects the final model. The authors point out that the same example is often used across multiple epochs. Its contributions from different visits can propagate through later training steps and either reinforce or cancel each other. Therefore, considering each visit separately may not accurately capture the effect of data replacement.
To study this problem, the authors introduce the finite-horizon stability dimension (FHSD) to measure how data replacement affects the final model parameters. They theoretically decompose this measure into interactions between training visits. Based on this analysis, they propose PathGram, which uses a shared low-dimensional Hessian subspace to track the effects of individual visits and predict loss changes without retraining the model for each candidate replacement. They also introduce a refinement method using four additional gradient measurements to improve prediction accuracy.
Experimental results show that cross-epoch interactions account for a large portion of the linear response energy in the randomly reshuffled BERT experiments. PathGram achieves NMAEs of 0.071–0.106 in the main experiments and correctly identifies the best replacement candidates in several settings. With the four-gradient refinement, NMAE decreases to 0.0027 on Swin-T and 0.0023 on BERT-tiny. The paper also includes ablation studies, error analysis, and computational cost comparisons to evaluate the method.
Critical strengths
S1. The paper addresses a meaningful research problem with a clear focus. It studies how the effects of repeated visits to the same training example propagate through training and reinforce or cancel each other across epochs. This perspective helps improve our understanding of how training data affects the final model and provides useful insights for more accurate data attribution.
S2. The theoretical analysis and method design are well connected, and the method has a clear motivation and a well-structured design. The authors introduce FHSD to analyze interactions between training visits and develop PathGram to track their effects using a shared low-dimensional Hessian subspace, reducing the cost of evaluating multiple candidate replacements. They also propose a four-gradient refinement based on their error analysis, which further improves prediction accuracy.
S3. The experiments provide empirical support. The authors evaluate their method on BERT and Swin-T models, using actual retraining results to assess prediction accuracy. The experiments include comparisons with existing attribution methods, analyses of cross-epoch interactions, component ablations, error analysis, and computational cost comparisons. In particular, the four-gradient refinement substantially reduces prediction errors, providing strong evidence of its effectiveness.
Critical weaknesses
W1. The theoretical novelty could be better clarified. The paper introduces FHSD to analyze interactions between training visits and derives an epoch-dependent upper bound. However, these results mainly rely on quadratic expansion and standard norm inequalities. While the analysis provides a useful perspective, its novelty and practical implications compared with existing trajectory-based attribution and stability analyses could be explained more clearly.
W2. The contributions of PathGram and its refinement are not fully separated. PathGram achieves good prediction accuracy, but its improvement over SOURCE is relatively small in the main experiments, with similar results in sign prediction and candidate selection. Meanwhile, the largest error reduction comes from the four-gradient refinement. It would be helpful to clarify the contributions of these two components and whether the refinement could also improve other trajectory-based methods.
W3. The experimental settings are relatively limited. Although the paper evaluates different model architectures and includes some larger-scale experiments, the main evaluations focus on small training datasets and controlled adaptation settings. It remains unclear how well the method scales to larger datasets, longer training runs, or more complex fine-tuning tasks, both in prediction accuracy and computational cost.
W4. The presentation of experimental results could be improved. Several useful results and figures are placed in the appendix. As a minor suggestion for improving readability, Sections 6.5–6.7 could be combined into a shorter subsection to leave more space for key figures or tables in the main paper. This is only a suggestion for future revision and does not need to be addressed during the rebuttal.
Questions for authors
Q1. Could the authors further clarify the novelty and practical implications of FHSD compared with existing trajectory-based attribution and stability analyses? (see W1)
Q2. How much of the improvement comes from PathGram itself versus the four-gradient refinement, and could the refinement also benefit other trajectory-based methods? (see W2)
Q3. Is there any additional evidence, even on a small scale, showing how the method performs on larger datasets or longer training runs? (see W3)
2 comments
Thanks for your response.
Thank you for your detailed responses and additional experiments, which have addressed several of my concerns.
I agree that the theoretical novelty, particularly regarding FHSD, could be clearer. Still, I find the analysis of cross-epoch interactions (PathGram and M4) and the improvements in accuracy and efficiency valuable. Overall, I will keep my weakly positive rating.
Thank you for your thoughtful review. We separate PathGram's contribution from M4's with a matched SOURCE+M4 comparison: both methods improve, while corrected PathGram retains an accuracy and cost advantage. We then relate this result to FHSD's visit-alignment characterization and the existing larger-data and longer-run evidence.
Q1: What does FHSD add to trajectory attribution and stability analysis?
FHSD quantifies visit reinforcement and cancellation and gives a sharp bound on their amplification across epochs. Trajectory methods, including SOURCE, already propagate perturbations through training. FHSD asks an additional question: how much of the squared terminal response comes from individual visits, and how much comes from their alignment? If two propagated visit effects are a and b, their combined energy is ∥a∥2+∥b∥2+2a⊤b. The last term distinguishes reinforcement from cancellation even when the individual energies are identical.
The practical consequence is that a sum of individual-visit energies can substantially misrepresent replacement sensitivity. Our decomposition identifies every visit-pair term under the realized training order. With at most E visits per example, total energy is at most E times the diagonal-only energy, and this factor is attainable even with strictly contractive updates. The contribution is this sharp characterization of the attribution quantity and its consequences, using standard algebraic tools. This distinction matters empirically: cross-epoch terms account for about 96% and 94% of linear-response energy in the reshuffled BERT experiments. Preserving visit alignment also reduces squared-loss prediction NMAE from 0.960 to 0.161 on BERT-tiny and from 0.947 to 0.129 on BERT-base, compared with sign randomization that preserves individual-visit energies.
Q2: How much comes from PathGram, and does M4 help SOURCE?
M4 supplies most of the error reduction in both methods, while PathGram+M4 has 89% lower mean error than SOURCE+M4 in nine matched Swin-T configurations. We evaluated three initialization/candidate seeds at three step sizes, with 16 replacements per configuration. Both methods use the same four additional full-gradient measurements per candidate and the same terminal loss readout. SOURCE uses four stationary segments with exact Hessian-vector products. The table reports mean normalized absolute error, with lower values better.
As the table shows, M4 reduces SOURCE's error by 79% and PathGram's by 97%. After correction, PathGram's error is 89% lower. Thus M4 is a transferable residual correction, while the shared PathGram response provides additional accuracy in this controlled comparison. All four methods select the best replacement in every configuration, so the added benefit concerns magnitude accuracy and computation.
The cost comparison also favors PathGram under the common budget ceiling. Both methods receive the same per-configuration Hessian-vector-product allowance and four-gradient correction allowance. PathGram+M4 uses 58–76 Hessian-vector products, compared with 2,746–4,268 for SOURCE+M4, and is 14.6 times faster by the median paired timing ratio. These timings include each method's curvature preparation, response, correction, and terminal readout. They describe this exact-Hessian SOURCE implementation and the fixed four-segment setting.
Q3: What evidence supports larger datasets and longer runs?
The enlarged text study preserves accurate replacement selection at 1,000 training examples across four BERT sizes and three seeds per model. Mean NMAE is 0.060, 0.028, 0.160, and 0.079 for tiny, mini, medium, and base, respectively. Eight of the twelve runs recover all six best replacements and the remaining four recover five, with maximum normalized selection regret 0.014. Estimator cost is 1.2%–5.8% of matched neighboring retraining. These measurements concern final-attention LoRA adaptation, which makes the trainable parameter space explicit.
Accuracy also persists as the evaluated Swin-T run grows from 20 to 80 updates. On Swin-T, PathGram's quadratic-readout NMAE is 0.038, 0.056, 0.065, and 0.071 at 20, 40, 60, and 80 updates. Each stopping point has its own evaluation derivatives and full retraining outcomes. A separate matched BERT scale panel extends to 320 updates. These results support finite-horizon adapter attribution across the measured sizes and horizons.
Thank you also for the presentation suggestion. Bringing the principal accuracy–cost comparison alongside the visit-interaction evidence would make the experimental argument easier to follow.