acceptodds
Under review as a conference paper at ICLR 2027

Cross-Epoch Interactions in Training Data Attribution

Abstract

Training data attribution predicts how modifying training data affects a model and helps practitioners choose corrections without retraining each candidate. Influence functions approximate retraining through inverse curvature at a unique optimum, although their predictions can differ substantially from deep-network retraining. Unrolled differentiation avoids this assumption by following the recorded optimization steps. Practical approximations reduce cost by shortening trajectories or averaging within training segments. However, these methods do not explicitly quantify how visits in different epochs combine. Propagated visit contributions can reinforce or cancel in the terminal response. We quantify these interactions through the finite-horizon stability dimension (FHSD), the average squared scaled replacement derivative of the final parameters. This quantity decomposes exactly over visit pairs and, under random reshuffling, is tightly bounded by the epoch count times the sum of single-visit energies. Building on this decomposition, we propose PathGram, which propagates each candidate's visit contributions through a shared leading Hessian eigenspace to predict signed loss changes. On randomly reshuffled BERT-tiny and BERT-base, cross-epoch pairs account for about 96% and 94% of the mean squared linear prediction, respectively. In the primary BERT and Swin-T experiments, PathGram achieves normalized mean absolute errors (NMAEs) of 0.071–0.106 and recovers both full-batch BERT oracle top-six sets. We further refine the approximate neighboring path with four gradient measurements. On independent confirmation runs, this refinement reduces NMAE from 0.1034 to 0.0027 for Swin-T feature adaptation and from 0.0741 to 0.0023 for BERT-tiny LoRA.

Primary area:
learning theory
Keywords:
algorithmic stability, training data attribution, influence functions, training dynamics, data reuse, Hessian-vector products, LoRA fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_sWUzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-epoch interactions in training data attribution, focusing on how replacing a training example affects the final model. The authors point out that the same example is often used across multiple epochs. Its contributions from different visits can propagate through later training steps and either reinforce or cancel each other. Therefore, considering each visit separately may not accurately capture the effect of data replacement.

To study this problem, the authors introduce the finite-horizon stability dimension (FHSD) to measure how data replacement affects the final model parameters. They theoretically decompose this measure into interactions between training visits. Based on this analysis, they propose PathGram, which uses a shared low-dimensional Hessian subspace to track the effects of individual visits and predict loss changes without retraining the model for each candidate replacement. They also introduce a refinement method using four additional gradient measurements to improve prediction accuracy.

Experimental results show that cross-epoch interactions account for a large portion of the linear response energy in the randomly reshuffled BERT experiments. PathGram achieves NMAEs of 0.071–0.106 in the main experiments and correctly identifies the best replacement candidates in several settings. With the four-gradient refinement, NMAE decreases to 0.0027 on Swin-T and 0.0023 on BERT-tiny. The paper also includes ablation studies, error analysis, and computational cost comparisons to evaluate the method.

Critical strengths

S1. The paper addresses a meaningful research problem with a clear focus. It studies how the effects of repeated visits to the same training example propagate through training and reinforce or cancel each other across epochs. This perspective helps improve our understanding of how training data affects the final model and provides useful insights for more accurate data attribution.

S2. The theoretical analysis and method design are well connected, and the method has a clear motivation and a well-structured design. The authors introduce FHSD to analyze interactions between training visits and develop PathGram to track their effects using a shared low-dimensional Hessian subspace, reducing the cost of evaluating multiple candidate replacements. They also propose a four-gradient refinement based on their error analysis, which further improves prediction accuracy.

S3. The experiments provide empirical support. The authors evaluate their method on BERT and Swin-T models, using actual retraining results to assess prediction accuracy. The experiments include comparisons with existing attribution methods, analyses of cross-epoch interactions, component ablations, error analysis, and computational cost comparisons. In particular, the four-gradient refinement substantially reduces prediction errors, providing strong evidence of its effectiveness.

Critical weaknesses

W1. The theoretical novelty could be better clarified. The paper introduces FHSD to analyze interactions between training visits and derives an epoch-dependent upper bound. However, these results mainly rely on quadratic expansion and standard norm inequalities. While the analysis provides a useful perspective, its novelty and practical implications compared with existing trajectory-based attribution and stability analyses could be explained more clearly.

W2. The contributions of PathGram and its refinement are not fully separated. PathGram achieves good prediction accuracy, but its improvement over SOURCE is relatively small in the main experiments, with similar results in sign prediction and candidate selection. Meanwhile, the largest error reduction comes from the four-gradient refinement. It would be helpful to clarify the contributions of these two components and whether the refinement could also improve other trajectory-based methods.

W3. The experimental settings are relatively limited. Although the paper evaluates different model architectures and includes some larger-scale experiments, the main evaluations focus on small training datasets and controlled adaptation settings. It remains unclear how well the method scales to larger datasets, longer training runs, or more complex fine-tuning tasks, both in prediction accuracy and computational cost.

W4. The presentation of experimental results could be improved. Several useful results and figures are placed in the appendix. As a minor suggestion for improving readability, Sections 6.5–6.7 could be combined into a shorter subsection to leave more space for key figures or tables in the main paper. This is only a suggestion for future revision and does not need to be addressed during the rebuttal.

Questions for authors

Q1. Could the authors further clarify the novelty and practical implications of FHSD compared with existing trajectory-based attribution and stability analyses? (see W1)

Q2. How much of the improvement comes from PathGram itself versus the four-gradient refinement, and could the refinement also benefit other trajectory-based methods? (see W2)

Q3. Is there any additional evidence, even on a small scale, showing how the method performs on larger datasets or longer training runs? (see W3)

2 comments

Thank you for your thoughtful review. We separate PathGram's contribution from M4's with a matched SOURCE+M4 comparison: both methods improve, while corrected PathGram retains an accuracy and cost advantage. We then relate this result to FHSD's visit-alignment characterization and the existing larger-data and longer-run evidence.

Q1: What does FHSD add to trajectory attribution and stability analysis?

FHSD quantifies visit reinforcement and cancellation and gives a sharp bound on their amplification across epochs. Trajectory methods, including SOURCE, already propagate perturbations through training. FHSD asks an additional question: how much of the squared terminal response comes from individual visits, and how much comes from their alignment? If two propagated visit effects are aa and bb, their combined energy is ∥a∥2+∥b∥2+2a⊤b\|a\|^2+\|b\|^2+2a^\top b. The last term distinguishes reinforcement from cancellation even when the individual energies are identical.

The practical consequence is that a sum of individual-visit energies can substantially misrepresent replacement sensitivity. Our decomposition identifies every visit-pair term under the realized training order. With at most EE visits per example, total energy is at most EE times the diagonal-only energy, and this factor is attainable even with strictly contractive updates. The contribution is this sharp characterization of the attribution quantity and its consequences, using standard algebraic tools. This distinction matters empirically: cross-epoch terms account for about 96% and 94% of linear-response energy in the reshuffled BERT experiments. Preserving visit alignment also reduces squared-loss prediction NMAE from 0.960 to 0.161 on BERT-tiny and from 0.947 to 0.129 on BERT-base, compared with sign randomization that preserves individual-visit energies.

Q2: How much comes from PathGram, and does M4 help SOURCE?

M4 supplies most of the error reduction in both methods, while PathGram+M4 has 89% lower mean error than SOURCE+M4 in nine matched Swin-T configurations. We evaluated three initialization/candidate seeds at three step sizes, with 16 replacements per configuration. Both methods use the same four additional full-gradient measurements per candidate and the same terminal loss readout. SOURCE uses four stationary segments with exact Hessian-vector products. The table reports mean normalized absolute error, with lower values better.

Response estimatorWithout M4With M4
PathGram0.1030.0027
SOURCE, four segments0.1130.024

As the table shows, M4 reduces SOURCE's error by 79% and PathGram's by 97%. After correction, PathGram's error is 89% lower. Thus M4 is a transferable residual correction, while the shared PathGram response provides additional accuracy in this controlled comparison. All four methods select the best replacement in every configuration, so the added benefit concerns magnitude accuracy and computation.

The cost comparison also favors PathGram under the common budget ceiling. Both methods receive the same per-configuration Hessian-vector-product allowance and four-gradient correction allowance. PathGram+M4 uses 58–76 Hessian-vector products, compared with 2,746–4,268 for SOURCE+M4, and is 14.6 times faster by the median paired timing ratio. These timings include each method's curvature preparation, response, correction, and terminal readout. They describe this exact-Hessian SOURCE implementation and the fixed four-segment setting.

Q3: What evidence supports larger datasets and longer runs?

The enlarged text study preserves accurate replacement selection at 1,000 training examples across four BERT sizes and three seeds per model. Mean NMAE is 0.060, 0.028, 0.160, and 0.079 for tiny, mini, medium, and base, respectively. Eight of the twelve runs recover all six best replacements and the remaining four recover five, with maximum normalized selection regret 0.014. Estimator cost is 1.2%–5.8% of matched neighboring retraining. These measurements concern final-attention LoRA adaptation, which makes the trainable parameter space explicit.

Accuracy also persists as the evaluated Swin-T run grows from 20 to 80 updates. On Swin-T, PathGram's quadratic-readout NMAE is 0.038, 0.056, 0.065, and 0.071 at 20, 40, 60, and 80 updates. Each stopping point has its own evaluation derivatives and full retraining outcomes. A separate matched BERT scale panel extends to 320 updates. These results support finite-horizon adapter attribution across the measured sizes and horizons.

Thank you also for the presentation suggestion. Bringing the principal accuracy–cost comparison alongside the visit-interaction evidence would make the experimental argument easier to follow.

Reviewer_sWUzOpenReview ↗

Thanks for your response.

Thank you for your detailed responses and additional experiments, which have addressed several of my concerns.

I agree that the theoretical novelty, particularly regarding FHSD, could be clearer. Still, I find the analysis of cross-epoch interactions (PathGram and M4) and the improvements in accuracy and efficiency valuable. Overall, I will keep my weakly positive rating.

Reviewer_Q8qAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the interactions between repeated visits to training examples across different epochs in training data attribution. The authors introduce the finite-horizon stability dimension (FHSD) to quantify these interactions and derive a decomposition and theoretical bound for their contributions to the final model response. Based on this analysis, they propose PathGram, which uses a shared low-rank Hessian approximation to efficiently predict the effects of training data replacement. A four-checkpoint residual correction is further introduced to improve prediction accuracy. Experiments on BERT and Swin-T demonstrate the importance of cross-epoch interactions and show that the proposed method can accurately predict loss changes with lower computational cost than full retraining.

Critical strengths

The paper provides a clear theoretical analysis of cross-epoch interactions in training data attribution, supported by empirical evidence demonstrating their substantial contribution to the final model response.

Critical weaknesses

Though it is supposed to explicitly examine the cross-epoch effects, such effects are in fact implicitly considered by current trajectory-based attribution methods by propagating sample perturbation during the training procedure. The empirical improvement of PathGram on SOURCE is diverse and inconsistent, while SOURCE still outperforms PathGram in some cases. The most significant improvement is obtained with our four-checkpoint correction (M4) scheme. Given that M4 relies on trajectory residual correction, it is also meaningful for other trajectory methods, e.g., SOURCE. Lack of comparison between SOURCE and M4 prevents us from determining whether such large improvement is inherent for PathGram or is a common phenomenon for other residual correction methods. This also questions the novelty and practical advantage of our method over the state of the art.

Questions for authors

Would the authors please indicate whether they can apply the proposed M4 modification on SOURCE as well, and report the results of PathGram/M4 and SOURCE/M4 under the same computational budget? This would enable us to investigate whether the large performance gain is mainly caused by PathGram design or general residual correction.

3 comments

Thank you for your constructive comments. We applied M4's four-gradient correction to PathGram and SOURCE under a common Hessian-vector-product ceiling. This comparison shows that M4 benefits both, while corrected PathGram remains more accurate and faster in every configuration, separating the transferable correction from the shared response approximation.

Does SOURCE benefit from the same four-checkpoint correction?

Yes. M4 reduces SOURCE's mean normalized absolute error from 0.113 to 0.024, a 79% reduction. We used nine Swin-T configurations comprising three initialization/candidate seeds and three step sizes, with 16 independently evaluated replacements per configuration. SOURCE uses four stationary segments with endpoint-averaged exact Hessian actions and replacement gradients. Each correction measures four full gradients at the same training times and propagates the resulting residual through the same shared correction operator.

To isolate response correction, we hold the terminal readout fixed. For every estimator, we evaluate the actual query-loss change at its predicted terminal parameters, then compare it with full neighboring retraining. The table gives mean NMAE and median attribution time per configuration.

EstimatorNMAETime (s)Hessian-vector products
PathGram0.1030.1758–76
PathGram+M40.00270.2358–76
SOURCE, four segments0.1133.652,688–4,192
SOURCE+M40.0243.792,746–4,268

As the table shows, the large correction gain is shared by both trajectory estimators. However, PathGram+M4 has 89% lower mean error than SOURCE+M4 and is more accurate in all nine configurations. This separates M4's general benefit from the advantage of the corrected PathGram response in this experiment. Every method selects the best replacement in all nine configurations, so the distinction is prediction accuracy and cost rather than a change in best-candidate selection.

How is the computational budget matched?

The matched budget is a common ceiling of 3,072–4,800 Hessian-vector products per configuration and exactly four new full gradients per candidate. Actual product use differs, as reported above. PathGram's shared curvature representation leaves most of the product allowance unused, whereas SOURCE's segmented propagation consumes more products. Under this common allowance, PathGram+M4 is 14.6 times faster by the median paired timing ratio. Timing includes method-specific curvature preparation, response propagation, correction, and query evaluation.

The conclusion applies to the specified four-segment SOURCE-style implementation. Its exact Hessian-vector products separate the propagation comparison from an additional curvature-factor approximation. The source response also receives its own initial residual in M4, so the correction does not assume that SOURCE follows the PathGram recurrence.

What is new if trajectory methods already include cross-epoch effects?

FHSD explicitly separates the visit interactions that trajectory propagation already carries implicitly. We agree that SOURCE and related methods can retain cross-epoch effects. Our analysis quantifies their contribution to squared replacement sensitivity through the pairwise inner products of propagated visits. Consequently, it identifies when a diagonal-only or magnitude-only description loses the dominant part of the response. The visit-count bound is attainable even for strictly contractive updates, so contraction alone does not make those interactions negligible.

Cross-epoch terms carry about 94%–96% of linear-response energy in the reshuffled BERT experiments, making visit alignment consequential for prediction. Randomizing visit signs preserves each visit's energy while increasing squared-loss prediction NMAE from 0.161 to 0.960 on BERT-tiny. PathGram then provides a shared approximation for many replacement candidates, and M4 corrects its finite-displacement residual. Thus the scientific contribution is the explicit characterization of visit alignment and its use in accurate replacement prediction, with the new SOURCE+M4 comparison establishing where the practical advantage remains.

Thank you for the additional experiments. The experienment reuslt is convincing and addresses my original concern.

However, after reading Reviewer Tjks's comments, I also share the concern about the novelty and significance of the theoretical contribution. Existing trajectory-based attribution methods already account for the propagation of sample effects across training, and the proposed visit-pair decomposition appears to follow naturally from expanding the squared norm of the accumulated contributions. It remains unclear what fundamentally new insight FHSD provides beyond explicitly quantifying these interactions.

Therefore, I remain concerned about the novelty of the work and will maintain my original rating.

Thank you for your follow-up. At the same base-training contraction rate, the ratio of complete tangent-response energy to summed individual-visit energies can grow with the visit count or saturate. Two fixed replacements below show how gradient evolution, combined with subsequent propagation, produces these distinct regimes.

What is the theoretical insight beyond the squared-norm identity?

Proposition 3 gives a sharp relation between complete response energy and individual-visit energies. Let DD be the mean squared tangent response (FHSD) and DdiagD_{\mathrm{diag}} the corresponding sum of individual-visit energies, measured in the same terminal metric. With at most EE visits per example,

0≤D≤EDdiag.0\le D\le E D_{\mathrm{diag}}.

The upper bound is attainable even under strictly contractive updates. Thus stable updates can coexist with a large relative contribution from repeated visits. SOURCE (Bae et al., 2024) and data value embeddings (Wang et al., 2025b) already propagate and aggregate those visits. Our analysis characterizes the reinforcement and cancellation within that aggregate, including its sharp dependence on the visit count.

What determines whether the ratio grows or saturates?

Two fixed replacements in the same quadratic training problem give different answers. Let the nn original sample losses be hw2/2hw^2/2, with h>0h>0, initialization w0≠0w_0\ne0, and one full-batch update per epoch. Choose a step size η\eta such that ρ=1−ηh∈(0,1)\rho=1-\eta h\in(0,1). The base path is ws=ρsw0w_s=\rho^s w_0. Below, vsv_s denotes a visit's contribution to the paper's scaled tangent response, with terminal metric M=1M=1.

Changing one sample's curvature gives the maximal ratio EE. Replace its loss by (h+β)w2/2(h+\beta)w^2/2, with 0<β<nρ/η0<\beta<n\rho/\eta. Every interpolated training path is then strictly contractive. The gradient difference is βρsw0\beta\rho^s w_0, and propagation through the remaining updates multiplies it by ρE−1−s\rho^{E-1-s}. Consequently,

vs=−ηβρE−1w0for every s,v_s=-\eta\beta\rho^{E-1}w_0\quad\text{for every }s,

so D/Ddiag=ED/D_{\mathrm{diag}}=E. Earlier visits have larger gradient differences that exactly offset their longer subsequent damping. This realizes the sharp bound with one fixed data replacement.

Shifting one sample's minimizer gives a saturating ratio. Replace its loss by h(w−c)2/2h(w-c)^2/2, with c≠0c\ne0. The update contraction remains ρ\rho, while the gradient difference is constant at −hc-hc. Hence vs=ηhcρE−1−sv_s=\eta hc\rho^{E-1-s}, and summing the geometric series gives

DDdiag=(1−ρE)(1+ρ)(1+ρE)(1−ρ).\frac{D}{D_{\mathrm{diag}}}=\frac{(1-\rho^E)(1+\rho)}{(1+\rho^E)(1-\rho)}.

For fixed ρ\rho, this ratio approaches (1+ρ)/(1−ρ)(1+\rho)/(1-\rho) as training continues. For example, at ρ=0.9\rho=0.9 and E=80E=80, the first replacement gives a ratio of 8080, whereas the second gives about 1919.

These calculations explain why base-path contraction alone does not determine relative accumulation. In the first case, the evolving gradient difference compensates for propagation decay. In the second, a constant gradient difference leaves earlier visits progressively downweighted. The geometric filter in the second case is shared with SOURCE. Here it connects the replacement's gradient evolution to the sharp energy range in Proposition 3.

Why does this distinction matter in the measured training runs?

The recorded Swin-T trajectory exhibits a large interaction effect: across 16 replacements and 80 visits per example, the mean complete linear-response energy is 66.3 times the mean individual-visit energy sum, about 83% of the visit-count bound (Appendix F.9). These quantities use the exact discrete tangent projected onto the evaluation gradient. Thus the bound describes a substantial effect in this measured trajectory, and the individual-visit energy sum alone misses most of its squared linear response.

This characterization and the predictor address complementary questions. FHSD explains how visit contributions combine. PathGram retains each visit's gradient difference while sharing curvature computation across candidates, and M4 corrects the approximate finite-replacement path. The matched SOURCE+M4 comparison that you found convincing establishes corrected PathGram's accuracy and cost advantage in the evaluated setting. We would appreciate your reassessment in light of this concrete theoretical interpretation and the matched empirical evidence.

Reviewer_TjksOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors address the problem of cross-epoch interactions that happen when replacing examples with another for the purpose of attribution. Influence functions are optimal only in settings where there are no ordering effects in optimization. However, in practical neural network optimization ordering across different batches matter. This can significantly affect the final response due to the fact that different visits of the same training example can reinforce or cancel in the terminal response.

The authors compute a measure FHSD, to quantify the training data interactions across epochs. They propose PathGram which predict the signed loss changes by propogating the visit contributions of the candidates. I could not understand the significance of the results beyond this.

Critical strengths
  • The paper starts off with a interesting premise of quantifying training data interactions across different epochs. The problem setting seems valid.
Critical weaknesses
  • The paper is extremely hard to understand, for a research like me who knows the area of TDA but does not have familiar with the temporal TDA literature. I could not understand what the authors were trying to achieve based on methods they proposed. I think the issue starts with the problem definition and terms get introduced without definition and this complicates further reading. I would recommend the authors revise the paper thoroughly so that a general researcher or reviewer can see its merits. It would help to get it reviewed by a general researcher before resubmission. I do not know how much AI was used in writing, but the narrative is quite hard, at least for me to follow.

  • One thing to note beyond the above was that the method heavily relies on second-order information and this is either quite hard or intractable even for slightly bigger models where such methods can have consequence.

Questions for authors

Conditioned on the previous weaknesses, further questions/comments:

  1. In the introduction, the problem itself (dependence of influence function computation on data order, the permutation invariance assumption that is violated in modern training regimes etc.) should be explained better. Wang et al. 2025b present the problem well and the authors should attempt to do this in this case as well.

  2. In Figure 1, what are you predicting?

  3. What is visit-pair decomposition? Can you explain what FHSD is simply without jargon?

  4. What is the use of replacement as defined in problem setup (sec. 3)? Are you trying to do counterfactual prediction as a way of estimating influence?

  5. Given the lack of clarity thus far, the seemingly simple section 3.1 adds to the confusion unfortunately.

  6. Can you simply describe what the scaled tangent response is and why it is important to the problem in hand?

2 comments

Thank you for your careful reading and feedback. We clarify the task and response notation through one concrete question: how would evaluation loss change if an example were replaced and the same training run repeated? This leads from repeated visits to their propagated effects, explaining how PathGram predicts many replacements with shared computation.

Q1 and Q4: What is the counterfactual, and why does data order matter?

Yes, replacement is a counterfactual definition of training-data influence. For example, we ask whether replacing one mislabeled or unsuitable example with a specified alternative would improve the trained model. The target is the new model's evaluation loss minus the original model's evaluation loss. A negative value indicates a beneficial replacement. The replacement persists throughout training, so every visit to the original example changes.

Data order matters because an early gradient change passes through all later updates before reaching the final model. Moving the same example to another batch changes both where its effect enters and which later updates transform it. Wang et al. (2025b), Capturing the Temporal Dependence of Training Data Influence, provides the relevant trajectory-based perspective. Our question builds on that perspective by measuring how different visits to the same replaced example reinforce or cancel after propagation. Endpoint influence calculations describe a stationary solution, whereas our target is the output of a specified finite training run.

Q3: What are FHSD and the visit-pair decomposition?

FHSD measures the average squared size of the local replacement response at the end of training. The chosen measurement can concern parameter movement or the linearized evaluation-loss change. To understand the decomposition, consider two visits whose effects at the final model are aa and bb. Their combined squared size is ∥a+b∥2=∥a∥2+∥b∥2+2a⊤b\|a+b\|^2=\|a\|^2+\|b\|^2+2a^\top b. The first two terms measure the visits separately. The last term measures their interaction: aligned effects reinforce, and opposing effects cancel.

The visit-pair decomposition applies this identity to every visit, after transporting each effect through the later updates. This makes a consequential quantity visible: cross-epoch terms account for about 96% and 94% of the linear-response energy in our two reshuffled BERT experiments. Summing only separate visit energies would therefore miss most of the measured response in these settings.

Q5 and Q6: What does Section 3.1 define, and why scale the tangent response?

The tangent response is the final-parameter derivative as we begin replacing one example's loss with another. It provides the local approximation used to predict the complete replacement. A mixing weight α\alpha makes this precise: zero uses the original loss, one uses its replacement, and intermediate values mix them. The derivative is evaluated at zero.

The factor nn only removes the dataset-average scaling, making this response easier to compare across dataset sizes. The paper calls the scaled derivative gTg_T, so gT/ng_T/n is the physical first-order parameter displacement. Its recurrence has two parts: later training transforms the displacement already accumulated, and the current visit adds the new-minus-old example gradient. PathGram approximates this recurrence, while M4 uses four additional gradient measurements to correct the residual associated with a complete replacement.

Q2: What exactly does Figure 1 predict?

Figure 1's prediction panel estimates the squared evaluation-loss change caused by replacement. To assess the role of visit interactions, its energy panel measures the squared linearized loss response and its cross-epoch terms. The prediction panel then compares squared quadratic-loss predictions with squared loss changes from actual replacement retraining, averaged over schedules. Thus the first panel explains the size of the interactions, while the second tests their predictive consequence. On BERT-tiny, retaining visit alignment reduces this prediction NMAE from 0.960 to 0.161 relative to sign randomization that preserves the separate visit energies.

Does second-order information make the method intractable?

Curvature is affordable in the evaluated adaptation setting: PathGram costs 1.2%–5.8% of matched neighboring retraining. It shares one low-rank representation across candidates, computed through Hessian-vector products, without forming or storing a dense Hessian per replacement. The evaluated BERT setting adapts the final attention block with LoRA, so curvature acts on the trainable coordinates. At 1,000 training examples, the four BERT sizes have mean replacement NMAE between 0.028 and 0.160. Thus sharing curvature makes replacement prediction practical in this adaptation setting.

Thanks for the responses.

Further question: How does the use of second-order information scale up with larger models?

Reviewer_GpgWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper finds that cross epoch interactions can affect training data attribution. The authors introduce FHSD to quantify how these visits impact each other, and propose PATHGRAM, which uses curvature information to predict how replacing a training example changes the model's evaluation loss.

Critical strengths

Offers new perspective on training data attribution

Critical weaknesses

The findings are very constrained, they are only on small classification tasks, therefore the value of the proposed method is unclear. Would be interesting to see this in more relevant/real world regimes like LLM.

Questions for authors
  1. Are the proposed methods able to transfer to a language modeling setting, how generalizable is this?
1 comment

Thank you for your review and suggestions. We clarify the transfer to language modeling by writing the response recurrence for autoregressive loss and relate it to the existing classification-adaptation measurements. The recurrence retains the same visit-pair structure, while the measured accuracy and cost show how shared computation can screen replacements without retraining for each candidate.

How does the method transfer to language modeling?

The replacement target and local response equations apply to autoregressive language modeling by treating a token sequence as one training example. Its loss is the sum or mean of negative next-token log probabilities. Replacing a sequence changes its gradient at every visit, and the evaluation target becomes the change in held-out language-model loss after the prescribed training run. Thus a negative predicted change identifies a replacement expected to improve next-token prediction.

The response recurrence is unchanged by this choice of loss. If hth_t denotes the local parameter displacement caused by the replacement, then

ht+1=(I−ηtHt)ht−ηtdt.h_{t+1}=(I-\eta_t H_t)h_t-\eta_t d_t.

Here HtH_t is the current training-batch Hessian and dtd_t is the new-minus-old sequence-gradient contribution to that batch. The first term transports the accumulated effect through the current update, and the second adds the effect of the current visit. Consequently, the FHSD visit-pair identity applies to language-model gradients exactly as it does to classification gradients. PathGram approximates these transported effects with a shared curvature representation. The local identity requires differentiability, and accurate prediction of a complete replacement additionally depends on how well this local response follows the neighboring training path.

Adapter tuning provides a direct implementation of this extension. When only adapters are trained, hth_t and HtH_t refer to their trainable coordinates, while differentiation still passes through the frozen transformer computation. This keeps the curvature calculation tied to the adaptation space. The evaluated recurrence concerns gradient descent with a fixed schedule, so changing the optimizer would require differentiating its corresponding state update as well.

What existing evidence establishes useful accuracy and computation?

At 1,000 training examples, shared replacement prediction costs 1.2%–5.8% of full neighboring retraining and recovers five or six of the six best replacements in every run. Compared with the main 96-example setting, we evaluated BERT-tiny, mini, medium, and base with three independent data/initialization seeds per model and 24 candidate replacements per run. These are classification losses under final-attention LoRA adaptation. The table reports mean normalized absolute error and estimator time relative to full neighboring retraining.

ModelReplacement NMAERelative cost
BERT-tiny0.0605.8%
BERT-mini0.0284.4%
BERT-medium0.1604.3%
BERT-base0.0791.2%

As the table shows, the shared computation gives substantial savings while retaining useful replacement accuracy across the four model sizes. Eight of the twelve runs recover all six best replacements, and the remaining four recover five. The maximum normalized selection regret is 0.014, linking prediction quality to an actionable data choice.

PathGram+M4 also outperforms SOURCE+M4 in accuracy and measured cost under the same correction allowance. Across nine Swin-T adapter configurations, PathGram+M4 has mean NMAE 0.0027 versus 0.024 for four-segment SOURCE+M4 under the same four-gradient allowance and Hessian-vector-product ceiling. PathGram+M4 is more accurate in every configuration and is 14.6 times faster by the median paired timing ratio. These measured results establish the practical value for the tested adaptation problems, while the language-model formulation above identifies the precise loss and training recurrence to which the analysis transfers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.