acceptodds
Withdrawn from ICLR 2027

Beyond Pointwise Attribution: Relationship-Aware Training Data Attribution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Training data attribution is commonly formulated by assigning each training example a scalar influence score for a target model behavior. This formulation treats influence as an intrinsic property of an individual sample, yet model parameters are learned by jointly optimizing over the entire training distribution. As a result, the contribution of one example is inherently conditional on the presence of others: perturbing one sample can change not only its own effect, but also how other samples contribute to the learned model. Existing pointwise attribution methods collapse these coupled effects into marginal scores and cannot explicitly represent substitution, reinforcement, or dependency among training examples. Motivated by this, we reformulate training data attribution from estimating isolated sample importance to characterizing how training examples influence one another’s contribution to a target behavior. Instead of asking only which samples are important, we ask how the importance of one part of the training data changes when another part is perturbed. This leads to a novel ***Relationship-Aware Training Data Attribution*** task that captures compensatory, reinforcing, and suppressive dependencies among training samples. Specifically, we organize training samples into coherent groups according to their attribution-relevant representations and construct a query-conditioned asymmetric attribution graph over these groups. Each node represents a coherent region of the training data, while each directed edge measures how perturbing one group changes the attribution assigned to another. Since inter-sample effects are inherently asymmetric, we explicitly preserve their directionality rather than treating relationships as reciprocal. By modeling training influence at the level of structured and asymmetric group interactions rather than isolated examples, our approach provides a more faithful characterization of how different parts of the training data collectively shape model behavior.

Primary area:
interpretability and explainable AI
Keywords:
Training Data Attribution, Data Attribution, Relationship-Aware Training Data Attribution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9mMQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper points out that pointwise training data attribution misses the information carried by the relations between training data. For instance, two data points with the same pointwise attribution score can exhibit different impact on other data, and this impact could even be asymmetric. To represent both pointwise and relational information, the paper proposes Relationship-Aware Training Data Attribution, which is a graph-based attribution method. Since processing all relations between data has quadratic complexity in the number of data points, the paper instead addresses the scores of data groups. The node of the graph represents the attribution score of the group, and the directional edge of the graph represents the extent of the difference in the target group score when the source group is progressively removed.

Critical strengths

The most interesting point I found is the asymmetry of the relationship between the groups. To the best of my knowledge, current training data attribution literature that addresses the relation between data treats it as symmetric, so if the paper’s claim is true, it would be a different type of relationship compared to current methods. Related to this, I’d like to see whether substituting the aa in Theorem 2 with the existing training data attribution methods such as TRAK or influence functions still yields the asymmetric relation or not.

Critical weaknesses
  • The asymmetric result in Figure 2 could be an artifact from the randomness of retraining. Figure 2 measures how much the TRAK score of the target data changed when removing the source data and then retraining the model, which depends on training randomness such as mini-batch selection. This randomness could not be controlled by fixing the training seed since different data removals affect the mini-batch selection even with the same seed. So the asymmetric result in Figure 2 couldn’t be evidence of the relation’s asymmetry. The same problem arises in the paper’s method, which defines the edge as the effect of retraining on TRAK ranking. Without an analysis on retraining noise, it is hard to know whether the proposed edge reflects the actual relation between training groups.
  • The paper uses the change of TRAK score as the main metric for analysis. However, TRAK itself is an approximation of the retraining effect. So the current analysis could not identify whether the results come from the inherent error of TRAK or the relation is actually important. One of the possible alternative metrics is leave-group-out retraining, which is a standard metric of training data attribution; however, I expect it to yield a symmetric relation. Then is the asymmetric relation derived by the paper an artifact from the inherent error of TRAK?
  • The proposed Relationship-aware training data attribution needs repeated model retraining to get edges, which raises concern about its prohibitive cost. For large-scale models or datasets, it could be challenging to apply the method due to this constraint.
Questions for authors
  • Could the authors distinguish the effects of the randomness of retraining and of the relation between training data in Figure 2? How much of the whole effect does the randomness effect account for? And is the relation still asymmetric after removing the randomness effect?
  • Could the authors explain the reason for using TRAK as the metric throughout the paper? If the leave-group-out retraining effect is used instead of TRAK, is the relation still asymmetric?
  • Could the proposed method extend to large-scale models or datasets? Could the authors provide the wall-time of the experiments of the paper?
Reviewer_PKfxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Relationship-Aware Training Data Attribution, using TRAK-based grouping and retraining interventions to build a directed graph of how data groups affect one another’s attribution. Theory and small-scale classification and diffusion experiments suggest these relationships are reproducible. The focus shifts from which data matter to how their importance depends on other data.

Critical strengths

S1. The paper studies an interesting and relevant problem in training data attribution.

S2. The motivation is clearly stated, and the intuitive example in Figure 1 helps convey the main idea.

Critical weaknesses

W1. Although the paper reports repeatability and source-specific retrieval results, it lacks explicit noise controls.

W2. The experiments rely on TRAK and D-TRAK, so it is unclear whether the findings generalize beyond this estimator family.

W3. The mathematical distinction remains unclear. For raw exact LOO attribution under full deletion, cross-attribution changes reduce to a symmetric second-order difference. Exact infinitesimal influence responses are also symmetric when they are mixed derivatives of the same smooth prediction function. The toy example in Appendix A instead defines a₁ = θ and a₂ = 2θ without establishing their connection to a standard training-data attribution method. The authors should clarify how Theorem 2 relates to these symmetric cases and to the estimators used in the experiments.

W4. Although the paper discusses related approaches and provides interaction comparisons and a predictive-utility experiment in Appendix L, the added value over existing interaction measures remains insufficiently established. A direct comparison on a common prediction or data-selection task would help clarify this contribution.

minor weakness m1. No standard deviations or confidence intervals are reported in the main tables, and the statistical significance of individual edges is not assessed.

m2. Sensitivity to the number of groups (K = 30) and to the clustering seed is not studied.

m3. The manuscript contains several typos and grammatical errors, e.g., “extenstions” and “differnces” (Section 2), “which training data important” (Section 1), and “cannot not guaranteed” (Appendix A).

Questions for authors

Q1. What evidence distinguishes the observed relationships from retraining noise and generic data-removal effects? Do you have comparisons with seed-only retraining and random-subset removal of matched size?

Q2. Do you have evidence that the findings hold for an attribution method outside the TRAK/D-TRAK family, such as TracIn or influence-function attribution?

Q3. How does Theorem 2 relate to the symmetric responses obtained from raw exact LOO under full deletion and from mixed derivatives of a smooth prediction function? How do the toy attribution functions a₁ = θ and a₂ = 2θ connect to standard attribution methods or the estimators used in the experiments?

Q4. Beyond the fixed-reference check in Appendix H, what evidence separates meaningful directional dependencies from effects of rank normalization, group size, and estimator approximation? What additional explanatory or predictive value does this directionality provide over existing interaction measures?

Reviewer_zTMEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how perturbing one training-data group changes the attribution assigned to other groups. It represents these responses as directed edges, estimated through repeated data removal, model retraining, and recomputation of attribution scores.

Critical strengths

The paper identifies a meaningful limitation of pointwise attribution: similar importance scores can hide different dependencies on the surrounding training data. Studying these relationships is a worthwhile direction.

Critical weaknesses
  • The accuracy of the inferred relationships is not verified. Table 1 measures changes in attribution rankings and their repeatability, but does not verify whether the identified relationships correctly represent compensation or dependency between groups.
  • The additional practical value is unclear. The relational scores describe changes in existing attribution, so they do not directly replace ordinary attribution. Their usefulness therefore depends on the additional insights or applications they enable. The experiments mainly examine attribution changes themselves, without demonstrating how the relationships improve explanations of model behavior or guide data decisions. They also do not demonstrate any additional benefit from using the full graph beyond individual relations or response profiles.
  • The computational cost is substantial. The reported setup requires 270 counterfactual training runs to construct a full graph, with attribution recomputed after each run. No runtime or scaling analysis is provided to establish feasibility for larger models.
Questions for authors

Comments on the paper

1 comment
Area_Chair_Lid3OpenReview ↗

From AC.

Dear Authors,

Given the deadline on 18 November, please submit your responses to the reviews to facilitate a productive discussion with the reviewers.

Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.