acceptodds
Under review as a conference paper at ICLR 2027

Knowledge Networks Enable Stable and Effective Attribution: Data Shapley Valuation in One Training Run

Abstract

In-Run Data Shapley makes principled data valuation feasible for modern foundation models by estimating sample contributions along one observed training trajectory. However, existing in-run estimators mainly treat examples as independent contributors, making attribution scores unstable under noisy gradient signals and prone to favoring semantically redundant samples. A direct second-order treatment could model these interactions, but Hessian-dependent computation is costly and difficult to scale. To address these challenges, we propose Graph-Aware In-Run Data Shapley, a simple but effective approach that introduces a sample-level knowledge network into one-run valuation. The network connects semantically related examples and is used to smooth structurally supported contributions while penalizing redundant neighbors during selection. This design stabilizes attribution, improves diversity-aware data valuation, and replaces expensive Hessian-based interaction modeling with sparse neighborhood aggregation. Extensive experiments on image and language learning tasks show that the proposed graph-aware estimator improves attribution stability, subset selection, and computational efficiency over state-of-the-art approaches.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Data Valuation, Data Shapley, Training Data Attribution, Graph Regularization, Data-centric Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JqtHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Graph-Aware In-Run Data Shapley, a data-valuation method that augments one-run Data Shapley (ICLR '26 and its Adam optimizer-aware variant) with a sample-level semantic knowledge network. Standard in-run valuation accumulates each sample’s contribution independently, so scores can be noisy and high-value near-duplicates can dominate selection. The proposed method builds a kNN graph over sample embeddings, adds an in-batch neighbor relation as a lightweight surrogate for second-order interactions. It then reuses the same graph for diversity-aware subset selection with a redundancy penalty, by a greedy algorithm. Experiments on Pile/GPT-2 and CIFAR-10 show improved attribution stability, corruption detection, selected-subset accuracy/diversity, and efficiency.

Critical strengths

Apply a well-known diversification framework to in-run Shapley and show improvements.

Critical weaknesses
  • The idea is very simple and incremental. Graph-based or more generally diversification frameworks are really well known and everywhere. The main idea of the paper is to instantiate the in-run Shapley value as the utility in this framework.

  • Analysis is relatively superficial. For example, why does graph help Noise AUROC metric in practice?

  • Experimental Details are missing both from the text and the code. How to corrupt text data? Very importantly, how do you apply KNN-Shapley to text data, which as far as I know can only be used for traditional classification or regression tasks, like mislabeled images? ALso, do you train on corrupted text data (after selection) to report the metric Val. Accuracy? Can you do the same without corruption?

  • Not enough details are given in the source code to reproduce the experimental results other than the authors' own implementation.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Beyond Diversification: Better Attribution, Up to 1.29 Percentage Points Higher Accuracy, and 47% Less GPU Memory

Thank you for the helpful comments. We have sharpened the methodological contribution, clarified the experimental protocols and baseline definitions, and consolidated the supporting evidence below. Tables 1–5 reproduce results from the submitted manuscript. All revisions made in response to your comments are highlighted in red in the revised manuscript.

Q1. Contribution beyond existing diversification frameworks

Our central contribution is graph-based correction of trajectory-level attribution before data selection, rather than diversification alone. Historical neighbor evidence incorporates cross-step sample relations into Adam-aware contributions; an in-batch pair signal can additionally account for interactions among examples evaluated together. The corrected scores then support redundancy-aware selection using the same sparse sample graph. Crucially, the attribution-only results in Table 1 assess this correction before diversity selection, isolating its effect on valuation.

Table 1. CIFAR-10 attribution ablations (three data seeds × three training seeds). AUROC and cross-seed Spearman are reported as mean (SD); no diversity selection is applied.

AttributionNoise AUROC ↑Cross-seed Spearman ↑
Adam-aware first order0.571 (0.009)0.700 (0.016)
Graph without gate0.558 (0.014)0.721 (0.015)
Graph without history0.584 (0.009)0.703 (0.016)
Graph without pair signal0.636 (0.009)0.727 (0.014)
Full Graph0.647 (0.009)0.733 (0.014)

Beyond attribution quality, the complete pipeline jointly improves downstream utility and selected-subset diversity at all four low-data budgets. Table 2 reports first-order results. For the second-order variant, the corresponding token-accuracy gains are 1.2940, 1.1872, 0.9802, and 0.4345 percentage points at 1%, 2%, 5%, and 10% retention, respectively.

Table 2. Pile low-budget selection: In-Run + Top-k versus the complete Graph pipeline. This experiment evaluates attribution and selection jointly; arrows indicate baseline → Graph.

RetentionToken accuracy (%) ↑Validation loss ↓Affinity/item ↓One-hop coverage ↑
1%40.7012 → 41.96723.4080 → 3.28370.0156 → 0.00950.1246 → 0.1375
2%41.2997 → 42.48823.3639 → 3.24750.0220 → 0.01590.2336 → 0.2536
5%42.0751 → 43.06993.2636 → 3.19170.0475 → 0.03390.4840 → 0.5054
10%42.8677 → 43.47013.2251 → 3.16040.0720 → 0.06090.6807 → 0.7471

The sparse graph also makes structural interaction correction computationally practical. We benchmark the HVP-free online Graph implementation on 1,000 Pile documents under the specified three-seed protocol; these measurements characterize this implementation rather than the separate 100k-document experiment.

Table 3. Online GPT-2 efficiency on 1,000 Pile documents (three seeds). Reported times include graph construction where applicable.

MethodInteraction (s) ↓Total time (s) ↓Peak memory (MiB) ↓
Adam-aware second order2.690 (0.014)14.770 (0.125)3320.1
HVP-free Graph0.037 (0.001)13.939 (0.098)1756.6

Under this protocol, Graph achieves 72.7× faster interaction computation, 5.6% lower total runtime, and 47.1% lower peak memory. The graph interactions serve as an efficient structural surrogate for sample dependencies without explicit Hessian-based interaction computation. Together, Tables 1–3 distinguish our contribution from a selection-only diversification method: the graph first improves attribution and then guides subset selection.

Q2. Why does graph correction help noise AUROC?

Graph correction uses neighborhood corroboration over the training trajectory. Historical messages accumulate relational evidence across steps, while the gradient-agreement gate used in the CIFAR-10 diagnostic attenuates inconsistent neighbor contributions. Table 1 quantifies these effects: relative to Full Graph, removing history reduces AUROC by 0.063, and removing gating reduces it by 0.089. Removing the in-batch pair signal also reduces AUROC from 0.647 to 0.636.

Table 4. CIFAR-10 propagation and selection diagnostics. Gated incoming mass and promotion refer to Full Graph. The 20%-retention contamination comparison uses raw versus graph-corrected attribution rankings without diversity selection.

DiagnosticClean samplesMislabeled samples
Mean gated incoming mass0.4670.043
Fraction promoted by Graph0.5660.021
Selection diagnosticAdam-awareGraph
Selected corruption fraction ↓0.0730.050

Clean samples receive substantially more gated neighborhood evidence than mislabeled samples, and the selected corruption fraction decreases by 31.5%. The parameter-sensitivity analysis reinforces this pattern: increasing the correction strength from α=0\alpha=0 to α=0.4\alpha=0.4 raises CIFAR-10 AUROC from 0.571 to 0.708 and lowers selected contamination from 0.073 to 0.023. The default configuration uses α=0.2\alpha=0.2.

Q3a. Exact text-corruption procedure

For the Pile noise-detection experiment, we select 10% of training documents without replacement using NumPy's default random generator seeded with run_seed + 991. For each selected recipient, define L=max⁡(8,min⁡(nwords,512))L=\max(8,\min(n_{\mathrm{words}},512)), where nwordsn_{\mathrm{words}} is its whitespace-tokenized word count. We combine the first ⌊L/2⌋\lfloor L/2\rfloor words from a randomly assigned donor with recipient words at positions ⌊L/2⌋\lfloor L/2\rfloor through L−1L-1, shuffle the resulting word sequence, and tokenize/truncate it to 128 GPT-2 tokens. Donors are assigned through a random permutation, and shorter donors can produce shorter mixtures. Validation documents remain clean. Corruption indicators are used only for evaluation, with lower attribution utility indicating greater likelihood of corruption. The revised appendix specifies the complete procedure.

Q3b. KNN-Shapley baseline definition on text

We have clarified the implementation and naming of the embedding-based retrieval baseline. The 100k-document entry previously labeled “KNN-Shapley” computes cosine similarity between each document embedding and the normalized mean embedding of the utility documents. We therefore relabel it Semantic Retrieval and explicitly state its scoring rule; it should not be interpreted as an implementation of the KNN-Shapley recurrence. Other simplified gradient baselines are likewise identified by their actual scoring functions. These nomenclature corrections do not change the reported scores or the matched Adam-aware versus Graph-aware comparisons, and we do not rely on the relabeled entry to claim superiority over a true KNN-Shapley implementation.

Q3c–d. Corrupted-text retraining and clean-data evaluation

The main-table subset-selection experiments use clean Pile text for attribution, selection, and retraining. Noise AUROC is evaluated separately using synthetically corrupted text. For subset evaluation, selected documents are used to fine-tune pretrained GPT-2 for one epoch with all model parameters trainable; during attribution, only the final LayerNorm is trained. The accuracy metric is next-token top-1 accuracy over nonpadding target tokens.

Table 5. Clean Pile subset selection (three runs): token accuracy (%) mean (SD). Gains are Graph minus the corresponding Adam-aware estimator, in percentage points.

Method20%40%60%80%
Adam-aware first order41.055 (0.005)41.180 (0.011)42.304 (0.007)43.366 (0.007)
Graph first order41.799 (0.020)42.132 (0.009)43.308 (0.021)44.345 (0.015)
Gain+0.744+0.952+1.004+0.979
Adam-aware second order40.803 (0.021)41.143 (0.016)43.261 (0.010)43.351 (0.019)
Graph second order41.825 (0.011)42.126 (0.025)43.275 (0.004)44.350 (0.021)
Gain+1.022+0.983+0.014+0.999

Graph-aware selection improves token accuracy in all eight attribution-order × retention-budget settings, establishing downstream benefits independently of synthetic text corruption.

Q4. Reproducibility details

We have consolidated the key settings in the revised protocol documentation:

ItemExplicit protocol
Data100k training / 10k held-out Pile documents; 128 tokens
AttributionFinal LayerNorm, 1,536 parameters; batch 8; AdamW learning rate 6×10−46\times10^{-4}
Utility16 fixed domain-stratified documents for trajectory gradients
OptimizerBetas (0.9, 0.95); weight decay 0.1; gradient clipping 1.0
Graph10 neighbors; α=0.2\alpha=0.2; recorded one-pass score replay for Graph v2
Pair termIncluded in second-order Graph v2; first-order configuration uses historical correction
SelectionSymmetric-graph greedy; λ=0.05×\lambda=0.05\times score SD
Subset retrainingAll parameters; one epoch; batch 16; learning rate 6×10−56\times10^{-5}
RepeatsSeeds 0, 1, 2; per-seed outputs for new diagnostics

The anonymous repository https://anonymous.4open.science/r/graph-aware-shapley-1244/ is the reference location for the experimental implementation. The revised reproduction materials document baseline scoring rules, corruption metadata, reproduction commands, executable protocol checks, and the two-GPU (A100) diagnostic workflow, including checkpoint hashes and per-seed outputs. Files not yet present in the anonymous release are identified as prepared revision materials rather than currently accessible artifacts.

In summary, the evidence supports three complementary contributions: (i) graph-based correction improves trajectory-level attribution stability, (ii) graph-guided selection yields stronger downstream training subsets across retention budgets, and (iii) sparse structural interactions can be computed efficiently. We appreciate the opportunity to clarify these points and the experimental conditions under which each result is obtained.

We thank the authors for their replies. I appreciate the experimental details. Also, renaming “KNN-Shapley” to Semantic Retrieval now makes sense; it is important to go over literature papers before using them. However, I maintain my main assessment of the methodology.

Reviewer_o5EWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper targets two pain points of In-Run Data Shapley (data valuation along a single training trajectory): (1) contribution scores accumulated independently per sample are unstable under noisy gradients/optimizer state changes; and (2) the valuation does not control diversity, so semantically near-duplicate samples can all receive high scores. The proposed approach builds a kNN “knowledge network” on sample embeddings.Experiments: Pile-100k + GPT-2 and CIFAR-10 + ResNet-18; it is claimed to be more stable, select better subsets, and be HVP-free (interaction time drop from 2.69s to 0.037s, peak GPU memory −47%).

Critical strengths

1.The problem is real and clearly scoped. In-Run Shapley really does accumulate contributions independently sample by sample; noise sensitivity and the redundancy blind spot are genuine flaws. The paper’s overview of the valuation-method lineage is accurate. 2.The method is cheap, orthogonal, and deployable. Each step incurs only graph overhead, does not change the training trajectory, and can be stacked on both first- and second-order estimators; being HVP-free cuts interaction cost by 72× and peak GPU memory by 47%. 3.The experimental coverage is broad: two modalities, 12+ baselines, retraining at four budget levels, contamination detection, cross-seed stability, paraphrase retrieval, and dynamic analysis of domain contributions.

Critical weaknesses

1.The “knowledge network” is overbranding. Strip away the naming, and the method is just a kNN graph over pretrained embeddings + score neighborhood smoothing + MMR-style greedy deduplication selection. There is no new graph algorithm, no convergence/consistency analysis, or any analysis of basic questions like when smoothing helps versus when it spreads the scores of bad actors to clean near neighbors

Questions for authors

1.Can you explain the diffenrence bettween your work and a method like "kNN graph over pretrained embeddings + score neighborhood smoothing + MMR-style greedy deduplication selection"?

2 comments
AuthorsOpenReview ↗

Beyond Static Smoothing: Trajectory-Aware Attribution and Reliability-Gated Evidence

Thank you for the constructive comments. We have refined the distinction between trajectory-aware attribution and static neighborhood smoothing, clarified the role of reliability gating, and strengthened the analysis with explicit smoothing conditions and properties of the implemented correction. The corresponding ablations and diagnostics provide direct empirical support. All revisions made in response to your comments are highlighted in blue in the revised manuscript.

Q1. Difference from embedding kNN + neighborhood smoothing + MMR-style selection

Our central contribution is graph-based correction of attribution along an observed training trajectory, rather than applying graph smoothing only to final sample scores. We use a sparse sample-relation graph to incorporate historical neighbor evidence and, in the pair-enabled configuration, interactions among samples in the current batch. The corrected attribution scores then support redundancy-aware subset selection.

ComponentStatic smoothing + MMR-style selectionOur implementation
Base evidenceFinal per-sample utility scoreAdam-aware contributions recorded along one training run
Neighbor evidenceNeighbor scores after accumulationHistorical raw contributions available before the current batch; current-batch pair evidence in the pair-enabled variant
ReliabilitySemantic affinity alone in the stated comparatorGradient-agreement gating in gated variants
Correction scaleFixed mixing coefficientBatch-centered correction scaled to raw contribution variation
SelectionRelevance–redundancy trade-offExact marginal of a sparse pairwise-redundancy objective

For batch tt, the pair-enabled correction is

\hat{\mathbf c}^{(t)} &=\mathbf c^{(t)}+\alpha\psi\!\left(\mathbf d^{(t)}\right) +\frac{\alpha}{2}\psi\!\left(\mathbf p^{(t)}\right),\\ \mathbf d^{(t)}&=R_tW\mathbf h_{t-1},\\ \mathbf p^{(t)}&=R_tWR_t^\top\mathbf c^{(t)}, \end{aligned}$$ where $\mathbf h_{t-1}$ contains accumulated **raw** contributions from earlier steps, $R_t$ selects the current batch, and $\psi$ centers and scale-aligns the correction signals within that batch. Gated variants use reliability-weighted affinities. In the 100k Pile experiments, the first-order configuration uses historical correction, whereas the pair-enabled second-order configuration uses both terms. Recorded contributions can also be replayed in their original batch order, preserving the temporal availability of neighbor evidence and the batch-dependent correction scales. The subsequent selector follows the established relevance–redundancy principle, with a sparse pairwise objective: $$J(S)=\sum_{i\in S}\hat v_i -\lambda\sum_{\substack{i,j\in S, i<j}}\bar W_{ij}, \qquad \Delta_iJ(S)=\hat v_i-\lambda\sum_{j\in S}\bar W_{ij}.$$ Unlike max-similarity MMR, this objective penalizes **total affinity to already selected samples**. Importantly, the attribution correction is evaluated independently of this selection step. **Table 1. Attribution-only CIFAR-10 ablations, three data seeds × three training seeds.** Values are mean (SD); neither metric uses diversity selection. | Attribution | Noise AUROC ↑ | Cross-seed Spearman ↑ | |---|---:|---:| | Adam-aware first order | 0.571 (0.009) | 0.700 (0.016) | | Without gate | 0.558 (0.014) | 0.721 (0.015) | | Without history | 0.584 (0.009) | 0.703 (0.016) | | Without pair evidence | 0.636 (0.009) | 0.727 (0.014) | | **Full Graph** | **0.647 (0.009)** | **0.733 (0.014)** | The full graph configuration raises noise AUROC from **0.571 to 0.647** and cross-seed Spearman from **0.700 to 0.733**, before any diversity-based selection. The ablations further identify complementary contributions from historical evidence, pair interactions, and reliability gating. For downstream clean Pile selection, first-order Graph improves next-token accuracy by **0.744, 0.952, 1.004, and 0.979 percentage points** at 20%, 40%, 60%, and 80% retention, respectively, over Adam-aware first order. Together, these results distinguish attribution-level improvements from the additional benefits of graph-guided selection.

Q2. When does smoothing help, and how does reliability gating affect propagation?

We provide a bias–variance analysis to characterize when neighbor information is beneficial. Consider a fixed-weight smoothing comparator:

\qquad c_i=u_i+\epsilon_i,$$ where the weights are fixed, nonnegative, row-normalized, and exclude self-edges, and the errors are independent, zero-mean, with common variance $\sigma^2$. Defining $\delta_i=\sum_jw_{ij}u_j-u_i$ gives $$\operatorname{MSE}(s_i) =a^2\delta_i^2+ \sigma^2\!\left[(1-a)^2+a^2\sum_jw_{ij}^2\right].$$ For positive smoothing strength, this MSE is lower than the raw-score MSE $\sigma^2$ precisely when $$0<a<\frac{2\sigma^2}{\delta_i^2+\sigma^2\left(1+\sum_jw_{ij}^2\right)}.$$ This result identifies the mechanism: neighborhood evidence reduces estimation variance when the neighboring utilities are sufficiently aligned, while utility mismatch increases bias. In our graph correction, **historical evidence supplies cross-step corroboration**, and **gradient-agreement gating, where enabled, attenuates unreliable neighbor messages**. **Table 2. CIFAR-10 propagation diagnostics for the gated Full Graph configuration.** | CIFAR-10 diagnostic | Clean | Mislabeled | |---|---:|---:| | Mean gated incoming mass | 0.467 | **0.043** | | Fraction promoted by Graph | 0.566 | **0.021** | The gated graph assigns substantially less incoming weight to mislabeled samples (**0.043** versus **0.467** for clean samples), and only **0.021** of mislabeled samples are promoted by the correction, compared with **0.566** of clean samples. At 20% retention, attribution-based ranking reduces the selected corruption fraction from **0.073 to 0.050**, a **31.5% relative reduction**. These measurements connect the improvement in noise AUROC to the selectivity of the propagated evidence. The Pile propagation diagnostic provides a complementary view: corrupted recipients receive **0.930** of their incoming semantic weight from corrupted sources. This illustrates why semantic affinity and reliability are distinct signals, and motivates the gated configuration. The additional small-scale Pile control did not establish an AUROC gain; accordingly, the noise-discrimination mechanism is supported most directly by the CIFAR-10 gated ablations, while the main Pile results substantiate improvements in clean-data subset utility. ## Q3. What theoretical properties apply to the implemented correction? We additionally analyze the **actual trajectory-aware correction**, rather than relying solely on the fixed-weight smoothing comparator. Let $\mathbf q^{(t)}=\hat{\mathbf c}^{(t)}-\mathbf c^{(t)}$. Its change in expected squared error satisfies the exact identity $$\mathbb E\!\left\|\mathbf c^{(t)}+\mathbf q^{(t)}-\mathbf u^{(t)}\right\|_2^2 -\mathbb E\!\left\|\mathbf c^{(t)}-\mathbf u^{(t)}\right\|_2^2 =2\mathbb E\!\left\langle\mathbf c^{(t)}-\mathbf u^{(t)},\mathbf q^{(t)}\right\rangle +\mathbb E\!\left\|\mathbf q^{(t)}\right\|_2^2.$$ Thus, the benefit depends on whether the correction aligns with the underlying attribution error, a relationship that the reliability-gating diagnostics probe empirically. Because $\psi$ centers each batch-level signal and matches its standard-deviation scale to the raw contributions, the implemented correction has two immediate properties before any subsequent clipping: $$\sum_{i\in B_t}\hat c_i^{(t)} =\sum_{i\in B_t}c_i^{(t)}, \qquad \left\|\hat{\mathbf c}^{(t)}-\mathbf c^{(t)}\right\|_2 \leq\frac{3\alpha}{2}\sqrt{|B_t|}\,\sigma\!\left(\mathbf c^{(t)}\right).$$ Here the bound uses the population-standard-deviation convention and the scale-matching normalization; with an added numerical stabilizer in $\psi$, the bound remains valid. For the historical-only configuration, the coefficient is $\alpha$ instead of $3\alpha/2$. The implementation reads accumulated **raw** contributions and applies a finite-pass, batch-wise correction rather than recursively diffusing corrected scores. Accordingly, these properties directly characterize the implemented update. The sparse pairwise selection objective additionally admits an explicit marginal gain and diminishing returns, supporting efficient greedy selection under a fixed budget. We have incorporated these distinctions into the revised analysis to align the theoretical statements with the actual algorithm and experimental configurations.
Reviewer_csXWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends the idea of Shapley-value-based attribution to evaluating the contributions of individual training samples in machine learning. The authors introduce a knowledge network to model relationships among training samples, aiming to address two limitations of existing methods: attribution instability and redundancy in data selection. Overall, this is a complete and interesting work with a meaningful methodological extension.

Critical strengths

Good writing and presentation. The first three sections are particularly well written. The authors clearly motivate the research problem, explain the limitations of existing approaches, and develop a coherent narrative leading to their proposed method. The paper is also well illustrated, with informative figures and generally good presentation.

Interesting methodological design. Introducing relationships among training samples into In-Run Data Shapley is a reasonable and interesting idea. The method is closely connected to the problems identified in the introduction, and its components are presented through a clear sequence of formulations and explanations. I particularly appreciate how the same graph structure is used for both attribution stabilization and diversity-aware data selection.

Critical weaknesses

Appendix organization and references.

The paper provides a fairly detailed appendix, including theoretical derivations, complexity analysis, ablation studies, and additional experiments. However, I could not find explicit references to the appendix in the main text. This is rather unusual, especially since some important methodological details and qualifications are only discussed in the appendix.

  1. The utility function and the interpretation of attribution need further clarification.

Although the utility function is formally defined in Section 2, its operational meaning and relationship to the final attribution scores remain somewhat unclear. In particular, the proposed method accumulates optimizer-aware gradient signals along a specific training trajectory, with the validation set defining the objective against which these updates are evaluated.

I would appreciate a clearer explanation of how these trajectory-dependent contributions relate to the final performance of the trained model. To what extent do the scores depend on the choice of validation data and utility objective? These questions are important because they concern the interpretation and practical validity of the attribution results.

  1. The knowledge network and its theoretical justification.

I find the term "Knowledge Network" somewhat overstated. As far as I understand, the proposed network is essentially a weighted KNN graph constructed from semantic similarity between sample embeddings. A more precise description, such as a semantic similarity graph, might better reflect the actual methodology.

More importantly, despite being the central innovation of the paper, the graph correction lacks sufficient theoretical justification. The appendix establishes some useful properties, such as score-sum preservation and computational complexity, but does not demonstrate why graph propagation should necessarily improve attribution stability or faithfulness. In particular, semantic similarity does not directly imply similarity in marginal contributions or second-order interactions. I believe this gap deserves further discussion.

  1. Experimental limitations and interpretation of the results.

The experiments cover several evaluation dimensions, which is appreciated, but I have some concerns:

Table 1 does not report the validation accuracy obtained by training on the complete dataset (100%). This baseline would help readers understand the practical effectiveness of data selection relative to full-data training. Some reported improvements are relatively modest, particularly for Noise AUROC and attribution stability on Pile. The claims should better reflect the magnitude of these improvements. Section 4.4 raises an important concern about the fairness and interpretation of the retrieval comparison. As the authors acknowledge, the Graph-aware results are obtained through a fusion of semantic and lexical retrieval with graph-regularized scores, rather than graph correction alone. It is therefore difficult to determine how much of the improvement actually comes from the proposed attribution method. A clearer description of the retrieval protocol and a controlled ablation would be helpful.

Questions for authors

My main conceptual question concerns the interpretation of the proposed attribution mechanism and its relationship to Shapley values.

The original Data Shapley framework (Ghorbani and Zou, 2019) defines data contributions through a cooperative game based on the marginal effect of training samples on model utility. This provides a clear counterfactual interpretation and a set of desirable axiomatic properties.

In-Run Data Shapley (Wang et al., 2025) changes this perspective by evaluating contributions along a particular training trajectory rather than through repeated retraining. While exact Shapley values can still be defined for the resulting local games, their interpretation is different from that of the original retraining-based Data Shapley.

The authors clearly identify the observed optimization trajectory as the basis of their attribution method. However, the proposed Graph-aware extension introduces an additional transformation based on neighboring samples. Although this may improve the practical usefulness of attribution scores, I believe two conceptual questions deserve further discussion:

How should the graph-corrected scores be interpreted in terms of Shapley axioms? Can they be justified as Shapley values under a well-defined cooperative game, or should they be regarded as Shapley-inspired attribution scores with additional graph regularization? Does graph correction improve attribution faithfulness, or does it mainly produce more stable scores that are useful for downstream data selection? I believe this distinction is particularly important and deserves more direct evaluation. More broadly, this relates to a fundamental distinction in training data attribution between the quantity estimated by an attribution method and the counterfactual effect it is intended to explain. Bae et al. (2022), for example, demonstrate that practical influence-function estimates may approximate a different mathematical object from the leave-one-out retraining effect that originally motivates them. Although the methodology here is different, I believe a similar distinction is worth considering: the proposed graph-corrected scores may be useful for data selection without necessarily providing a more faithful estimate of the original Shapley-based contribution.

I do not consider this necessarily a flaw in the practical motivation of the method. However, I would encourage the authors to clarify its conceptual positioning and distinguish more explicitly between Shapley-based counterfactual attribution, training-dynamics-based attribution, and downstream data selection utility.

References mentioned:

Ghorbani and Zou (2019). Data Shapley: Equitable Valuation of Data for Machine Learning. ICML. Wang et al. (2025). Data Shapley in One Training Run. ICLR. Bae et al. (2022). If Influence Functions Are the Answer, Then What Is the Question?. NeurIPS.

4 comments
AuthorsOpenReview ↗

Clarifying Trajectory-Based Attribution, Downstream Utility, and Controlled Retrieval Evidence

Thank you for the constructive comments. We have clarified the interpretation of trajectory-based attribution, strengthened the connections between the main text and appendix, and added controlled experiments on full-data training, validation-objective dependence, and retrieval components. We distinguish results from the submitted manuscript from the additional small-scale controls throughout. All revisions made in response to your comments are highlighted in green in the revised manuscript.

Q1. Appendix organization and main-text references

We have strengthened the main-text explanations and added direct references to the supporting analyses and experimental protocols in the appendix:

Main-text topicReferenced appendix material
Optimizer-aware contributionsLocal-game derivation and approximation details
Graph correctionScale alignment, score-sum preservation, and reliability conditions
Selection and efficiencyExact marginal selection rule and complexity accounting
Noise detection and stabilityComponent ablations, propagation diagnostics, and representative cases
Language experimentsCorruption procedure, clean-data retraining, and baseline definitions
Retrieval interpretationFusion protocol and controlled component comparison

The main text now also defines the operational utility and explains how graph-corrected scores should be interpreted, making the central claims accessible without relying on appendix material alone.

Q2. What do the scores explain, and how do they relate to Shapley values?

Our scores quantify validation-objective-specific contributions along the observed training trajectory. For GPT-2, the utility is the negative mean next-token cross-entropy on designated validation documents. A positive raw contribution indicates that the sample-conditioned update is locally aligned with reducing this validation loss. Accumulated scores summarize contributions along the observed run; token accuracy after retraining on selected subsets provides a complementary measure of downstream utility.

QuantityInterpretationCorresponding validation
Retraining-based Data ShapleyMarginal utility averaged over retrained coalitionsCoalition-retraining comparison
Exact local-game ShapleyAllocation of the specified trajectory-conditioned local gameExact local-game allocation
Implemented raw in-run scoreOptimizer-aware Taylor approximation along the observed runApproximation validation
Graph-corrected scoreShapley-inspired trajectory attribution enhanced by graph-based structural evidenceStability, corruption diagnostics, and downstream evaluation

This formulation identifies the mathematical and empirical role of each score. In particular, batch centering preserves the aggregate raw proxy contribution while graph correction incorporates neighborhood information; the corrected score is interpreted as trajectory attribution rather than an exact retraining-based Shapley value. This distinction is consistent with the importance of specifying an attribution method's target, as discussed by Bae et al. (2022).

Validation-objective dependence. To characterize how utility targets affect attribution, we varied only the validation query on the same recorded trajectory in an additional 256-candidate, 12-query, three-seed control:

Query-pair comparisonRaw score Spearman, mean (SD)Graph score Spearman, mean (SD)
Same designated source, different query wording/type0.5686 (0.0411)0.5767 (0.0405)
Different designated sources0.5088 (0.0336)0.5181 (0.0369)

These descriptive statistics first average dependent query pairs within each seed, then report the mean and SD across seeds. Graph correction produces slightly higher rank agreement in both settings. The distinction between the two query categories further illustrates that the scores reflect usefulness relative to the chosen validation objective, a desirable property for task-oriented data valuation.

Q3. Semantic graph, reliability, and attribution evidence

We now consistently use the term semantic similarity graph. Embedding similarity supplies a sparse structural prior for sample relationships, while historical contributions and, where enabled, gradient-agreement gating determine how this evidence informs attribution. The following CIFAR-10 results evaluate the attribution scores before diversity selection, isolating the contribution of graph-based correction.

CIFAR-10 attribution-only diagnostics, nine data–training seed combinations; mean (SD).

MethodNoise AUROC ↑Cross-seed Spearman ↑
Adam-aware first order0.571 (0.009)0.700 (0.016)
Graph without gate0.558 (0.014)0.721 (0.015)
Graph without history0.584 (0.009)0.703 (0.016)
Graph without pair evidence0.636 (0.009)0.727 (0.014)
Full Graph0.647 (0.009)0.733 (0.014)

The full gated variant improves noise AUROC from 0.571 to 0.647 and cross-seed Spearman from 0.700 to 0.733 relative to Adam-aware first order. The propagation diagnostics help explain this result: mean gated incoming mass is 0.467 for clean samples versus 0.043 for mislabeled samples. At 20% retention, graph-corrected attribution ranking lowers selected corruption from 0.073 to 0.050, before applying the diversity selector. These findings demonstrate that graph-based evidence can strengthen stability and corruption discrimination when its reliability is controlled. The appendix additionally provides conditional smoothing analysis to characterize the role of neighborhood agreement, and distinguishes these diagnostics from direct comparison with retraining-based Shapley values.

Q4a. Full-data training reference

To contextualize subset-selection utility, we added a 100% training reference in a separate clean-data control with 256 training documents, 16 disjoint utility documents, and 112 disjoint evaluation documents, evaluated over three seeds. Each configuration fine-tunes all GPT-2 parameters for one epoch; the comparison fixes epochs rather than the number of optimizer updates.

Retention / methodEvaluation accuracy (%) ↑Evaluation loss ↓
20% Random40.7778 (0.0679)3.3238 (0.0034)
20% Adam-aware40.8063 (0.1471)3.3136 (0.0086)
20% Graph + diversity40.8492 (0.2072)3.3202 (0.0114)
80% Random41.4251 (0.0805)3.2606 (0.0052)
80% Adam-aware41.5179 (0.2597)3.2543 (0.0064)
80% Graph + diversity41.4632 (0.1035)3.2561 (0.0016)
100% full data41.5917 (0.1592)3.2520 (0.0026)

The full-data reference provides a useful anchor for interpreting the retention–performance trade-off. In this small-scale control, the 80% Graph subset achieves 41.4632% accuracy, compared with 41.5917% for full-data training, a difference of 0.1285 percentage points. This result is reported as an additional control rather than as a 100% entry for the separate 100k-document experiment.

Q4b. Quantifying improvements across evaluation objectives

The strongest language-model evidence concerns downstream utility from clean-data subset selection. In the submitted 100k-document Pile experiment, Graph-aware first-order attribution consistently improves token accuracy over the corresponding Adam-aware estimator at every retention level.

Submitted 100k clean Pile experiment: first-order token accuracy (%), mean (SD), three runs.

Method20%40%60%80%
Adam-aware41.055 (0.005)41.180 (0.011)42.304 (0.007)43.366 (0.007)
Graph41.799 (0.020)42.132 (0.009)43.308 (0.021)44.345 (0.015)
Gain (percentage points)+0.744+0.952+1.004+0.979

These improvements of 0.744–1.004 percentage points are obtained from clean data without synthetic corruption. Complementary robustness results show that, under the main Pile protocol, noise AUROC changes from 0.560 to 0.563 and cross-seed Spearman from 0.164 to 0.183. We report the magnitude of each result directly and distinguish the Pile subset-selection gains from the larger CIFAR-10 attribution improvements. Together, they characterize the method across downstream utility, attribution stability, and corruption discrimination without conflating these evaluation targets.

Q4c. Retrieval fairness and controlled fusion ablation

We have clarified that the original retrieval results concern a semantic/lexical fusion system, with graph-derived scores serving as one potential component. To separate component effects, we added a controlled comparison using the same 256 candidates, 12 fixed repository queries, initial pretrained encoder, visible 128-token windows, and three training seeds. The fusion thresholds are held fixed from the original protocol without tuning on target labels.

Mean designated-source rank (lower is better). Similar-topic queries have no unique provenance source; their ranks are presented as descriptive retrieval diagnostics.

MethodPartial exactParaphraseStrong paraphraseSimilar topic
BM251.001.001.0057.33
Embedding cosine1.001.006.0023.67
Raw attribution only3.787.7823.2259.56
Graph attribution only3.897.8928.4466.89
Fusion without attribution1.001.001.0041.33
Fusion + raw attribution1.001.001.0041.33
Fusion + graph attribution1.001.001.0041.33

The controlled decomposition establishes the contribution of the individual retrieval components and makes the system-level interpretation explicit. Under the inherited fusion thresholds, the attribution component receives zero weight for all 12 queries, so the three fused configurations have identical ranks in this control. We therefore evaluate graph-aware attribution primarily through the independent attribution ablations and clean-data selection experiments, while presenting fusion as a complementary retrieval application. Per-query ranks, component scores, fusion weights, and per-seed results are documented in the additional experiment report.

Thank you for the detailed responses. The new simulation results look good to me. However, my concern on the theoretical justification has not been well-addressed. Thus, I'd maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.