- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Graph-Aware In-Run Data Shapley, a data-valuation method that augments one-run Data Shapley (ICLR '26 and its Adam optimizer-aware variant) with a sample-level semantic knowledge network. Standard in-run valuation accumulates each sample’s contribution independently, so scores can be noisy and high-value near-duplicates can dominate selection. The proposed method builds a kNN graph over sample embeddings, adds an in-batch neighbor relation as a lightweight surrogate for second-order interactions. It then reuses the same graph for diversity-aware subset selection with a redundancy penalty, by a greedy algorithm. Experiments on Pile/GPT-2 and CIFAR-10 show improved attribution stability, corruption detection, selected-subset accuracy/diversity, and efficiency.
Critical strengths
Apply a well-known diversification framework to in-run Shapley and show improvements.
Critical weaknesses
-
The idea is very simple and incremental. Graph-based or more generally diversification frameworks are really well known and everywhere. The main idea of the paper is to instantiate the in-run Shapley value as the utility in this framework.
-
Analysis is relatively superficial. For example, why does graph help Noise AUROC metric in practice?
-
Experimental Details are missing both from the text and the code. How to corrupt text data? Very importantly, how do you apply KNN-Shapley to text data, which as far as I know can only be used for traditional classification or regression tasks, like mislabeled images? ALso, do you train on corrupted text data (after selection) to report the metric Val. Accuracy? Can you do the same without corruption?
-
Not enough details are given in the source code to reproduce the experimental results other than the authors' own implementation.
Questions for authors
See weaknesses.
4 comments
Q2. Why does graph correction help noise AUROC?
Graph correction uses neighborhood corroboration over the training trajectory. Historical messages accumulate relational evidence across steps, while the gradient-agreement gate used in the CIFAR-10 diagnostic attenuates inconsistent neighbor contributions. Table 1 quantifies these effects: relative to Full Graph, removing history reduces AUROC by 0.063, and removing gating reduces it by 0.089. Removing the in-batch pair signal also reduces AUROC from 0.647 to 0.636.
Table 4. CIFAR-10 propagation and selection diagnostics. Gated incoming mass and promotion refer to Full Graph. The 20%-retention contamination comparison uses raw versus graph-corrected attribution rankings without diversity selection.
| Diagnostic | Clean samples | Mislabeled samples |
|---|---|---|
| Mean gated incoming mass | 0.467 | 0.043 |
| Fraction promoted by Graph | 0.566 | 0.021 |
| Selection diagnostic | Adam-aware | Graph |
|---|---|---|
| Selected corruption fraction ↓ | 0.073 | 0.050 |
Clean samples receive substantially more gated neighborhood evidence than mislabeled samples, and the selected corruption fraction decreases by 31.5%. The parameter-sensitivity analysis reinforces this pattern: increasing the correction strength from to raises CIFAR-10 AUROC from 0.571 to 0.708 and lowers selected contamination from 0.073 to 0.023. The default configuration uses .
Q3a. Exact text-corruption procedure
For the Pile noise-detection experiment, we select 10% of training documents without replacement using NumPy's default random generator seeded with run_seed + 991. For each selected recipient, define , where is its whitespace-tokenized word count. We combine the first words from a randomly assigned donor with recipient words at positions through , shuffle the resulting word sequence, and tokenize/truncate it to 128 GPT-2 tokens. Donors are assigned through a random permutation, and shorter donors can produce shorter mixtures. Validation documents remain clean. Corruption indicators are used only for evaluation, with lower attribution utility indicating greater likelihood of corruption. The revised appendix specifies the complete procedure.
Q3b. KNN-Shapley baseline definition on text
We have clarified the implementation and naming of the embedding-based retrieval baseline. The 100k-document entry previously labeled “KNN-Shapley” computes cosine similarity between each document embedding and the normalized mean embedding of the utility documents. We therefore relabel it Semantic Retrieval and explicitly state its scoring rule; it should not be interpreted as an implementation of the KNN-Shapley recurrence. Other simplified gradient baselines are likewise identified by their actual scoring functions. These nomenclature corrections do not change the reported scores or the matched Adam-aware versus Graph-aware comparisons, and we do not rely on the relabeled entry to claim superiority over a true KNN-Shapley implementation.
Q3c–d. Corrupted-text retraining and clean-data evaluation
The main-table subset-selection experiments use clean Pile text for attribution, selection, and retraining. Noise AUROC is evaluated separately using synthetically corrupted text. For subset evaluation, selected documents are used to fine-tune pretrained GPT-2 for one epoch with all model parameters trainable; during attribution, only the final LayerNorm is trained. The accuracy metric is next-token top-1 accuracy over nonpadding target tokens.
Table 5. Clean Pile subset selection (three runs): token accuracy (%) mean (SD). Gains are Graph minus the corresponding Adam-aware estimator, in percentage points.
| Method | 20% | 40% | 60% | 80% |
|---|---|---|---|---|
| Adam-aware first order | 41.055 (0.005) | 41.180 (0.011) | 42.304 (0.007) | 43.366 (0.007) |
| Graph first order | 41.799 (0.020) | 42.132 (0.009) | 43.308 (0.021) | 44.345 (0.015) |
| Gain | +0.744 | +0.952 | +1.004 | +0.979 |
| Adam-aware second order | 40.803 (0.021) | 41.143 (0.016) | 43.261 (0.010) | 43.351 (0.019) |
| Graph second order | 41.825 (0.011) | 42.126 (0.025) | 43.275 (0.004) | 44.350 (0.021) |
| Gain | +1.022 | +0.983 | +0.014 | +0.999 |
Graph-aware selection improves token accuracy in all eight attribution-order × retention-budget settings, establishing downstream benefits independently of synthetic text corruption.
Q4. Reproducibility details
We have consolidated the key settings in the revised protocol documentation:
| Item | Explicit protocol |
|---|---|
| Data | 100k training / 10k held-out Pile documents; 128 tokens |
| Attribution | Final LayerNorm, 1,536 parameters; batch 8; AdamW learning rate |
| Utility | 16 fixed domain-stratified documents for trajectory gradients |
| Optimizer | Betas (0.9, 0.95); weight decay 0.1; gradient clipping 1.0 |
| Graph | 10 neighbors; ; recorded one-pass score replay for Graph v2 |
| Pair term | Included in second-order Graph v2; first-order configuration uses historical correction |
| Selection | Symmetric-graph greedy; score SD |
| Subset retraining | All parameters; one epoch; batch 16; learning rate |
| Repeats | Seeds 0, 1, 2; per-seed outputs for new diagnostics |
The anonymous repository https://anonymous.4open.science/r/graph-aware-shapley-1244/ is the reference location for the experimental implementation. The revised reproduction materials document baseline scoring rules, corruption metadata, reproduction commands, executable protocol checks, and the two-GPU (A100) diagnostic workflow, including checkpoint hashes and per-seed outputs. Files not yet present in the anonymous release are identified as prepared revision materials rather than currently accessible artifacts.
In summary, the evidence supports three complementary contributions: (i) graph-based correction improves trajectory-level attribution stability, (ii) graph-guided selection yields stronger downstream training subsets across retention budgets, and (iii) sparse structural interactions can be computed efficiently. We appreciate the opportunity to clarify these points and the experimental conditions under which each result is obtained.
We thank the authors for their replies. I appreciate the experimental details. Also, renaming “KNN-Shapley” to Semantic Retrieval now makes sense; it is important to go over literature papers before using them. However, I maintain my main assessment of the methodology.
Beyond Diversification: Better Attribution, Up to 1.29 Percentage Points Higher Accuracy, and 47% Less GPU Memory
Thank you for the helpful comments. We have sharpened the methodological contribution, clarified the experimental protocols and baseline definitions, and consolidated the supporting evidence below. Tables 1–5 reproduce results from the submitted manuscript. All revisions made in response to your comments are highlighted in red in the revised manuscript.
Q1. Contribution beyond existing diversification frameworks
Our central contribution is graph-based correction of trajectory-level attribution before data selection, rather than diversification alone. Historical neighbor evidence incorporates cross-step sample relations into Adam-aware contributions; an in-batch pair signal can additionally account for interactions among examples evaluated together. The corrected scores then support redundancy-aware selection using the same sparse sample graph. Crucially, the attribution-only results in Table 1 assess this correction before diversity selection, isolating its effect on valuation.
Table 1. CIFAR-10 attribution ablations (three data seeds × three training seeds). AUROC and cross-seed Spearman are reported as mean (SD); no diversity selection is applied.
Beyond attribution quality, the complete pipeline jointly improves downstream utility and selected-subset diversity at all four low-data budgets. Table 2 reports first-order results. For the second-order variant, the corresponding token-accuracy gains are 1.2940, 1.1872, 0.9802, and 0.4345 percentage points at 1%, 2%, 5%, and 10% retention, respectively.
Table 2. Pile low-budget selection: In-Run + Top-k versus the complete Graph pipeline. This experiment evaluates attribution and selection jointly; arrows indicate baseline → Graph.
The sparse graph also makes structural interaction correction computationally practical. We benchmark the HVP-free online Graph implementation on 1,000 Pile documents under the specified three-seed protocol; these measurements characterize this implementation rather than the separate 100k-document experiment.
Table 3. Online GPT-2 efficiency on 1,000 Pile documents (three seeds). Reported times include graph construction where applicable.
Under this protocol, Graph achieves 72.7× faster interaction computation, 5.6% lower total runtime, and 47.1% lower peak memory. The graph interactions serve as an efficient structural surrogate for sample dependencies without explicit Hessian-based interaction computation. Together, Tables 1–3 distinguish our contribution from a selection-only diversification method: the graph first improves attribution and then guides subset selection.