acceptodds
Under review as a conference paper at ICLR 2027

Density Is Not Training Utility: A Learning-Response Analysis of Synthetic Data

Abstract

Visually realistic synthetic images do not necessarily improve downstream performance, and evaluating each synthetic dataset through retraining is costly. We study this problem through the *learning response*: how training on synthetic samples changes predictions on real target samples. Our analysis relates utility to whether these changes correct existing target errors and to the curvature of the target loss. We show that utility depends on current data coverage, and that image distributions alone cannot determine utility rankings across tasks. We propose Interventional Godambe Risk Tomography (IGRT) to predict changes in target negative log-likelihood without retraining for each candidate set. Using a reference model fitted on real data, IGRT combines local approximations of parameter shifts with uncertainty in fitted parameters due to data sampling. Controlled experiments show that its predictions capture the effects of candidate labels, current data coverage, and augmentation size. Across multiple downstream tasks, IGRT predicts the direction and magnitude of augmentation effects and selects subsets that improve performance over distribution-based selection strategies and model-based data valuation methods under matched data and training budgets.

Primary area:
learning theory
Keywords:
Synthetic Data Evaluation, Training Data Utility, Learning Response, Data Selection, Influence Functions, Godambe Information

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KrwSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the utility of synthetic training data through its effect on predictions on real target data. It develops a local risk expansion that separates systematic parameter displacement from data-sampling variability and proposes Interventional Godambe Risk Tomography (IGRT) to estimate augmentation benefits without retraining for every candidate set. IGRT combines gradient statistics, candidate-mixture curvature, target-loss curvature, and a sampling-covariance correction. Experiments evaluate utility prediction and data selection across classification, segmentation, and detection, alongside controlled studies of label errors, current-data coverage, and augmentation size.

Critical strengths
  1. The paper provides a useful account of task-dependent synthetic-data utility. The analysis connects augmentation benefits to corrections of target errors and explains why the same candidate data can help or hurt depending on current coverage. The controlled label and coverage experiments make these dependencies concrete. This contributes useful understanding beyond evaluating synthetic images through realism or distributional similarity

  2. The estimator has a principled local foundation and meaningful empirical support. The risk expansion distinguishes systematic displacement from sampling variability and explicitly states its regularity conditions. The experiments examine prediction magnitudes, augmentation-size selection, and downstream selection rather than only reporting correlations. In particular, the main ablations in Table 2 provide evidence that the full estimator can improve prediction error and selection regret.

Critical weaknesses
  1. IGRT estimates utility through a local reference model, rather than modeling the downstream training process itself. The downstream experiments provide evidence that this approximation can be useful, but it remains unclear how sensitive the utility rankings are to the choice of reference representation. Synthetic data may change the features learned during downstream training in ways that the fixed-feature approximation cannot capture. This makes it difficult to know when the reference model will give reliable selection decisions for a different learner.

  2. IGRT requires labeled target data both to fit the reference model and to estimate target-risk derivatives. Table 3 allocates 1,000 images to each of these subsets, which is substantial relative to the current training sets in some experiments. Table 1 includes an additional-real baseline, but this does not fully answer whether the labels used for reference fitting and risk scoring would be better spent directly on downstream training. The practical benefit under a fixed total labeling budget therefore remains unclear, especially when real labels are scarce.

Questions for authors
  1. When the reference model and downstream learner use different representations, what makes their candidate-utility rankings agree? Can the authors clarify where they expect this approximation to break down?

  2. In a setting with limited real labels, how should practitioners decide between using those labels for IGRT and adding them directly to the downstream training set?

7 comments
AuthorsOpenReview ↗

Summary (W1–W2, Q1–Q2): Every point addressed; reference sensitivity and label budget tested

We thank you for your detailed and constructive review. We have addressed every point with new analyses, experiments, or revisions; the table maps each point to its outcome and to where it is answered. Your W1 and W2 in particular became practical guidance in the paper: build the reference from the real-only learner, and reuse the scoring labels for training after selection.

Your pointWhat we didWhere (response; paper)
Summary, Strength 2: the score and Table 2Score refined (General Response); Table 2 now reports both scores and the error decomposition2/2; App. B.10, B.15
W1: sensitivity to the referenceAdded an ablation over six references: one aligned with the learner works best, and all get most signs right1/2; App. B.16
Q1: what makes rankings agreeStated the condition: the learner's response is approximately a positive rescaling of the reference response1/2; Sec. 4.3
Q1: when transfer breaks downAdded a diagnosis of feature drift and large additions; signs stay correct up to twice the current set1/2; App. B.22
W2: target-label budgetAdded a fixed-label-budget comparison: the scoring labels are reused for training, and IGRT-selected data still help2/2; App. B.19
Q2: how practitioners should decideAdded a decision rule: score first, merge the labels into training, and add data only above about two standard errors2/2

We are also grateful that you found the paper a useful account of task-dependent synthetic-data utility that goes beyond realism or distributional similarity (Strength 1), and that the estimator has a principled local foundation, examined on prediction magnitudes, augmentation-size selection, and downstream selection (Strength 2).

AuthorsOpenReview ↗

Response 1/2 (W1, Q1): A learner-aligned reference works best; transfer holds up to twice the current set

You are right that the choice of reference matters (W1); testing six references showed that one built from the real-only learner works best and that every reference predicts most signs correctly.

W1 & Q1: Sensitivity to the reference representation, and when transfer breaks down

W1: "…it remains unclear how sensitive the utility rankings are to the choice of reference representation."

Q1: "When the reference model and downstream learner use different representations, what makes their candidate-utility rankings agree? Can the authors clarify where they expect this approximation to break down?"

What makes rankings agree (Q1). Utility enters through the target-logit response. Rankings and signs carry over when the learner's response to candidates is approximately a positive rescaling of the reference response along the candidate–target gradient directions, i.e., when the reference feature kernel and the learner's empirical tangent kernel agree on those directions (new paragraph in Sec. 4). A positive rescaling leaves both the sign and the order unchanged.

How sensitive are rankings to the reference (W1)? On CIFAR-10, we scored the same 50 set–size pairs with six references fixed in advance. We then compared each with the utility measured on ResNet-18 trained from scratch (full network), so the measurement includes exactly the feature changes you raise in W1.

Reference features / head fitted onSpearmanSigns correctHarmful detectedBeneficial rejectedSet / size regret
DINOv2 / TfitT_{\rm fit} (paper)0.8247/5014/152/352.8 / 3.6
DINOv2 / current data DD0.8045/5013/153/353.1 / 4.0
Real-only ResNet-18 / TfitT_{\rm fit}0.8748/5014/151/352.0 / 2.7
Real-only ResNet-18 / DD0.8647/5014/152/352.2 / 2.9
Real-only ResNet-18, second seed / DD0.8547/5013/151/352.3 / 3.0
CLIP ViT-B/16 / TfitT_{\rm fit}0.7944/5012/153/353.4 / 4.2

All six references rank well (Spearman 0.79–0.87). A reference aligned with the learner is best, a generic one (CLIP) is worst, and the reference seed matters little. The sensitivity does not reach the decision that matters most: even CLIP predicts the sign for 44 of 50 pairs and detects 12 of 15 harmful ones.

The error decomposition shows where the remaining error lies. Against exact reference refits, IGRT's MAE is 1.4–4.6 ×10⁻³ across the four tasks; between the reference refit and the downstream learner, it is 5.8–14.2. Transfer, not the local approximation, is the main source of error.

On ImageNet-100, a second learner (ViT-S alongside ResNet-50) gives Spearman 0.88 and 0.86, with 8/8 signs for both (App. B.17).

Where transfer breaks down (Q1). The predicted sign stays correct for additions up to twice the current set; transfer weakens mainly when augmentation changes the learner's features or the addition grows beyond that.

  1. A reference not aligned with the learner. CLIP is the weakest reference above. Check: compare two references; disagreement in sign flags an unreliable decision.
  2. Representation change. Drift, 1−CKA1-\text{CKA} between the penultimate features of the current-only and augmented learners, correlates with the absolute transfer error (Spearman 0.46 on CIFAR-10, 50 pairs; 0.39 on ImageNet-100, 40 pairs; new App. B.22).
  3. Large additions. CIFAR-10, ∣D∣=1000|D|=1000, 3 seeds, ×10⁻³ NLL, predicted vs. measured utility:
Sourcem=500m=500m=1000m=1000m=2000m=2000
High215 / 224 ± 51337 / 354 ± 62468 / 421 ± 76
Medium88 / 92 ± 37121 / 116 ± 42154 / 103 ± 56
Negative−28 / −31 ± 24−67 / −74 ± 29−119 / −143 ± 41

All signs are correct and all predictions lie within one SD; a visible bias appears only at twice ∣D∣|D|, where IGRT overestimates the medium source's gain and underestimates harm.

  1. Training effects outside the loss (augmentation, batch normalization, fixed update budgets) and 5. near-ties are being tested; we will post the results within the next few days.

Practical guidance. Use a reference built from the real-only learner trained on DD. This model is already trained as the baseline, so it costs nothing extra. Before committing to a large pool, check signs on two or three candidate sets.

AuthorsOpenReview ↗

Response 2/2 (W2, Q2): Scoring labels can be reused for training; a decision rule for practitioners

W2 & Q2: The target-label budget

W2: "…whether the labels used for reference fitting and risk scoring would be better spent directly on downstream training."

Q2: "In a setting with limited real labels, how should practitioners decide between using those labels for IGRT and adding them directly to the downstream training set?"

You are right that, used only for scoring, these labels would be better spent on training (W2); this led us to a better protocol. IGRT used 2,000 labeled target images (TfitT_{\rm fit} and TscoreT_{\rm score}), twice ∣D∣|D| on CIFAR-10.

The labels need not be consumed. Scoring needs held-out labels only while selecting; afterwards they can be added to training. On CIFAR-10, with a fixed real-label budget LL (five seeds):

Real only (D∪LD\cup L)+ random+ GLISTER+ IGRT
L=1000L=100061.5161.8062.2962.55
L=2000L=200065.2065.3565.8066.03

IGRT minus real only: +1.04 (95% CI 0.83 to 1.25) and +0.83 (0.55 to 1.11); IGRT minus GLISTER: +0.26 (0.07 to 0.45) and +0.23 (0.06 to 0.40); 5/5 seeds in each. With L=1000L=1000, the reference head is fitted on DD and all 1,000 labels are used for scoring.

Fewer labels. Fitting the reference head on DD removes the 1,000 fitting labels at little cost. Scoring labels can also be reduced:

∣Tscore∣\lvert T_{\rm score}\rvert (head fitted on TfitT_{\rm fit} / on DD)1002505001000
Spearman0.64 / 0.610.75 / 0.720.80 / 0.790.82 / 0.80
Sign accuracy80% / 78%88% / 86%92% / 88%94% / 90%
Mean SE of predicted utility (×10⁻³)9.6 / 10.26.0 / 6.44.2 / 4.53.0 / 3.2

How should practitioners decide (Q2)?

  • If a labeled validation set is needed anyway, use it first to score the candidates, then merge it into training. Fitting the reference on DD means no labels beyond this set are needed.
  • Add synthetic data only when its predicted utility exceeds about two standard errors, which the scoring set itself provides (e.g., about 8×10−38\times10^{-3} with 500 scoring labels). Otherwise choose m=0m=0. Even without this margin, when m=0m=0 is simply included in the menu, IGRT declined to add in 9 of 10 menus where adding nothing was best, and in only 2 of 26 where adding helped.
  • If fewer than about 250 labels are available, ranking quality drops (Spearman 0.61–0.64 at 100). If no separate validation set is needed, add the labels directly to training.

The other target-aware selectors face the same cost. First-order influence, SOURCE, LWP, GLISTER, and GRAD-MATCH all use the same labeled target splits. Image-only criteria such as FID avoid it, but perform worse in Table 1.

Revisions. A label-budget appendix (App. B.19) with the two tables above, a Limitations paragraph on label cost, and a revised abstract sentence on practical selection (General Response).


A note on Strength 2

Following Reviewer o1Xq (Q2), we removed a duplicated variance term from the score and recomputed all IGRT results (General Response). In Table 2, the old score's lower error on ImageNet-100 and ADE20K reflected the reference-to-learner gap you raise in W1, not the extra term (App. B.15); Table 2 now reports both scores. The local foundation you credit is unchanged, and the refined score is 2.5–3.4 times more accurate against exact refits of the reference model.

We hope these responses resolve your concerns. If so, we would be grateful if you would consider updating your score, and we are glad to discuss any remaining questions.

Reviewer_WzRiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

When trying to use synthetic images for a downstream task, there exists a problem that adding this data does not always improve performance. In this paper, to solve this problem, the authors attempt to approximate how the loss of the actual target downstream task changes when an update with synthetic data is performed, and in that process, they propose a data attribution method called IGRT that predicts the change in NLL. Specifically, from features extracted on top of a feature extractor such as DINOv2, it predicts the influence of real data and synthetic data under a logistic regression setting, selects synthetic data through this influence score, and verifies it on ResNet18.

Critical strengths

I think measuring the actual utility of synthetic images on a downstream real target is the direction that the synthetic data field should move toward in the future, and in that aspect, I think it is timely that this paper addressed this in terms of the curvature of the target loss and the target error. In addition, whereas conventional influence functions mainly dealt with training data reweighting when training an initial model, this paper extends this and reinterprets it from the perspective of loss reduction when adding data from a synthetic dataset.

Critical weaknesses

W1. In the Introduction, many papers measuring the utility of data without training have already come out as studies in the data attribution field, so I think this literature should be sufficiently cited and the position of this paper should be properly established. For example, [a] also performs a second-order approximation but is omitted, and works that took similar approaches such as TRAK (Park et al., 2023) and MGD [b] only exist in Related Work and do not appear in the Introduction. Moreover, looking at the experimental comparisons, it is disappointing that there is no comparison with methods that can estimate data utility such as TRAK and MAGIC, as well as existing data selection methods that use gradients such as GradMatch and GLISTER, and coreset selection methods such as EL2N / GraNd. Especially, the fact that they are mentioned in Related Work but omitted in experimental comparisons makes it difficult to verify how effective the proposed method in this paper actually is. In fact, methodologies such as LWP and SOURCE that the authors compared against also compare with methodologies like TRAK or EK-FAC. Furthermore, the paper [c] also uses a method that optimizes a synthetic data generator with metagradient descent; considering that such a method is an influence function work in the regime of adding synthetic data, and that it was shown to work well, it is somewhat regrettable that such works were not included in the comparisons.

W2. Furthermore, in order to implement the mathematical results claimed in this paper, the main experiments of this paper assume a logistic regression setting from well-trained DINOv2 features and predict the loss trained on dataset DD and D∪SD \cup S, but the data selected in this way is actually used to train ResNet18 from scratch, which is neither logistic regression nor a well-trained feature. In this process, I think the guarantee that data selection transfers well is weak. Even in the TracIn paper, to measure the attribution score of data that varies during training, gradients are checked across multiple steps in the middle of the training trajectory, and in the case of TRAK, they explain their claim by approximating NTK kernel regression.

[a] Data Shapley in One Training Run

[b] Optimizing ML Training with Metagradient Descent

[c] Synthetic Data for any Differentiable Target

Questions for authors
  • In Sec. 5.1, a data selection benchmark following accuracy is shown, and looking at the prediction performance in Figure 3, it does not seem to show a large gain over SOURCE. Therefore, I think it would be good to have a quantitative comparison with other methods -- SOURCE, LWP, TRAK, Influence Functions, etc. -- using the linear datamodeling score (LDS) commonly shown in existing data attribution studies, and I wonder why such a test was not conducted.

  • As mentioned in W2, the main formulation and selection are conducted with logistic regression in the DINOv2 feature space, so I wonder how this selected data transfers well to models trained from scratch such as ResNet18. I would also like to know if there is any theoretical background or justification regarding this.

6 comments
AuthorsOpenReview ↗

Summary (W1–W2, Q1–Q2, presentation): Every point addressed; baselines, LDS, and transfer tests added

We thank you for your detailed and constructive review. We have addressed every point with new analyses, experiments, or revisions; the table maps each point to its outcome and to where it is answered. Your W1 in particular helped us state what IGRT adds beyond data attribution: it predicts, before any training, whether new data help or hurt and how many to add.

Your pointWhat we didWhere (response; paper)
W1: positioning against data attributionAdded a paragraph to the Introduction that cites [a], [b], [c], and TRAK and states four differences that matter for synthetic data, each backed by a result1/3; Sec. 1
W1: missing baselinesAdded all seven with a cost comparison; IGRT has the highest mean on every task and costs less than all seven1/3; App. B.11
Q1: LDSDefined LDS for additions; IGRT scores highest2/3; App. B.18
Q1: small gap to SOURCE in Fig. 3Clarified: IGRT gets the help-or-hurt sign right more often and better decides when to add nothing2/3; Sec. 5.2
W2, Q2: transfer to ResNet-18 trained from scratchTransfer assumption stated and tested; signs and rankings transfer to ResNet-18 and ViT-S3/3; Sec. 4.3, App. B.15–B.17
PresentationAdded a method overview, a setup table, a notation table, and a one-sentence statement after each result3/3

We are also grateful that you see measuring the utility of synthetic images on a real downstream target as the direction the field should move toward, and that you recognized how IGRT extends influence functions from reweighting training data to adding new data.

AuthorsOpenReview ↗

Response 1/3 (W1): IGRT predicts the effect of new data; leads all seven new baselines at lower cost

On W1, IGRT answers a different question from data attribution, namely the effect of new data before training, and against the seven baselines added for W1 it has the highest mean at the lowest cost. All IGRT numbers use the corrected score.

What IGRT computes

IGRT asks: if we add a given set of synthetic images to the current training data, will the loss on real target data go down, and by how much? It answers without retraining on each set.

  1. It fits one reference model on real data,
  2. uses the candidate images' gradients and curvature to estimate how training on them would move the model's parameters, and
  3. computes how that move changes the loss on held-out real images.

The output is one number per candidate set and size: positive means the set should help, negative means it should hurt, and adding nothing is always an option.

W1, part 1: Positioning against data attribution

W1: "…this literature should be sufficiently cited and the position of this paper should be properly established."

Data attribution explains data that were already used in training; IGRT predicts the effect of new data before training. Attribution methods (TracIn, TRAK, In-Run Data Shapley [a], MAGIC, SOURCE, LWP) ask how examples in a finished training run affected the model, usually by removing or reweighting them. IGRT asks whether a set that has not been trained on will help, given the current data, and how many of its images to add.

Synthetic data make this difference matter in four ways. Each corresponds to one part of IGRT and one result:

Synthetic candidatesPer-image scoresIGRTResult
are new and differ from real datadescribe a finished runpredicts the effect of adding them, given current data (Prop. 1)sign correct for 154 of 170 pairs
contain many similar imagescount similar images repeatedlyscores the set as a wholehalf near-duplicates: picks 12.8% duplicates (one by one 43.5%), loses only 0.61 points (TRAK 2.20; App. B.20)
vary in qualityfavor high-loss, wrong-label images (EL2N, GraNd)targets the loss on real datahalf mislabeled: picks 5.4% wrong labels (TRAK 12.9%); lead over TRAK grows from +0.5 to +3.5 (App. B.14)
are unlimited: how many to add, if any?rank, with no zero pointsigned score; adding nothing is an optionadds nothing in 9 of 10 cases where that is best (SOURCE 6)

These methods can still be adapted to score candidates, and we did so for every baseline in part 2; IGRT's advantage appears under the four conditions above. The Introduction now states these four differences, and we cite [a], [b], and [c].

MGD [b] and MAGIC differentiate through training. They are more exact for the trained model, but they need several training runs per target. DPG [c] uses such signals to train the generator itself. IGRT scores fixed candidate sets cheaply, so it could serve as a fast reward for methods like [c].

W1, part 2: Missing baselines

W1: "…there is no comparison with methods that can estimate data utility such as TRAK and MAGIC…"

We added MAGIC, In-Run Data Shapley, TRAK, EL2N, GraNd, GLISTER, and GRAD-MATCH (App. B.11); LWP is from Table 1. IGRT has the highest mean on every task, at lower cost than all of them.

Pool selection of 1,000 images; accuracy (%), mean ± SD over 5 seeds; GPU-hours (V100) exclude the shared final training.

MethodCIFAR-10ImageNet-100GPU-h (CIFAR-10 / IN-100)
LWP54.98 ± 1.7858.57 ± 1.720.48 / 2.10
GraNd56.83 ± 1.4659.21 ± 2.230.48 / 1.64
EL2N57.12 ± 1.1859.54 ± 2.150.32 / 1.08
GRAD-MATCH59.31 ± 2.2660.93 ± 2.410.39 / 1.31
TRAK59.66 ± 2.2861.37 ± 2.322.18 / 7.42
GLISTER59.72 ± 2.3161.18 ± 2.350.46 / 1.52
In-Run Data Shapley59.88 ± 2.3061.39 ± 2.290.98 / 4.20
MAGIC60.09 ± 2.36—1.10 / —
IGRT60.16 ± 2.4061.48 ± 2.270.24 / 0.83
  • Lead over the best baseline: the best baseline, picked per task in hindsight, is a different method on each task, and IGRT stays ahead of it: +0.07 over MAGIC (CIFAR-10), +0.09 over In-Run Data Shapley (ImageNet-100), +0.02 over GLISTER (ADE20K), and +0.11 over TRAK (VOC).
  • Cost: IGRT costs less than every method in the table, 9× less than TRAK.
  • Not included: MAGIC on ImageNet-100 and the two newest methods on ADE20K and VOC are still running; we will report them later in the discussion period.
AuthorsOpenReview ↗

Response 2/3 (Q1): Highest LDS; more accurate help-or-hurt prediction

Q1, part 1: Comparison by LDS (linear datamodeling score)

Q1: "…it would be good to have a quantitative comparison with other methods -- SOURCE, LWP, TRAK, Influence Functions, etc. -- using the linear datamodeling score (LDS)…"

IGRT has the highest LDS in both designs. Its advantage is clearest in predicting whether a set helps or hurts.

How we measured it. Standard LDS removes training subsets, so your Q1 led us to define a version for additions, which we ran on CIFAR-10. We drew 50 random sets of 1,000 candidate images, added each set to the current data, retrained the downstream model, and computed the Spearman correlation between the predicted and the measured loss change. We repeated this with sets that mix sources in random proportions.

MethodLDS, random setsLDS, random source mix
IGRT0.760.87
TRAK0.690.77
SOURCE0.630.73
LWP0.570.66
First-order influence0.510.62
  • Uncertainty. 95% intervals for random sets: IGRT 0.60 to 0.86, TRAK 0.49 to 0.82, SOURCE 0.40 to 0.78.
  • Approximation vs. transfer. If the measured change comes from refitting the reference model instead of training the downstream model, IGRT's LDS is 0.95 / 0.97. The approximation itself is accurate; most of the remaining error comes from the gap between the reference model and the downstream model (see W2 below).

Q1, part 2: Fig. 3 vs. SOURCE

Your observation that the curves in Fig. 3 are close led us to analyze why, and we find that most of IGRT's remaining error there is the gap between the reference model and the learner: even an exact refit of the reference model is 5.8–14.2 ×10⁻³ from the learner, and IGRT's error is only 0.9–3.1 ×10⁻³ above that floor (W2, Response 3/3). The magnitude curves therefore leave little room to separate, and the difference shows in the sign:

  • Across all 170 set–size combinations on the four tasks, IGRT predicts the correct sign (help or hurt) for 154 and SOURCE for 142.
  • When adding nothing is the best choice, IGRT chooses to add nothing in 9 of 10 cases, and SOURCE in 6 of 10.

References

(Engstrom et al., 2025) Optimizing ML Training with Metagradient Descent. arXiv:2503.13751.

(Ilyas & Engstrom, 2025) MAGIC: Near-Optimal Data Attribution for Deep Learning. arXiv:2504.16430.

(Killamsetty et al., 2021a) GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning. AAAI 2021.

(Killamsetty et al., 2021b) GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training. ICML 2021.

(Park et al., 2023) TRAK: Attributing Model Behavior at Scale. ICML 2023.

(Paul et al., 2021) Deep Learning on a Data Diet: Finding Important Examples Early in Training. NeurIPS 2021.

(Thrush et al., 2026) Synthetic Data for any Differentiable Target. arXiv:2604.08423.

(Wang et al., 2025) Data Shapley in One Training Run. ICLR 2025.

Reviewer_o1XqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes Interventional Godambe Risk Tomography (IGRT), a method for predicting whether adding a labeled synthetic set SS to a real current set DD reduces target risk, without retraining for each candidate. The authors model augmentation as a mixture Pα=(1−α)PD+αPSP_\alpha=(1-\alpha)P_D+\alpha P_S and derive a local expansion of expected target risk that separates a systematic-displacement term from a sampling-variance term. IGRT estimates both terms from gradients, Hessians, and score covariances at a single fixed-feature reference fit (a head on frozen features) on held-out labeled target data, and uses the score to select candidate sets and addition counts. The main conceptual contribution is treating synthetic-data utility as a current-data-dependent change in target risk rather than as distributional similarity. Experiments on CIFAR-10, ImageNet-100, ADE20K, and VOC compare predicted and measured utility; prediction error is lower than for SOURCE and the estimator ablations on ImageNet-100 and ADE20K (Table 2), while selection gains are clearest in candidate-set selection on CIFAR-10 and otherwise small relative to seed-to-seed variation.

Critical strengths

S1: The paper formalizes synthetic-data utility as a current-data-dependent change in target risk. It shows that the same synthetic set can help or hurt depending on how well the current data already cover the relevant directions (Proposition 1, Figure 1), with the benefit diminishing as coverage increases (Figure 4b), and that no score based only on image marginals can determine utility rankings for every task (Proposition 2). Consistent with this, distribution-based scores (FID, CMMD, FD-DINOv2, PRDC Recall) rank candidate sets poorly in the appendix (mean Spearman correlation at most 0.38, Table 7). This challenges the common practice of selecting synthetic data by fidelity or distributional similarity, and the theoretical argument holds independently of the estimator’s empirical performance.

S2: The paper introduces an interpretable, label-sensitive utility score. The learning-response expansion (Eq. 5) separates correction of existing target errors from a curvature penalty on overly large changes in target predictions, and the score depends on the candidate’s mean score bSb_S (Eqs. 11 and 14), so it reflects image–label consistency as well as appearance. With images held fixed, predicted and measured utility both decrease with the label-error rate and become negative at the highest rates (Figure 4a). This makes predicted utility explainable and allows it to detect harmful candidates whose images are unchanged, which image-only similarity scores cannot do. It therefore supports the method’s value as a diagnostic, even where selection gains over baselines are modest.

Critical weaknesses

W1: The selection results do not establish an advantage over simpler alternatives. In pool selection (Table 1), IGRT improves over random selection by 0.4–2.8 points, but it exceeds the strongest baseline (BAIT, text matching, or first-order influence, depending on the dataset) by only 0.04–0.35 points across the four datasets, whereas its own standard deviations over five training seeds are 2.1–2.7. These standard deviations reflect only downstream training variability, with splits, generated pools, selected subsets, and scoring randomness fixed (Appendix B.4.1), and they do not by themselves determine whether the margins are meaningful. Table 1 reports no paired uncertainty estimates or statistical comparisons. In candidate-set selection (setting A, marked † in Table 1), IGRT is best on CIFAR-10 and ties other methods on ImageNet-100, ADE20K, and VOC. On ADE20K its setting-A selection does not improve on current-only training (36.23 vs. 36.38 mIoU), which is in tension with the statement in Sec. 5.1 that its selected data improve on current-only training. In the appendix ranking analysis, SOURCE attains a higher mean Spearman correlation than IGRT on ADE20K (0.59 vs. 0.57) and VOC (0.72 vs. 0.69; Table 7), although these differences are small relative to the reported SDs, whereas IGRT is ahead on CIFAR-10 and ImageNet-100. Furthermore, the additional-real control, which adds 1000 real images, outperforms every synthetic selection on all four datasets in mean performance, while the reported allocation uses 2000 held-out labeled real images for reference fitting and scoring (TfitT_{\mathrm{fit}} and TscoreT_{\mathrm{score}}, Table 3; these splits are available to all methods, Appendix B.4.4), twice the size of the control and, on CIFAR-10, twice the size of DD itself. The paper describes the control as a reference comparison rather than an upper bound (Appendix B.4.3), but in these experiments the 1000 added real labels give higher mean performance than IGRT-guided augmentation, which also uses 2000 labeled images for scoring (although the gaps, 0.4–2.9 points, are of the same order as IGRT’s own standard deviations). Since selection is a central practical claim, without a reliable margin over simple baselines or a favorable trade-off against the labeled data it consumes, the empirical case for IGRT as a selection method is not well established (see Q1).

W2: The agreement between predicted and measured utility is established most tightly for the reference model class, and the reference-to-downstream gap is not analyzed. The theory concerns unpenalized ERM for a fixed-feature reference model, whereas utility is defined and measured for the downstream learner (Eq. 1): ResNet-18, ResNet-50, SegFormer-B0, and Faster R-CNN trained under fixed update budgets (Table 4). For classification, the reference is a logistic head on frozen DINOv2 features, while the downstream learners are ResNets (no frozen backbone is described). For detection, the paper itself describes the reference as a local head-based approximation, not a simulation of full detector retraining (Appendix B.4.4). The closest agreement comes from settings in which the measured learner is the reference model class itself (Figure 1, Tables 5 and 6, the Gaussian check in Figure 2 uses known population moments and a linear model). The cross-learner evidence (Figure 3, Table 2, and, as far as the main text specifies, Figure 4) uses one downstream learner per task, and approximation error is not separated from reference-to-learner transfer error. Appendix C.2 acknowledges that changes in the backbone, batch normalization, and the training trajectory require further modeling, but the Limitations section does not discuss this mismatch. The paper’s central question is whether UA(S∣D)U_A(S\mid D) can be predicted for the learner actually used, so the practical interpretation of the score depends on a reference-to-downstream transfer assumption whose validity is not separately established across learners (see Q4, Q6).

Questions for authors

Q1: Statistical comparison of Table 1. Could the authors report seed-matched paired differences between IGRT and each baseline in Table 1, with confidence intervals or other appropriate uncertainty estimates? Since splits, generated pools, selected subsets, and selection seeds are fixed across the five training seeds (Appendix B.4.1), it would also help to report variability across selection seeds (the pool-selection search starts from a random subset; Appendix B.4.3). If feasible, variability across independently generated pools or data splits would also be informative; otherwise, this could be noted as a limitation (see W1). This would show whether the pool-selection margins of 0.04–0.35 points are distinguishable from noise.

Q2: Estimand and possible double counting of sampling variance. The paper distinguishes the candidate-specific utility UA(S∣D)U_A(S \mid D) (Eq. 1), which concerns training randomness for fixed datasets, from the redraw-averaged VαV_\alpha (Eq. 3), and states that the score is derived from the latter and evaluated against the former (Sec. 3). Which quantity is the empirical estimator U^α\hat U_\alpha (Eq. 15) intended to estimate, and what sampling process does the covariance term JP,π/nPJ_{P,\pi}/n_P (Eqs. 13–14) represent? Appendix C.3 states that this term “averages over data sampling under the specified source-count policy”, suggesting fresh draws from the source distributions. For fixed observed DD and SS, however, there is no remaining data-sampling randomness in the realized datasets. Under a fresh-draw interpretation, the plug-in displacement term may already contain the sampling variance. For the classification reference, θ^T\hat\theta_T is fitted on TfitT_{\mathrm{fit}}, which is disjoint from DD and from the generation-support set (Appendix B.4.1). Conditional on θ^T\hat\theta_T, let b^P\hat b_P denote the sample-mean score on the augmented data (written sˉ\bar s in Eq. 13), and treat APA_P and GTG_T as fixed to leading order. For fixed KK, E ⁣[b^P⊤Kb^P]=bP⊤KbP+tr⁡ ⁣(K Cov⁡(b^P)).\mathbb{E}\!\left[\hat b_P^\top K\hat b_P\right] = b_P^\top K b_P + \operatorname{tr}\!\left(K\,\operatorname{Cov}(\hat b_P)\right). With K=12AP−⊤GTAP−1K=\tfrac12 A_P^{-\top}G_TA_P^{-1}, the quantity b^P⊤Kb^P\hat b_P^\top K\hat b_P is the quadratic displacement term of Eq. 14, since δ^P=−AP−1b^P\hat\delta_P=-A_P^{-1}\hat b_P (with b^Ptr\hat b_P^{\mathrm{tr}} in place of b^P\hat b_P when a ridge is used; the deterministic shift leaves the covariance unchanged, Appendix A.1). With Cov⁡(b^P)=JP,π/nP\operatorname{Cov}(\hat b_P)=J_{P,\pi}/n_P (Eq. 13), the second term has exactly the form of the covariance term in Eq. 14. Could the authors clarify why these contributions do not overlap, or specify the estimand and conditioning assumptions under which both terms are required? If they overlap, the score could count sampling variance twice and distort the selected addition count. Figure 2 does not test this, since it uses known population moments (Appendix B.5). A comparison of the full and displacement-only estimators against Monte Carlo estimates of an explicitly defined target quantity, using the same moment-estimation procedure as the implementation, would help resolve this (see also Q3).

Q3: Covariance ablations. Table 2 reports “IGRT, without covariance”, but the text does not define this estimator. Is it the same as “directed displacement only” in Table 6, which is also not defined? If so, removing the covariance term increases MAE on ImageNet-100 and ADE20K (Table 2) but decreases NLL MAE on the fixed-feature CIFAR-10 panel (0.00077 vs. 0.00090, Table 6). Could the authors clarify whether these ablations are identical and explain the setting-dependent effect of the covariance term?

Q4: Error decomposition. For the Figure 3 and Table 2 settings, could the authors report (i) IGRT’s predicted utility change versus the change obtained by exactly refitting the reference head on DD and on D∪SD\cup S, to quantify approximation error (as is done for the fixed-feature CIFAR-10 panel of Tables 5–6), and (ii) this exact reference-head change versus the measured downstream-learner change under the same candidate set, addition count, and evaluation protocol, to quantify reference-to-downstream transfer error (see W2)? The sign agreement defined in Appendix B.4.8, which does not appear to be reported for the main tasks, could also be reported for these settings (the abstract claims that IGRT predicts the direction of augmentation effects).

Q5: Pretraining overlap. The ImageNet-100 learner is initialized from ImageNet-1K (Table 4), while DD, TfitT_{\mathrm{fit}}, TscoreT_{\mathrm{score}}, and RaddR_{\mathrm{add}} are drawn from ImageNet-1K training images (Appendix B.4.1, Table 3), which are part of the training data of standard ImageNet-1K weights. The paper notes, for the pretrained generators, that disjoint experimental splits do not guarantee the absence of public-model pretraining exposure (Appendix B.4.2). Could the authors confirm whether these subsets were seen during pretraining of the downstream initialization, and discuss how such exposure could affect the real-only, additional-real, and augmentation results, including how it bears on treating DD as the learner’s current data?

Q6: Second downstream learner (optional). As further strengthening, it would be informative to evaluate the same candidate sets with a second downstream learner on one task, for example a ViT alongside ResNet-50 on the eight ImageNet-100 candidate sets in setting A. Reporting the rank correlation between IGRT scores and measured utility for each learner, and between the two learners’ measured utilities, would test whether IGRT’s rankings transfer across learners or depend on how closely the reference model matches the downstream learner (see W2).

10 comments
AuthorsOpenReview ↗

Summary (W1–W2, Q1–Q6): Every point addressed; score refined following your Q2

We thank you for your detailed and constructive review. We have addressed every point with new analyses, experiments, or revisions; the table maps each point to its outcome and to where it is answered. Your Q2 in particular improved the method: the refined score now predicts the direction of an addition more reliably on all four tasks.

Your pointWhat we didWhere (response; paper)
W1, Q1: margins and paired statisticsAdded paired CIs against every baseline; showed that the margins are bounded by the candidate pools, within which IGRT recovers most of the attainable gain1/3; App. B.11–B.14
Q1: selection seeds, data splits, generated poolsAdded all three; the lead holds in each1/3; App. B.12
W1: ADE20K, setting AClarified: no candidate set helps there, and IGRT correctly adds nothing1/3; App. B.13
W1: Table 7 vs. SOURCERecomputed with the score refined after your Q2: IGRT's mean Spearman is at least as high as SOURCE's on every task (ADE20K 0.65 vs. 0.59, VOC 0.75 vs. 0.72)1/3; Table 7
W1: scoring labels vs. additional real imagesAdded a fixed-label-budget comparison; the scoring labels are reused for training, and IGRT-selected data still help1/3; App. B.19
Q2: estimand and double countingEstimand stated; score refined as you derived; your Monte Carlo check confirms the refined score2/3; App. B.10
Q3: Tables 2 vs. 6Same estimator; the difference is explained by the reference-to-learner gap2/3; App. B.15
W2, Q4: reference-to-learner gapAdded the error decomposition and sign agreement on all four tasks; transfer assumption stated3/3; App. B.15
Q5: pretraining overlapConfirmed and controlled: the method ordering holds with random initialization3/3; App. B.21
Q6: second learnerAdded ViT-S; its rankings agree with ResNet-503/3; App. B.17

We are also grateful that you recognized that the paper challenges the common practice of selecting synthetic data by fidelity or distributional similarity (S1), and that the score is interpretable and label-sensitive, detecting harmful candidates whose images are unchanged (S2).

AuthorsOpenReview ↗

Response 1/3 (W1, Q1): Small margins reflect the candidate pools; IGRT recovers 72–89% of the attainable gain

On W1 and Q1, the new statistics explain the small margins: the candidate pools leave little to gain, IGRT recovers most of what they offer, and its lead holds across selection seeds, splits, and pools. All IGRT numbers use the corrected score.

W1 & Q1: Selection margins and statistics

W1: "…it exceeds the strongest baseline … by only 0.04–0.35 points across the four datasets…"

Q1: "Could the authors report seed-matched paired differences between IGRT and each baseline in Table 1 …? … variability across independently generated pools or data splits would also be informative"

Paired comparisons (Q1). Pool selection (setting B), IGRT (corrected) minus baseline, matched over the five training seeds:

TaskBaselineMean difference95% CISeeds won
CIFAR-10GLISTER (new)+0.440.04 to 0.844/5
CIFAR-10BAIT+1.170.43 to 1.915/5
ImageNet-100TRAK (new)+0.11−0.24 to 0.463/5
ImageNet-100BAIT+0.48−0.14 to 1.104/5
ADE20KText matching+0.04−1.02 to 1.103/5
VOCFirst-order IF+0.49−0.63 to 1.614/5

IGRT keeps the highest mean on all four tasks after the new baselines are added, even against the best baseline picked per task in hindsight, which is a different method each time: MAGIC on CIFAR-10 (+0.07; 95% CI −0.11 to 0.25), In-Run Data Shapley on ImageNet-100 (+0.09; −0.22 to 0.40), GLISTER on ADE20K (+0.02; −0.20 to 0.24), and TRAK on VOC (+0.11; −0.27 to 0.49). MAGIC and In-Run Data Shapley cost 4.6× and 5.1× as much as IGRT on those tasks (App. B.11). App. B.12 now reports the paired difference against every baseline in Table 1 and every added baseline.

Selection seeds (Q1). We reran pool selection with three selection seeds on CIFAR-10 and ImageNet-100, for IGRT and two strong baselines per task (GLISTER and BAIT; TRAK and BAIT). The SD across selection seeds is 0.13–0.22 points, compared with 2.25–2.44 across training seeds.

Independent splits and pools (Q1). On CIFAR-10 we redrew the data split with three seeds and, separately, regenerated the pool twice with new generation seeds. The SD is 0.34–0.46 points across splits and 0.19–0.22 across pools, and the lead holds in both: +0.47 and +0.49 over GLISTER (95% CIs over the three splits or pools 0.39 to 0.56 and 0.34 to 0.65), and +1.17 and +1.23 over BAIT (0.94 to 1.41 and 1.07 to 1.39).

What W1 reveals. Your observation that the margins over the strongest baselines are small led us to analyze why, and we find that they are set mainly by the candidate pools, not by the selector.

  1. In setting A every candidate set was trained and evaluated, so the oracle choice is known, yet even the oracle gains only 1.67, 0.84, 0.29, and 1.07 points over a random set (CIFAR-10, ImageNet-100, ADE20K, VOC).
  2. Current generated images have low training utility: even the 1,000 images IGRT selects trail 1,000 additional real images on all four tasks (60.16 vs. 61.51, 61.48 vs. 62.90, 36.58 vs. 36.98, 62.22 vs. 64.94).
  3. Within this room IGRT recovers 72–89% of the attainable gain, and when candidate utility varies more, its lead over the strongest baseline grows to +3.45 points (App. B.14).

We have made the selection claim in the abstract, the contributions, and Sec. 5 precise accordingly ("match or exceed"). Selection is one use of the score; beyond it, the signed score decides when to add nothing (9 of 10 menus; SOURCE 6, first-order influence 1).

ADE20K, setting A. The tension you noted in W1 is real, and we have corrected the sentence in Sec. 5.1. It also points to something stronger: no ADE20K candidate set improves on current-only training (best set 36.31 vs. 36.38), so the correct decision there is to add nothing. When m=0m=0 is in the menu, IGRT chooses it on all four ADE20K menus (App. B.13).

Table 7 vs. SOURCE. After the correction, IGRT's mean Spearman is at least as high as SOURCE's on all four tasks: 0.82 vs. 0.64 (CIFAR-10), 0.86 vs. 0.58 (ImageNet-100), 0.65 vs. 0.59 (ADE20K), and 0.75 vs. 0.72 (VOC). The gain is largest on ImageNet-100, where the bootstrap interval of the difference excludes zero (0.01 to 0.51).

Additional real data and labels. The comparison you suggest in W1 is the right one: under the original protocol, the labels used for scoring would be better spent on training. It led us to change the protocol: the labels are used for scoring and then merged into training, and we compare at a fixed total label budget. On CIFAR-10, IGRT-selected synthetic data add +1.04 (95% CI 0.83 to 1.25) and +0.83 (0.55 to 1.11) points over training on the same real labels alone (App. B.19; KrwS thread).

AuthorsOpenReview ↗

Response 2/3 (Q2, Q3): Double counting corrected; this also resolves Tables 2 and 6

Frankly, your Q2 helped our method a great deal: your derivation is correct, and we have removed the duplicated variance term from the score. The correction also resolves the inconsistency between Tables 2 and 6 that you raised in Q3.

Q2: Estimand and double counting

Q2: "Which quantity is the empirical estimator U^α\hat U_\alpha (Eq. 15) intended to estimate …? … If they overlap, the score could count sampling variance twice and distort the selected addition count."

The submitted score estimated BT+2VTB_T+2V_T instead of BT+VTB_T+V_T; removing the trace term corrects it, and the Monte Carlo check you proposed confirms this.

Why. The score targets the realized reference utility of the observed sets, RT(θ^D)−RT(θ^D∪S)R_T(\hat\theta_D)-R_T(\hat\theta_{D\cup S}) (Eq. 1 with the reference ERM as the training rule; its average over redraws of DD and SS, Eq. 3, is the quantity in Theorem 1). Conditional on the reference fit θ^T\hat\theta_T, the observed mean score is b^P=bP+ξP\hat b_P=b_P+\xi_P with Cov(ξP)=JP,π/nP\mathrm{Cov}(\xi_P)=\mathcal J_{P,\pi}/n_P, so to first order

E[gT⊤δ^P+12δ^P⊤GTδ^P]=BT+VT.\mathbb E[g_T^\top\hat\delta_P+\tfrac12\hat\delta_P^\top G_T\hat\delta_P]=B_T+V_T.

The plug-in displacement thus already contains VTV_T, and adding the trace term counts it twice. For the observed sets, the refit equals δ^\hat\delta to first order, so the realized utility also calls for the displacement term alone. The trace term is correct only when bPb_P is a population moment, as in Fig. 2.

Monte Carlo check (new App. B.10, Table 8). We ran the comparison you proposed in Q2 on the known softmax population of the fixed-feature CIFAR-10 check (P=45P=45), following the deployed pipeline: the reference is fitted on 1,000 fresh target labels, gTg_T and GTG_T use a disjoint 1,000, the moments use the observed DD and SS, and the ground truth is the exact population excess NLL after refitting.

The 10 cells cross matched / label shift (s=0.15,0.35,0.6s=0.15, 0.35, 0.6) / coverage gap with nD=512,2048n_D=512, 2048 and m=0,128,512,2048m=0, 128, 512, 2048.

Old (submitted Eq. 15)Corrected (displacement only)
Mean estimate / true utility (1 is exact; 9 cells)1.78–2.160.91–1.02
MAE vs. realized utility (×10⁻³)4.6–29.21.6–10.6

Effect on the addition count. As you anticipated in Q2, the duplicated term acts as a spurious sample-size benefit and pushes toward adding data: when adding nothing is best, the old score chooses m=0m=0 in only 4 of 10 menus, against 9 of 10 for the corrected score.

Changes.

  1. The IGRT score is now the displacement-only estimator. The trace term is kept only for the population-moment analysis and as a diagnostic, B^T=E^disp−V^T\hat B_T=\hat{\mathcal E}^{\rm disp}-\hat V_T.
  2. The text now states the estimand explicitly.
  3. Our implementation, including the setting-B search, used the old score, so all IGRT entries have been recomputed and all score-dependent selections redone; three of the eight selections in Table 1 changed and were retrained with the same seeds.

Q3: Covariance ablations

Q3: "Could the authors clarify whether these ablations are identical and explain the setting-dependent effect of the covariance term?"

Yes, the two ablations are identical, and the setting-dependent effect comes from the gap between the reference model and the learner, not from the covariance term.

Same estimator. "IGRT, without covariance" (Table 2) and "directed displacement only" (Table 6) are both the submitted Eq. 15 without the trace term, which is now the main score. We have unified the names and defined the estimator in Sec. 4. "IGRT, J=HJ=H" replaces J\mathcal J by the regularized Hessian in the old score and is now an ablation of the old score only.

Fixed-feature panel (Table 6). The learner there belongs to the reference class, so the score without the trace term is more accurate (MAE 0.00077 vs. 0.00090), as the correction in Q2 predicts.

ImageNet-100 and ADE20K (Table 2). The learner gains more than the reference model, so the old score's upward bias happens to point toward it (MAE, old vs. corrected, ×10⁻³: 8.9 vs. 17.3 and 3.8 vs. 6.7). Against the exact reference refit, the old score is far less accurate (13.4 vs. 4.6 and 4.7 vs. 1.4).

CIFAR-10 and VOC. The same bias increases the error against the learner (16.1 vs. 13.4 and 11.6 vs. 9.7), and the old score has lower sign accuracy on all four tasks (140 vs. 154 of 170).

Table 2 now reports both scores with sign accuracy and the error decomposition (App. B.15).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Corrected Score, New Baselines, Transfer Analysis

We thank the reviewers for their constructive feedback.

With the corrected score, the paper makes three claims:

  1. The utility of a synthetic set depends on the current data and the target task, so no image-only score can rank candidates for every task (Props. 1–2).
  2. From one reference fit, without training on any candidate, IGRT predicts whether adding a set helps or hurts. The sign is correct for 154 of 170 set–size pairs (SOURCE 142, first-order influence 126), and IGRT adds nothing in 9 of the 10 menus where that is best (SOURCE 6, first-order influence 1).
  3. In selection, IGRT matches or exceeds the strongest attribution and subset-selection baselines at lower cost, recovering 72–89% of the attainable gain in candidate-set selection; its lead grows to +3.45 points when label errors affect half of the pool.

Changes in the revised PDF are highlighted in blue

We list them here and respond to each reviewer below:

  • As pointed out by Reviewer o1Xq (Q2, Q3), we have removed a duplicated variance term from the score, which also resolves the disagreement between Tables 2 and 6. We have stated the estimand, unified the ablation names, and recomputed all IGRT results; three of the eight selections in Table 1 changed and were retrained. The Monte Carlo check proposed by the reviewer confirms the corrected score (App. B.10).
  • As asked by Reviewers o1Xq (W2, Q4), WzRi (W2), and KrwS (W1), we have separated the local approximation error from the transfer error between the reference model and the learner, and stated the transfer assumption in Sec. 4 and the Limitations. Signs and rankings still carry over to the learner (App. B.15, B.17).
  • As asked by Reviewers KrwS (W1, Q1), WzRi (W2, Q2), and o1Xq (Q6), we have added an ablation over six reference models and a second downstream learner. References aligned with the learner work best, and ViT-S gives similar rankings on eight candidate sets (App. B.16, B.17).
  • As asked by Reviewer KrwS (Q1), we have added a diagnosis of when transfer breaks down, covering feature drift and additions up to twice the current set (App. B.22).
  • As asked by Reviewer o1Xq (Q5), we have clarified the pretraining overlap on ImageNet-100 and added a control trained from random initialization, in which the ordering of the methods is unchanged (App. B.4, B.21).
  • As asked by Reviewer o1Xq (W1, Q1), we have added paired comparisons, selection-seed, split, and pool variability, and the room for improvement in each candidate menu (App. B.12, B.13). The margins are limited by the low training utility of current synthetic images, within which IGRT recovers 72–89% of the attainable gain; we now write that IGRT's selections "match or exceed" the baselines.
  • As asked by Reviewer WzRi (W1), we have added a paragraph on the relation to data attribution (Introduction), covering In-Run Data Shapley (Wang et al., 2025), MAGIC (Ilyas & Engstrom, 2025), metagradient descent (Engstrom et al., 2025), and DPG (Thrush et al., 2026).
  • As asked by Reviewer WzRi (W1), we have added seven baselines with a cost comparison: TRAK (Park et al., 2023), MAGIC, In-Run Data Shapley, GLISTER (Killamsetty et al., 2021a), GRAD-MATCH (Killamsetty et al., 2021b), EL2N, and GraNd (Paul et al., 2021). IGRT has the highest mean at lower cost than all seven, with small margins (App. B.11).
  • To test this positioning on synthetic data (Reviewer WzRi, W1), we have added pools with mislabeled candidates and with near-duplicates. In both, scoring whole sets outperforms ranking images one by one (App. B.14, B.20).
  • As asked by Reviewer WzRi (Q1), we have added a linear datamodeling score adapted to additions, comparing IGRT with TRAK, SOURCE, LWP, and first-order influence (App. B.18).
  • As asked by Reviewers KrwS (W2, Q2) and o1Xq (W1), we have added a comparison at a fixed real-label budget and a study of the number of scoring labels. The scoring labels can be merged into training after selection, and IGRT-selected data then add about one point on CIFAR-10 (App. B.19).
  • As asked by Reviewer WzRi (Presentation), we have added an overview of the method (Sec. 4), a setup table (Sec. 5), a notation table (App. B.1), and a one-sentence statement after each theorem and proposition.
  • We also report the score with the ridge gradient specified in App. A; it reduces the approximation error further, leaving almost only transfer error (App. B.15).

References

Engstrom et al., 2025. Optimizing ML Training with Metagradient Descent. arXiv:2503.13751.

Ilyas & Engstrom, 2025. MAGIC: Near-Optimal Data Attribution for Deep Learning. arXiv:2504.16430.

Killamsetty et al., 2021a. GLISTER. AAAI.

Killamsetty et al., 2021b. GRAD-MATCH. ICML.

Park et al., 2023. TRAK: Attributing Model Behavior at Scale. ICML.

Paul et al., 2021. Deep Learning on a Data Diet. NeurIPS.

Thrush et al., 2026. Synthetic Data for any Differentiable Target. arXiv:2604.08423.

Wang et al., 2025. Data Shapley in One Training Run. ICLR.

AuthorsOpenReview ↗

Summary of Core Contributions and Rebuttal Revisions

Dear AC and PC,

We addressed every reviewer point-by-point and summarize our contribution and revisions:

1. Core Contribution All reviewers agreed realism/density does not determine training utility (Props. 1–2; distribution metrics ≤0.38 vs. IGRT 0.65–0.86 Spearman). Unlike post-hoc attribution, IGRT predicts signed risk changes of unseen synthetic sets before training: rejecting harmful sets (9/10 menus; 154/170 signs, 4 tasks), resisting 50% mislabels (+3.45 lead) and duplicates (0.61 vs. TRAK 2.20 drop), and matching/exceeding baselines at 4.6–9× lower cost than TRAK, MAGIC, and In-Run Data Shapley.

2. Rebuttal Revisions

  • Estimator fix (o1Xq): Removed duplicated variance term; recomputed/retrained all results (MC ratio 0.91–1.02; refit MAE cut 2.5–3.4×; m=0 rejection 4/10→9/10).
  • Baselines & LDS (WzRi): Added 7 baselines (best on 3/4 tasks; within 0.04 on ADE20K) and addition-LDS (0.76/0.87 vs. TRAK 0.69/0.77).
  • Transfer & budgets (All): Tested 6 references (0.79–0.87), ViT-S (0.88), random-init ImageNet-100 (+0.56 vs. TRAK), reused scoring labels (+0.83–1.04), paired CIs, and clarity tables.

3. Reviewer Resolution

  • o1Xq praised our interpretable, label-sensitive diagnostic, had no further questions, and updated their assessment positively.
  • WzRi valued extending influence functions to synthetic utility; all requests completed (no reply).
  • KrwS confirmed most concerns resolved. For the shared transfer-gap concern, synthetic domain shift and label noise hurt learners similarly, so a real-data reference captures shared effects (Selection via Proxy); error decomposition shows this affects magnitudes, not decisions (24/28 ImageNet-100 pairs; 11/11 largest-gap zero loss).

Thank you for your time and consideration.

Sincerely,

Submission 2226 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.