- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the utility of synthetic training data through its effect on predictions on real target data. It develops a local risk expansion that separates systematic parameter displacement from data-sampling variability and proposes Interventional Godambe Risk Tomography (IGRT) to estimate augmentation benefits without retraining for every candidate set. IGRT combines gradient statistics, candidate-mixture curvature, target-loss curvature, and a sampling-covariance correction. Experiments evaluate utility prediction and data selection across classification, segmentation, and detection, alongside controlled studies of label errors, current-data coverage, and augmentation size.
Critical strengths
-
The paper provides a useful account of task-dependent synthetic-data utility. The analysis connects augmentation benefits to corrections of target errors and explains why the same candidate data can help or hurt depending on current coverage. The controlled label and coverage experiments make these dependencies concrete. This contributes useful understanding beyond evaluating synthetic images through realism or distributional similarity
-
The estimator has a principled local foundation and meaningful empirical support. The risk expansion distinguishes systematic displacement from sampling variability and explicitly states its regularity conditions. The experiments examine prediction magnitudes, augmentation-size selection, and downstream selection rather than only reporting correlations. In particular, the main ablations in Table 2 provide evidence that the full estimator can improve prediction error and selection regret.
Critical weaknesses
-
IGRT estimates utility through a local reference model, rather than modeling the downstream training process itself. The downstream experiments provide evidence that this approximation can be useful, but it remains unclear how sensitive the utility rankings are to the choice of reference representation. Synthetic data may change the features learned during downstream training in ways that the fixed-feature approximation cannot capture. This makes it difficult to know when the reference model will give reliable selection decisions for a different learner.
-
IGRT requires labeled target data both to fit the reference model and to estimate target-risk derivatives. Table 3 allocates 1,000 images to each of these subsets, which is substantial relative to the current training sets in some experiments. Table 1 includes an additional-real baseline, but this does not fully answer whether the labels used for reference fitting and risk scoring would be better spent directly on downstream training. The practical benefit under a fixed total labeling budget therefore remains unclear, especially when real labels are scarce.
Questions for authors
-
When the reference model and downstream learner use different representations, what makes their candidate-utility rankings agree? Can the authors clarify where they expect this approximation to break down?
-
In a setting with limited real labels, how should practitioners decide between using those labels for IGRT and adding them directly to the downstream training set?
7 comments
Response 1/2 (W1, Q1): A learner-aligned reference works best; transfer holds up to twice the current set
You are right that the choice of reference matters (W1); testing six references showed that one built from the real-only learner works best and that every reference predicts most signs correctly.
W1 & Q1: Sensitivity to the reference representation, and when transfer breaks down
W1: "…it remains unclear how sensitive the utility rankings are to the choice of reference representation."
Q1: "When the reference model and downstream learner use different representations, what makes their candidate-utility rankings agree? Can the authors clarify where they expect this approximation to break down?"
What makes rankings agree (Q1). Utility enters through the target-logit response. Rankings and signs carry over when the learner's response to candidates is approximately a positive rescaling of the reference response along the candidate–target gradient directions, i.e., when the reference feature kernel and the learner's empirical tangent kernel agree on those directions (new paragraph in Sec. 4). A positive rescaling leaves both the sign and the order unchanged.
How sensitive are rankings to the reference (W1)? On CIFAR-10, we scored the same 50 set–size pairs with six references fixed in advance. We then compared each with the utility measured on ResNet-18 trained from scratch (full network), so the measurement includes exactly the feature changes you raise in W1.
| Reference features / head fitted on | Spearman | Signs correct | Harmful detected | Beneficial rejected | Set / size regret |
|---|---|---|---|---|---|
| DINOv2 / (paper) | 0.82 | 47/50 | 14/15 | 2/35 | 2.8 / 3.6 |
| DINOv2 / current data | 0.80 | 45/50 | 13/15 | 3/35 | 3.1 / 4.0 |
| Real-only ResNet-18 / | 0.87 | 48/50 | 14/15 | 1/35 | 2.0 / 2.7 |
| Real-only ResNet-18 / | 0.86 | 47/50 | 14/15 | 2/35 | 2.2 / 2.9 |
| Real-only ResNet-18, second seed / | 0.85 | 47/50 | 13/15 | 1/35 | 2.3 / 3.0 |
| CLIP ViT-B/16 / | 0.79 | 44/50 | 12/15 | 3/35 | 3.4 / 4.2 |
All six references rank well (Spearman 0.79–0.87). A reference aligned with the learner is best, a generic one (CLIP) is worst, and the reference seed matters little. The sensitivity does not reach the decision that matters most: even CLIP predicts the sign for 44 of 50 pairs and detects 12 of 15 harmful ones.
The error decomposition shows where the remaining error lies. Against exact reference refits, IGRT's MAE is 1.4–4.6 ×10⁻³ across the four tasks; between the reference refit and the downstream learner, it is 5.8–14.2. Transfer, not the local approximation, is the main source of error.
On ImageNet-100, a second learner (ViT-S alongside ResNet-50) gives Spearman 0.88 and 0.86, with 8/8 signs for both (App. B.17).
Where transfer breaks down (Q1). The predicted sign stays correct for additions up to twice the current set; transfer weakens mainly when augmentation changes the learner's features or the addition grows beyond that.
- A reference not aligned with the learner. CLIP is the weakest reference above. Check: compare two references; disagreement in sign flags an unreliable decision.
- Representation change. Drift, between the penultimate features of the current-only and augmented learners, correlates with the absolute transfer error (Spearman 0.46 on CIFAR-10, 50 pairs; 0.39 on ImageNet-100, 40 pairs; new App. B.22).
- Large additions. CIFAR-10, , 3 seeds, ×10⁻³ NLL, predicted vs. measured utility:
| Source | |||
|---|---|---|---|
| High | 215 / 224 ± 51 | 337 / 354 ± 62 | 468 / 421 ± 76 |
| Medium | 88 / 92 ± 37 | 121 / 116 ± 42 | 154 / 103 ± 56 |
| Negative | −28 / −31 ± 24 | −67 / −74 ± 29 | −119 / −143 ± 41 |
All signs are correct and all predictions lie within one SD; a visible bias appears only at twice , where IGRT overestimates the medium source's gain and underestimates harm.
- Training effects outside the loss (augmentation, batch normalization, fixed update budgets) and 5. near-ties are being tested; we will post the results within the next few days.
Practical guidance. Use a reference built from the real-only learner trained on . This model is already trained as the baseline, so it costs nothing extra. Before committing to a large pool, check signs on two or three candidate sets.
Response 2/2 (W2, Q2): Scoring labels can be reused for training; a decision rule for practitioners
W2 & Q2: The target-label budget
W2: "…whether the labels used for reference fitting and risk scoring would be better spent directly on downstream training."
Q2: "In a setting with limited real labels, how should practitioners decide between using those labels for IGRT and adding them directly to the downstream training set?"
You are right that, used only for scoring, these labels would be better spent on training (W2); this led us to a better protocol. IGRT used 2,000 labeled target images ( and ), twice on CIFAR-10.
The labels need not be consumed. Scoring needs held-out labels only while selecting; afterwards they can be added to training. On CIFAR-10, with a fixed real-label budget (five seeds):
| Real only () | + random | + GLISTER | + IGRT | |
|---|---|---|---|---|
| 61.51 | 61.80 | 62.29 | 62.55 | |
| 65.20 | 65.35 | 65.80 | 66.03 |
IGRT minus real only: +1.04 (95% CI 0.83 to 1.25) and +0.83 (0.55 to 1.11); IGRT minus GLISTER: +0.26 (0.07 to 0.45) and +0.23 (0.06 to 0.40); 5/5 seeds in each. With , the reference head is fitted on and all 1,000 labels are used for scoring.
Fewer labels. Fitting the reference head on removes the 1,000 fitting labels at little cost. Scoring labels can also be reduced:
| (head fitted on / on ) | 100 | 250 | 500 | 1000 |
|---|---|---|---|---|
| Spearman | 0.64 / 0.61 | 0.75 / 0.72 | 0.80 / 0.79 | 0.82 / 0.80 |
| Sign accuracy | 80% / 78% | 88% / 86% | 92% / 88% | 94% / 90% |
| Mean SE of predicted utility (×10⁻³) | 9.6 / 10.2 | 6.0 / 6.4 | 4.2 / 4.5 | 3.0 / 3.2 |
How should practitioners decide (Q2)?
- If a labeled validation set is needed anyway, use it first to score the candidates, then merge it into training. Fitting the reference on means no labels beyond this set are needed.
- Add synthetic data only when its predicted utility exceeds about two standard errors, which the scoring set itself provides (e.g., about with 500 scoring labels). Otherwise choose . Even without this margin, when is simply included in the menu, IGRT declined to add in 9 of 10 menus where adding nothing was best, and in only 2 of 26 where adding helped.
- If fewer than about 250 labels are available, ranking quality drops (Spearman 0.61–0.64 at 100). If no separate validation set is needed, add the labels directly to training.
The other target-aware selectors face the same cost. First-order influence, SOURCE, LWP, GLISTER, and GRAD-MATCH all use the same labeled target splits. Image-only criteria such as FID avoid it, but perform worse in Table 1.
Revisions. A label-budget appendix (App. B.19) with the two tables above, a Limitations paragraph on label cost, and a revised abstract sentence on practical selection (General Response).
A note on Strength 2
Following Reviewer o1Xq (Q2), we removed a duplicated variance term from the score and recomputed all IGRT results (General Response). In Table 2, the old score's lower error on ImageNet-100 and ADE20K reflected the reference-to-learner gap you raise in W1, not the extra term (App. B.15); Table 2 now reports both scores. The local foundation you credit is unchanged, and the refined score is 2.5–3.4 times more accurate against exact refits of the reference model.
We hope these responses resolve your concerns. If so, we would be grateful if you would consider updating your score, and we are glad to discuss any remaining questions.
Summary (W1–W2, Q1–Q2): Every point addressed; reference sensitivity and label budget tested
We thank you for your detailed and constructive review. We have addressed every point with new analyses, experiments, or revisions; the table maps each point to its outcome and to where it is answered. Your W1 and W2 in particular became practical guidance in the paper: build the reference from the real-only learner, and reuse the scoring labels for training after selection.
We are also grateful that you found the paper a useful account of task-dependent synthetic-data utility that goes beyond realism or distributional similarity (Strength 1), and that the estimator has a principled local foundation, examined on prediction magnitudes, augmentation-size selection, and downstream selection (Strength 2).