acceptodds
Withdrawn from ICLR 2027

Training-Aware Target Coverage for Synthetic Data Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Synthetic data are increasingly used to scale LLM training, yet more synthetic data do not necessarily produce better models. Useful synthetic data must add information relevant to the target task without introducing errors that offset their benefit, and the value of an example can change as the training set grows. We develop a linear theory that characterizes this tradeoff and determines where synthetic data are useful, how much should be added, and the marginal value of adding one example to an existing set. The analysis shows the conditions when input coverage alone is sufficient and when synthetic errors must also be considered. Guided by these results, we introduce Training-Aware Target Coverage (TATC), a synthetic data selection method for LLM fine-tuning. TATC identifies candidates whose training effects are beneficial to the target task and selects among them to expand coverage of target-relevant directions not already represented by the available data. Experiments on text and image data verify the linear theory. With mathematical reasoning tasks, TATC selects synthetic solutions for fine-tuning Qwen2.5-Math-1.5B-Instruct and outperforms alternative synthetic-data selection methods on GSM8K across selection budgets. In summary, we provide a principled approach to synthetic data selection by quantifying and maximizing its value to the target task.

Primary area:
learning theory
Keywords:
synthetic data, data selection, supervised fine-tuning, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1u6BOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which and how many synthetic examples are worth adding to the real training set to further improve models' performance. In a linear theory, it splits the value of a synthetic set into three parts: an information gain from new input directions, a cost from random label noise, and a cost from systematic label error. Based on this, the authors propose TATC for LLM finetuning, which first ranks candidates by a predicted training effect. After that, it picks the candidate that adds the most coverage of probe-relevant directions not yet covered. Experimental results demonstrate its effectiveness in further improving the model's performance when finetuning it to a target task.

Critical strengths
  • The paper solves a practical question: which synthetic examples help, and how many should be used? It argues well that neither quality nor diversity alone answers such a question perfectly. An example can add new information but carry a wrong label, or have a correct label but add nothing new.

  • The linear analysis is simple and easy to follow. It shows that the value of synthetic data depends on direction, on amount, and on what is already in the training set.

  • The proofs are complete, and the text experiment shows that coverage alone cannot separate helpful from harmful examples

Critical weaknesses
  • Weak connection between the theory and the method, and limited novelty. The theory's main insights are the optimal amount of synthetic data and the interaction between a candidate's error and the existing bias. TATC uses neither: the budget is fixed by hand, and Table 4 admits the bias interaction is omitted. The training-effect score is a local influence estimate, close to the score used by OptimSyn, and is not derived from the theory. The coverage term is the A-optimal design criterion used by BAIT, which the paper acknowledges, and the bias–variance analysis of synthetic data is close to Rezaei et al. (2026). The conclusion's claim of a "novel perspective" (that an example's value depends on the existing data) also overstates the contribution, since Data Shapley and active learning rest on the same idea.

  • Limited experiments. The LLM evaluation uses only one small model on one benchmark (GSM8K), and the text and image experiments test only the linear theory, so it is unclear whether TATC works on other tasks and models. The abstract should make this clear, and evaluating TATC on at least one non-math text or image task would better support its effectiveness. The only ablation removes the training-effect score; there is no variant using the training-effect score alone, so the value of the coverage step is unclear, and there is no sensitivity analysis for κ, the PCA dimension, or ρ. The paper also does not report the compute or time cost of computing per-example gradients and AdamW updates for all 4,096 candidates. Finally, Figure 3(b) shows only five of the ten methods; all baselines should be included across budgets.

Questions for authors
  • Does TATC improve results on other tasks (e.g., MATH, code, or instruction following) or with larger models? Can it be applied to an image task?

  • How could TATC use the theory's amount-selection result to choose the budget, instead of fixing it by hand? How much performance is lost by omitting the interaction between a candidate's error and the existing bias?

  • How does the training-effect score differ from OptimSyn's influence score, beyond the curvature term? Can it be derived from the marginal value in Theorem 7?

  • How does TATC perform with the training-effect score alone? How sensitive is it to k, the PCA dimension, and p?

  • What is the compute and time cost of TATC (per-example gradients and AdamW updates for all 4,096 candidates), compared with the baselines?

  • Beyond BAIT's A-optimal criterion and the analysis in Rezaei et al. (2026), what are the new theoretical results in this paper?

Reviewer_i9mKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors develop a theoretical framework for characterizing the value of synthetic data, showing how downstream benefit depends jointly on the information added by synthetic inputs and the errors they introduce. They introduce Training-Aware Target Coverage (TATC), a synthetic data selection method in LLM fine-tuning. TATC outperforms existing synthetic data selection methods on GSM8K mathematical reasoning.

Critical strengths

The paper is clearly written and easy to follow. The progression from the risk decomposition to the selection method is well organized.

The method is well motivated by a unified decomposition of information gain, random noise, and systematic bias. This explains why broader input coverage does not necessarily produce more useful training data.

The controlled text and image experiments help illustrate the theory. In particular, varying response errors while keeping inputs fixed demonstrates why input coverage alone is insufficient.

Critical weaknesses

The novelty of the “Predicted local training effect” score relative to GREATS is insufficiently established, since both use Taylor approximations to estimate reductions in validation loss. The paper should distinguish its fixed-checkpoint, optimizer-aware score and separate coverage criterion from GREATS’s online scoring and interaction correction, with a direct empirical comparison.

The LLM evidence is substantially insufficient: it covers only one 1.5B model, 4,096 candidates, and a GSM8K holdout, while the two panels of Figure 3 use different generation limits. Experiments across multiple models, datasets, and larger candidate pools are needed to establish generality, using consistent evaluation settings.

The ablation includes coverage-only selection but omits training-effect-only selection. It therefore does not establish how much dynamic coverage adds beyond simply ranking candidates by predicted training benefit.

Questions for authors

Please add more experiments and clarify the novelty mentioned in Critical Weaknesses.

Reviewer_rwjmOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to select synthetic training data that improve performance on a target task. Under a linear regression model, the authors define synthetic-data value as the expected reduction in downstream risk relative to training on real data alone. They decompose this value into target-relevant information gain, excess label-noise cost, and systematic label-error cost. The analysis further characterizes beneficial feature directions, optimal amounts of synthetic information, and the conditional marginal value of adding an example to an existing dataset.

Critical strengths

Relevant problem and a useful set-dependent perspective.

The paper addresses an important limitation of synthetic-data scaling: additional examples can be redundant or harmful. Its formulation explicitly considers what a candidate contributes beyond the real data and previously selected examples, providing a coherent perspective on quality, coverage, and diminishing returns.

Interpretable theoretical analysis.

The risk decomposition clearly distinguishes input information from random and systematic label errors. Theorem 7 is particularly informative because it exposes the interaction between a candidate’s error and the bias accumulated in the selected set. This explains why an imperfect example can either mitigate or reinforce existing bias, depending on the selection context.

Well-motivated controlled experiments.

The text and image experiments hold candidate inputs fixed while varying their targets or pseudo-label reliability. This design isolates label-error effects from changes in input coverage and makes the theoretical mechanisms easier to interpret.

Promising downstream results.

At a selection budget of 128, TATC achieves 87.34% exact match, compared with 86.07% for OptimSyn and 85.19% for the coverage-only ablation. The consistent improvement over coverage-only selection across the evaluated budgets supports incorporating response-dependent training information.

Critical weaknesses

The practical algorithm does not capture the central error interaction identified by the theory.

Theorem 7 emphasizes that candidate value depends on interactions between candidate error and the current set’s accumulated bias. However, TATC computes its response-dependent score only once at the initial checkpoint; only input coverage changes during selection. Consequently, the algorithm does not explicitly track whether a response cancels or reinforces the evolving bias. Although this limitation is acknowledged, the paper provides limited evidence that the fixed score preserves the conditional marginal-value behavior motivating the method.

The ablations do not isolate the contributions of the main design choices.

The coverage-only ablation establishes the value of adding a training-effect signal, but there is no direct comparison against selecting the top-BB examples using hjh_j alone. It therefore remains unclear whether dynamic coverage contributes beyond the same training score. OptimSyn does not resolve this question because its implementation also changes normalization and omits the quadratic term. Additional ablations are needed for the quadratic penalty, shortlist expansion, and expansion factor κ\kappa.

The LLM evaluation is too narrow to establish broad effectiveness.

Experiments use one 1.5B model, one target task, and one pool of 4,096 synthetic trajectories, with selection budgets up to 512. Generalization across model scales, tasks, generators, and error rates is untested. Moreover, the improvement over the strongest baseline is modest at B=128B=128, and the methods are effectively tied at B=512B=512. Three training repetitions provide useful evidence, but do not characterize variability across data partitions or candidate pools.

The local training-effect surrogate lacks direct empirical validation.

The curvature matrix is constructed from squared batch gradients using a diagonal approximation, but its ability to predict actual probe-loss changes is not measured. Furthermore, the score evaluates the update difference Δθj=Uj−U∅\Delta\theta_j=U_j-U_{\varnothing}, whereas training applies complete AdamW updates over multiple steps. Proposition 8 bounds local loss changes along Δθj\Delta\theta_j; it does not directly establish the utility of the eventual training procedure. Correlation and sign-agreement measurements against actual loss reductions would strengthen this component.

Baseline coverage and implementation choices leave alternative explanations unresolved.

Although verifier outcomes are available for the candidate pool, the experiments omit straightforward baselines such as random selection among verifier-approved examples. These would help establish the benefit beyond readily available correctness filtering. In addition, LESS is implemented using fixed-checkpoint sketched-gradient cosine similarity, while BAIT uses the candidate distribution as its target rather than the real-probe distribution used by TATC. These choices complicate attribution of the gains to the proposed selection mechanism. A direct comparison with SPICE, discussed as closely related work, is also missing.

Efficiency and evaluation reporting are incomplete.

The method requires per-example gradient and optimizer-update computations, but selection time, memory usage, and overhead relative to fine-tuning are not reported. Its practicality for substantially larger pools is therefore unclear. Additionally, the two panels of Figure 3 use different generation limits, preventing direct comparison of absolute accuracy at the same budget. Appendix D.4 states that nontermination rates are reported, but corresponding results are not presented in the manuscript.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.