- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies which and how many synthetic examples are worth adding to the real training set to further improve models' performance. In a linear theory, it splits the value of a synthetic set into three parts: an information gain from new input directions, a cost from random label noise, and a cost from systematic label error. Based on this, the authors propose TATC for LLM finetuning, which first ranks candidates by a predicted training effect. After that, it picks the candidate that adds the most coverage of probe-relevant directions not yet covered. Experimental results demonstrate its effectiveness in further improving the model's performance when finetuning it to a target task.
Critical strengths
-
The paper solves a practical question: which synthetic examples help, and how many should be used? It argues well that neither quality nor diversity alone answers such a question perfectly. An example can add new information but carry a wrong label, or have a correct label but add nothing new.
-
The linear analysis is simple and easy to follow. It shows that the value of synthetic data depends on direction, on amount, and on what is already in the training set.
-
The proofs are complete, and the text experiment shows that coverage alone cannot separate helpful from harmful examples
Critical weaknesses
-
Weak connection between the theory and the method, and limited novelty. The theory's main insights are the optimal amount of synthetic data and the interaction between a candidate's error and the existing bias. TATC uses neither: the budget is fixed by hand, and Table 4 admits the bias interaction is omitted. The training-effect score is a local influence estimate, close to the score used by OptimSyn, and is not derived from the theory. The coverage term is the A-optimal design criterion used by BAIT, which the paper acknowledges, and the bias–variance analysis of synthetic data is close to Rezaei et al. (2026). The conclusion's claim of a "novel perspective" (that an example's value depends on the existing data) also overstates the contribution, since Data Shapley and active learning rest on the same idea.
-
Limited experiments. The LLM evaluation uses only one small model on one benchmark (GSM8K), and the text and image experiments test only the linear theory, so it is unclear whether TATC works on other tasks and models. The abstract should make this clear, and evaluating TATC on at least one non-math text or image task would better support its effectiveness. The only ablation removes the training-effect score; there is no variant using the training-effect score alone, so the value of the coverage step is unclear, and there is no sensitivity analysis for κ, the PCA dimension, or ρ. The paper also does not report the compute or time cost of computing per-example gradients and AdamW updates for all 4,096 candidates. Finally, Figure 3(b) shows only five of the ten methods; all baselines should be included across budgets.
Questions for authors
-
Does TATC improve results on other tasks (e.g., MATH, code, or instruction following) or with larger models? Can it be applied to an image task?
-
How could TATC use the theory's amount-selection result to choose the budget, instead of fixing it by hand? How much performance is lost by omitting the interaction between a candidate's error and the existing bias?
-
How does the training-effect score differ from OptimSyn's influence score, beyond the curvature term? Can it be derived from the marginal value in Theorem 7?
-
How does TATC perform with the training-effect score alone? How sensitive is it to k, the PCA dimension, and p?
-
What is the compute and time cost of TATC (per-example gradients and AdamW updates for all 4,096 candidates), compared with the baselines?
-
Beyond BAIT's A-optimal criterion and the analysis in Rezaei et al. (2026), what are the new theoretical results in this paper?