- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether individual-level scoring is sufficient for synthetic data curation and argues that effective selection requires group-level signals that capture interactions among samples. The authors construct groups with similar average individual oracle scores but different compositions, showing that their group oracle scores and downstream utility can differ. They then compare individual- and group-level selection methods under increasing paraphrase redundancy, finding that methods incorporating relational information generally perform better on generative tasks. Finally, they motivate a gradient-diversity diagnostic through an oracle-gap bound and use it to allocate a limited budget of group-oracle evaluations.
Critical strengths
The paper addresses an important practical problem.
As synthetic data becomes increasingly prevalent in model training, understanding how redundancy and interactions affect dataset utility is valuable. Moving beyond individual sample quality toward the utility of sample combinations is a meaningful perspective.
The proxy-matched experiments provide a targeted investigation of individual scoring limitations.
Constructing groups with similar average individual scores but different compositions helps demonstrate that the average individual score does not fully characterize group utility.
The evaluation covers both pre-training and post-training.
Studying both settings broadens the empirical scope. Separating generative and multiple-choice benchmarks also reveals differences that aggregate performance alone could obscure.
Critical weaknesses
The theoretical analysis does not directly match the experimental implementation.
Appendix C assumes a single ordinary gradient step, under which the group update equals the average of individual updates. However, the oracle implementation in Appendix B.1 uses AdamW, whose transformation of gradients generally does not preserve this relationship. Consequently, the cancellation underlying the theoretical bound does not directly apply to the experimental oracle.
Moreover, the theory uses the variance of unnormalized gradients, whereas the implemented diagnostic uses pairwise cosine diversity of normalized projected gradients. The paper should establish a clearer connection between these quantities or explicitly limit the theoretical claim to an idealized setting.
The controlled experiments establish insufficiency of the mean individual score, but not the broader necessity claim.
Section 3 deliberately restricts the variation in average individual oracle scores while varying group composition. This is useful for demonstrating a limitation of that particular statistic, but it also reduces the baseline’s ability to discriminate among candidate groups by construction.
The evidence therefore supports the narrower conclusion that groups with similar average individual scores can have different utility. Establishing that effective synthetic data curation requires group-level signals would need additional evidence on naturally constructed candidate pools and stronger alternatives based on individual-score distributions or randomized selection.
The experiments do not sufficiently separate synthetic-data effects from ordinary redundancy and coverage effects.
In Section 4, increasing the rephrase factor while fixing pool size simultaneously reduces the number of unique source documents. Thus, the experiment changes redundancy, source coverage, and effective information content together. All pools are also synthetic, so this is not directly an experiment on increasing the proportion of synthetic data.
Controls such as repeated organic documents, source-balanced sampling, and retaining one paraphrase per source would help determine whether the observed effects are specific to synthetic data or primarily reflect reduced content coverage.
Stronger baselines and matched ablations are needed to isolate the value of relational modeling.
Individual-level methods largely use deterministic top-score selection, while group-level methods introduce diversity or redundancy penalties. Their advantage may therefore reflect the benefit of a general diversity constraint.
Relevant comparisons include quality filtering followed by deduplication, per-source quotas, quality combined with embedding diversity, and temperature-based sampling from individual scores. For GMRel, an ablation using the same oracle supervision with and without the relational component would help isolate the contribution of relationship modeling.
The budget-allocation method lacks end-to-end validation and requires clearer score calibration.
Section 5 evaluates recovery of oracle rankings rather than downstream performance after selecting data and training a model. Recovering high-ranked groups does not necessarily establish that the method preserves the training benefits of full group scoring.
The implementation also combines oracle loss changes for audited groups with GradSim scores for unaudited groups. These quantities need not share the same scale or orientation, and the calibration procedure should be specified.
Finally, an upper bound increasing with gradient diversity does not establish that high-diversity groups actually have large errors or that auditing them is optimal. The allocation strategy should be presented as an empirically supported heuristic rather than a direct consequence of the bound.
Some claims are stronger than, or inconsistent with, the reported results.
At , Table 9 shows that default GMRel and GREATS both underperform random selection and the initial model. Furthermore, Figure 5d appears to place the enhanced relational-weight result at approximately 28.9%, below the initial-model baseline of 29.65% in Table 9, despite the text stating that it surpasses the baseline. This discrepancy needs clarification.
The efficiency claims also rely on a FLOPs-based estimate of cumulative memory-time cost rather than measured peak memory or GPU runtime. Direct measurements would better substantiate the practical savings.
Questions for authors
Please refer to the weaknesses.
6 comments
Author Response (2/4)
Establishing that effective synthetic data curation requires group-level signals would need additional evidence on naturally constructed candidate pools and stronger alternatives based on individual-score distributions or randomized selection.
Thank you for this comment. In order to establish the effectiveness of group-level signals beyond the pools from Section 3 of the paper, we conduct data selection using the individual proxy oracle and group oracle on a GooseReason pool. The setup is similar to the one in Section 3 (500 groups of 128 samples, where each group is drawn at random across all 20 embedding clusters), except that we do not control for the individual proxy scores of the groups, mirroring naturally constructed candidate pools.
From this pool, we select 25 groups with Gumbel-Top-k sampling (temperature 0.5) using either the individual proxy oracle or the group oracle, and compare with random group selection. We then train Qwen2.5-1.5B-Instruct with GRPO for 50 steps and report the average over six benchmarks (MBPP, MBPP+, GPQA-Diamond, BBH, IFEval, AIME24) across three seeds.
| Selection | Seed 1 | Seed 2 | Seed 3 | Mean | vs. base model, (32.88) |
|---|---|---|---|---|---|
| Random | 32.03 | 33.10 | 33.69 | 32.94 | +0.2% |
| Individual proxy oracle | 32.89 | 31.77 | 32.87 | 32.51 | −1.1% |
| Group oracle | 32.95 | 33.57 | 34.87 | 33.80 | +2.8% |
As shown above, the group oracle outperforms the individual proxy oracle on every seed, and is the only method that improves over the base model. This shows that group-level signals remain beneficial even when the individual proxy scores are not controlled. We will include these results in the new version of the paper pdf, and expand our analysis to additional settings as well.
Controls such as repeated organic documents, source-balanced sampling, and retaining one paraphrase per source would help determine whether the observed effects are specific to synthetic data or primarily reflect reduced content coverage.
We would like to clarify that synthetic data recycling is a multi-faceted process in which rephrasing intensity, source coverage, and near-duplicate redundancy change simultaneously, as reflected in prior work [3, 4, 5]. The goal of our study is not to isolate the effect of each factor, but to analyze data selection in a realistic synthetic data recycling setting. Specifically, in this setting the data pool contains rephrased low-quality documents (as in RePro [6] and REWIRE [3]), and the source corpus shrinks, mirroring the approaching limit on organic web text [7]. Since this form of synthetic recycling is increasingly adopted in frontier LLMs [5], we believe it is important to study it holistically.
[3] Nguyen et al. Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models. COLM 2025.
[4] Maini et al. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. ACL 2024.
[5] Kimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534, 2025.
[6] Yu and Xiong. RePro: Training Language Models to Faithfully Recycle the Web for Pretraining. arXiv:2510.10681, 2025.
[7] Villalobos et al. Position: Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data. ICML 2024.
Stronger baselines and matched ablations are needed to isolate the value of relational modeling. Individual-level methods largely use deterministic top-score selection, while group-level methods introduce diversity or redundancy penalties. Their advantage may therefore reflect the benefit of a general diversity constraint.
Thank you for this comment. Below, we include additional comparisons at r = 10 with BM25 and GradSim using Gumbel sampling (to introduce more diversity) as stronger baselines beyond top-k selection. As shown below, even with added diversity, these individual-level selection methods fall short of the group-level methods. This highlights that it is not just added diversity that improves performance, but rather the relational information captured by the group-level methods (e.g., GMRel's learned relational weight and the gradient-based relational penalty in GREATS).
| Method | Mean |
|---|---|
| GMRel | 30.40 |
| GREATS | 30.27 |
| BM25 (top-k) | 28.40 |
| FineWeb-Edu (top-k) | Paper |
| GradSim (top-k) | 28.04 |
| GradSim Gumbel (temp = 0.1) | 28.09 |
| GradSim Gumbel (temp = 0.3) | 29.79 |
| BM25 Gumbel (temp = 0.1) | 29.47 |
| BM25 Gumbel (temp = 0.3) | 29.27 |
| FineWeb-Edu Gumbel (temp = 0.1) | 28.29 |
| FineWeb-Edu Gumbel (temp = 0.3) | 29.06 |
| FineWeb-Edu Gumbel (temp = 0.4) | 29.75 |
Author Response (3/4)
For GMRel, an ablation using the same oracle supervision with and without the relational component would help isolate the contribution of relationship modeling.
Thank you for this suggestion. Below, we provide results for GMRel without its relational component, compared against GMRel with relational weight τ = 1.01 and τ = 0.30 at r = 50. As shown below, GMRel without the relational component performs worse than GMRel with it and than random selection, highlighting the necessity of the relational component.
| Method | Seed 1 | Seed 2 | Seed 3 | Mean |
|---|---|---|---|---|
| GMRel (no relational component) | 27.17 | 28.06 | 27.22 | 27.48 |
| GMRel τ = 1.01 | 27.71 | 28.50 | 27.46 | 27.89 |
| GMRel τ = 0.30 | 27.89 | 28.38 | 29.18 | 28.48 |
| Random | 27.75 | 28.19 | 29.35 | 28.43 |
The implementation also combines oracle loss changes for audited groups with GradSim scores for unaudited groups. These quantities need not share the same scale or orientation, and the calibration procedure should be specified.
Thank you for pointing this out. We clarify the calibration procedure below and will include it in the new version of the paper pdf. In our experiments, we fit group oracle ≈ a · GradSim + b · GradDiv + c on the audited groups to calibrate the scales.
Recovering high-ranked groups does not necessarily establish that the method preserves the training benefits of full group scoring.
Thank you for this point. To test this directly, we trained on groups selected with GradDiv-guided allocation, using the pre-training pool and training setup from Section 3. We audit using a 20% budget with the group oracle, calibrate the unaudited groups' scores (using the caliration procedure described above), and then select 40 groups with Gumbel-Top-k (temp = 0.5).
We report the results of training using groups selected using GradDiv-guided allocation, versus using full group oracle. Using only 20% of the oracle calls, GradDiv allocation (31.64) outperforms both the individual proxy oracle, and random selection, recovering part of the benefit of full group scoring (31.98).
| Method | Mean |
|---|---|
| Random (paper) | 30.86 |
| Individual proxy oracle (paper) | 31.20 |
| Group oracle (paper) | 31.98 |
| GradDiv audit (200) + calibration | 31.64 |
Finally, an upper bound increasing with gradient diversity does not establish that high-diversity groups actually have large errors or that auditing them is optimal. The allocation strategy should be presented as an empirically supported heuristic rather than a direct consequence of the bound.
Thank you for this observation. While an upper bound does not guarantee that high-diversity groups have large errors, it does tell us that groups with low gradient diversity cannot have a large gap between the individual proxy and the group oracle. This motivates spending group oracle calls on high-diversity groups as verification. We will clarify this in our paper revision.
Empirically, we show in our paper that this heuristic holds up well. GradDiv correlates with the actual gap between the individual proxy and the group oracle (Spearman 0.58–0.63 across our four candidate pools), and at a 20% oracle budget, auditing the most gradient-diverse groups recovers 75% of the group oracle's top-10% groups, compared to 20% under random allocation (Section 5, Finding 1).
Author Response (1/4)
We appreciate your acknowledgement that our work addresses an important practical problem, that the proxy-matched experiments provide a targeted investigation of the limitations of individual scoring, and that our evaluation covers both pre-training and post-training. We address your comments directly below, and we will revise our paper PDF based on your feedback; the updated version will be uploaded.
Thank you for this feedback. While our theoretical analysis was conducted for SGD, below we show that the same bound also applies to AdamW.
Recall the AdamW update:
mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2,
m^t=1−β1tmt,v^t=1−β2tvt,
θt=θt−1−η(v^t+ϵm^t+λθt−1).
Here, θt are the model parameters at step t, gt is the loss gradient, mt and vt are running estimates of the gradient's first and second moments (m^t, v^t are their bias-corrected versions), β1,β2 are their decay rates, η is the learning rate, λ is the weight decay, and ϵ is a small constant for numerical stability.
Our oracle creates a fresh optimizer for each group, so m0=v0=0. After one step, bias correction gives
m^1=g,v^1=g2,
and the update is
u(g)=−η∣g∣+ϵg−ηλθ≈−ηsign(g)−ηλθ.
Bound. We apply the same Taylor/Jensen argument as in Proposition 5.1 to the sign updates:
uz=−ηsign(gz),uG=−ηsign(gG),uˉ=∣G∣1∑z∈Guz.
This gives
∣Iproxy−Iorcl∣≤2η∥∇Φ(uG)∥P+2βη2P,
where
P=∣G∣1∑z∈Gdz,
and dz is the number of coordinates in which the sign of sample z's gradient differs from that of the group gradient gG. That is, P is the average number of coordinates in which a sample's gradient sign disagrees with the group's. The bound is zero when all per-sample gradients agree in sign with the group, and it grows as their directions diverge. This mirrors the effect shown in the theoretical analysis of our paper: the more the samples pull the model in different directions, the more the group oracle can deviate from the individual proxy oracle.
We will include these details in the new version of the paper pdf.
Thank you for this question. First, we compare the two versions of GradDiv empirically: the raw version and the practical version (computed from JL-projected, L2-normalized gradients). The figure here (https://files.catbox.moe/eyjs2s.png) shows the correlation between the two versions on ~650 groups of size 512 from GooseReason.
The two versions are positively correlated (Spearman ρ = 0.61), indicating that the practical version of GradDiv can serve as a strong proxy for raw GradDiv.
Second, we note that JL projection and L2 normalization, the two components of the practical GradDiv, are widely used in practice. For example, LESS [1] and TRAK [2] both use JL-projected and L2-normalized gradients to score and select training data. Together, these results show that our practical GradDiv is not arbitrary, but rather follows procedures commonly adopted in the LLM data influence and broader data curation communities.
[1] Xia et al. LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 2024.
[2] Park et al. TRAK: Attributing Model Behavior at Scale. ICML 2023.