acceptodds
Under review as a conference paper at ICLR 2027

Effective Synthetic Data Curation Requires Group-Level Signals

Abstract

Synthetic data now is essential to LLM training, used to strengthen advanced capabilities such as autonomous and long-horizon task execution. Yet recent work shows that training on it at scale can degrade model generation, making it important to decide what synthetic data is worth training on. While current data curation practices do so with individual-level signals (i.e., estimates of each data sample's training utility in isolation), across pre-training and post-training settings we show that this is insufficient for synthetic data, and that group-level signals (i.e., estimates of utility that account for interactions among data samples) are necessary for effective data curation. First, we show that individual-level signals are blind to how samples jointly affect training: synthetic datasets with different compositions can be indistinguishable under individual-level influence yet differ sharply under group-level influence, and curating by the latter yields better downstream performance, particularly in generative capability. Second, we find that group-level signals matter more as training pipelines become increasingly synthetic: among widely used data curation methods, only those incorporating them improve over baseline, with gains increasing when weights capturing relations among samples are amplified. Finally, we translate these findings into practice – for model developers under a compute budget, we offer a cheap diagnostic that prioritizes which groups of synthetic data most need group-level estimation, recovering much of the benefit of full group-level scoring at a fraction of the compute cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
synthetic data, data curation, data influence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CfqMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether individual-level scoring is sufficient for synthetic data curation and argues that effective selection requires group-level signals that capture interactions among samples. The authors construct groups with similar average individual oracle scores but different compositions, showing that their group oracle scores and downstream utility can differ. They then compare individual- and group-level selection methods under increasing paraphrase redundancy, finding that methods incorporating relational information generally perform better on generative tasks. Finally, they motivate a gradient-diversity diagnostic through an oracle-gap bound and use it to allocate a limited budget of group-oracle evaluations.

Critical strengths

The paper addresses an important practical problem.

As synthetic data becomes increasingly prevalent in model training, understanding how redundancy and interactions affect dataset utility is valuable. Moving beyond individual sample quality toward the utility of sample combinations is a meaningful perspective.

The proxy-matched experiments provide a targeted investigation of individual scoring limitations.

Constructing groups with similar average individual scores but different compositions helps demonstrate that the average individual score does not fully characterize group utility.

The evaluation covers both pre-training and post-training.

Studying both settings broadens the empirical scope. Separating generative and multiple-choice benchmarks also reveals differences that aggregate performance alone could obscure.

Critical weaknesses

The theoretical analysis does not directly match the experimental implementation.

Appendix C assumes a single ordinary gradient step, under which the group update equals the average of individual updates. However, the oracle implementation in Appendix B.1 uses AdamW, whose transformation of gradients generally does not preserve this relationship. Consequently, the cancellation underlying the theoretical bound does not directly apply to the experimental oracle.

Moreover, the theory uses the variance of unnormalized gradients, whereas the implemented diagnostic uses pairwise cosine diversity of normalized projected gradients. The paper should establish a clearer connection between these quantities or explicitly limit the theoretical claim to an idealized setting.

The controlled experiments establish insufficiency of the mean individual score, but not the broader necessity claim.

Section 3 deliberately restricts the variation in average individual oracle scores while varying group composition. This is useful for demonstrating a limitation of that particular statistic, but it also reduces the baseline’s ability to discriminate among candidate groups by construction.

The evidence therefore supports the narrower conclusion that groups with similar average individual scores can have different utility. Establishing that effective synthetic data curation requires group-level signals would need additional evidence on naturally constructed candidate pools and stronger alternatives based on individual-score distributions or randomized selection.

The experiments do not sufficiently separate synthetic-data effects from ordinary redundancy and coverage effects.

In Section 4, increasing the rephrase factor while fixing pool size simultaneously reduces the number of unique source documents. Thus, the experiment changes redundancy, source coverage, and effective information content together. All pools are also synthetic, so this is not directly an experiment on increasing the proportion of synthetic data.

Controls such as repeated organic documents, source-balanced sampling, and retaining one paraphrase per source would help determine whether the observed effects are specific to synthetic data or primarily reflect reduced content coverage.

Stronger baselines and matched ablations are needed to isolate the value of relational modeling.

Individual-level methods largely use deterministic top-score selection, while group-level methods introduce diversity or redundancy penalties. Their advantage may therefore reflect the benefit of a general diversity constraint.

Relevant comparisons include quality filtering followed by deduplication, per-source quotas, quality combined with embedding diversity, and temperature-based sampling from individual scores. For GMRel, an ablation using the same oracle supervision with and without the relational component would help isolate the contribution of relationship modeling.

The budget-allocation method lacks end-to-end validation and requires clearer score calibration.

Section 5 evaluates recovery of oracle rankings rather than downstream performance after selecting data and training a model. Recovering high-ranked groups does not necessarily establish that the method preserves the training benefits of full group scoring.

The implementation also combines oracle loss changes for audited groups with GradSim scores for unaudited groups. These quantities need not share the same scale or orientation, and the calibration procedure should be specified.

Finally, an upper bound increasing with gradient diversity does not establish that high-diversity groups actually have large errors or that auditing them is optimal. The allocation strategy should be presented as an empirically supported heuristic rather than a direct consequence of the bound.

Some claims are stronger than, or inconsistent with, the reported results.

At r=50r=50, Table 9 shows that default GMRel and GREATS both underperform random selection and the initial model. Furthermore, Figure 5d appears to place the enhanced relational-weight result at approximately 28.9%, below the initial-model baseline of 29.65% in Table 9, despite the text stating that it surpasses the baseline. This discrepancy needs clarification.

The efficiency claims also rely on a FLOPs-based estimate of cumulative memory-time cost rather than measured peak memory or GPU runtime. Direct measurements would better substantiate the practical savings.

Questions for authors

Please refer to the weaknesses.

6 comments
AuthorsOpenReview ↗

Author Response (1/4)

We appreciate your acknowledgement that our work addresses an important practical problem, that the proxy-matched experiments provide a targeted investigation of the limitations of individual scoring, and that our evaluation covers both pre-training and post-training. We address your comments directly below, and we will revise our paper PDF based on your feedback; the updated version will be uploaded.

The theoretical analysis does not directly match the experimental implementation. Consequently, the cancellation underlying the theoretical bound does not directly apply to the experimental oracle.

Thank you for this feedback. While our theoretical analysis was conducted for SGD, below we show that the same bound also applies to AdamW.

Recall the AdamW update:

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2,m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2,

m^t=mt1−β1t,v^t=vt1−β2t,\hat m_t = \frac{m_t}{1-\beta_1^{t}}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^{t}},

θt=θt−1−η(m^tv^t+ϵ+λθt−1).\theta_t = \theta_{t-1} - \eta\left(\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon} + \lambda \theta_{t-1}\right).

Here, θt\theta_t are the model parameters at step tt, gtg_t is the loss gradient, mtm_t and vtv_t are running estimates of the gradient's first and second moments (m^t\hat m_t, v^t\hat v_t are their bias-corrected versions), β1,β2\beta_1, \beta_2 are their decay rates, η\eta is the learning rate, λ\lambda is the weight decay, and ϵ\epsilon is a small constant for numerical stability.

Our oracle creates a fresh optimizer for each group, so m0=v0=0m_0 = v_0 = 0. After one step, bias correction gives

m^1=g,v^1=g2,\hat m_1 = g, \qquad \hat v_1 = g^2,

and the update is

u(g)=−ηg∣g∣+ϵ−ηλθ≈−ηsign(g)−ηλθ.u(g) = -\eta \frac{g}{|g|+\epsilon} - \eta\lambda\theta \approx -\eta \mathrm{sign}(g) - \eta\lambda\theta.

Bound. We apply the same Taylor/Jensen argument as in Proposition 5.1 to the sign updates:

uz=−ηsign(gz),uG=−ηsign(gG),uˉ=1∣G∣∑z∈Guz.u_z = -\eta \mathrm{sign}(g_z), \qquad u_G = -\eta \mathrm{sign}(g_G), \qquad \bar u = \frac{1}{|G|}\sum_{z\in G} u_z.

This gives

∣Iproxy−Iorcl∣≤2η∥∇Φ(uG)∥P+2βη2P,|\mathcal I_{\mathrm{proxy}} - \mathcal I_{\mathrm{orcl}}| \le 2\eta \lVert\nabla\Phi(u_G)\rVert \sqrt{P} + 2\beta\eta^2 P,

where

P=1∣G∣∑z∈Gdz,P = \frac{1}{|G|}\sum_{z\in G} d_z,

and dzd_z is the number of coordinates in which the sign of sample zz's gradient differs from that of the group gradient gGg_G. That is, PP is the average number of coordinates in which a sample's gradient sign disagrees with the group's. The bound is zero when all per-sample gradients agree in sign with the group, and it grows as their directions diverge. This mirrors the effect shown in the theoretical analysis of our paper: the more the samples pull the model in different directions, the more the group oracle can deviate from the individual proxy oracle.

We will include these details in the new version of the paper pdf.

Moreover, the theory uses the variance of unnormalized gradients, whereas the implemented diagnostic uses pairwise cosine diversity of normalized projected gradients. The paper should establish a clearer connection between these quantities or explicitly limit the theoretical claim to an idealized setting.

Thank you for this question. First, we compare the two versions of GradDiv empirically: the raw version and the practical version (computed from JL-projected, L2-normalized gradients). The figure here (https://files.catbox.moe/eyjs2s.png) shows the correlation between the two versions on ~650 groups of size 512 from GooseReason.

The two versions are positively correlated (Spearman ρ = 0.61), indicating that the practical version of GradDiv can serve as a strong proxy for raw GradDiv.

Second, we note that JL projection and L2 normalization, the two components of the practical GradDiv, are widely used in practice. For example, LESS [1] and TRAK [2] both use JL-projected and L2-normalized gradients to score and select training data. Together, these results show that our practical GradDiv is not arbitrary, but rather follows procedures commonly adopted in the LLM data influence and broader data curation communities.

[1] Xia et al. LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 2024.

[2] Park et al. TRAK: Attributing Model Behavior at Scale. ICML 2023.

AuthorsOpenReview ↗

Author Response (2/4)

Establishing that effective synthetic data curation requires group-level signals would need additional evidence on naturally constructed candidate pools and stronger alternatives based on individual-score distributions or randomized selection.

Thank you for this comment. In order to establish the effectiveness of group-level signals beyond the pools from Section 3 of the paper, we conduct data selection using the individual proxy oracle and group oracle on a GooseReason pool. The setup is similar to the one in Section 3 (500 groups of 128 samples, where each group is drawn at random across all 20 embedding clusters), except that we do not control for the individual proxy scores of the groups, mirroring naturally constructed candidate pools.

From this pool, we select 25 groups with Gumbel-Top-k sampling (temperature 0.5) using either the individual proxy oracle or the group oracle, and compare with random group selection. We then train Qwen2.5-1.5B-Instruct with GRPO for 50 steps and report the average over six benchmarks (MBPP, MBPP+, GPQA-Diamond, BBH, IFEval, AIME24) across three seeds.

SelectionSeed 1Seed 2Seed 3Meanvs. base model, (32.88)
Random32.0333.1033.6932.94+0.2%
Individual proxy oracle32.8931.7732.8732.51−1.1%
Group oracle32.9533.5734.8733.80+2.8%

As shown above, the group oracle outperforms the individual proxy oracle on every seed, and is the only method that improves over the base model. This shows that group-level signals remain beneficial even when the individual proxy scores are not controlled. We will include these results in the new version of the paper pdf, and expand our analysis to additional settings as well.

Controls such as repeated organic documents, source-balanced sampling, and retaining one paraphrase per source would help determine whether the observed effects are specific to synthetic data or primarily reflect reduced content coverage.

We would like to clarify that synthetic data recycling is a multi-faceted process in which rephrasing intensity, source coverage, and near-duplicate redundancy change simultaneously, as reflected in prior work [3, 4, 5]. The goal of our study is not to isolate the effect of each factor, but to analyze data selection in a realistic synthetic data recycling setting. Specifically, in this setting the data pool contains rephrased low-quality documents (as in RePro [6] and REWIRE [3]), and the source corpus shrinks, mirroring the approaching limit on organic web text [7]. Since this form of synthetic recycling is increasingly adopted in frontier LLMs [5], we believe it is important to study it holistically.

[3] Nguyen et al. Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models. COLM 2025.

[4] Maini et al. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. ACL 2024.

[5] Kimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534, 2025.

[6] Yu and Xiong. RePro: Training Language Models to Faithfully Recycle the Web for Pretraining. arXiv:2510.10681, 2025.

[7] Villalobos et al. Position: Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data. ICML 2024.

Stronger baselines and matched ablations are needed to isolate the value of relational modeling. Individual-level methods largely use deterministic top-score selection, while group-level methods introduce diversity or redundancy penalties. Their advantage may therefore reflect the benefit of a general diversity constraint.

Thank you for this comment. Below, we include additional comparisons at r = 10 with BM25 and GradSim using Gumbel sampling (to introduce more diversity) as stronger baselines beyond top-k selection. As shown below, even with added diversity, these individual-level selection methods fall short of the group-level methods. This highlights that it is not just added diversity that improves performance, but rather the relational information captured by the group-level methods (e.g., GMRel's learned relational weight and the gradient-based relational penalty in GREATS).

MethodMean
GMRel30.40
GREATS30.27
BM25 (top-k)28.40
FineWeb-Edu (top-k)Paper
GradSim (top-k)28.04
GradSim Gumbel (temp = 0.1)28.09
GradSim Gumbel (temp = 0.3)29.79
BM25 Gumbel (temp = 0.1)29.47
BM25 Gumbel (temp = 0.3)29.27
FineWeb-Edu Gumbel (temp = 0.1)28.29
FineWeb-Edu Gumbel (temp = 0.3)29.06
FineWeb-Edu Gumbel (temp = 0.4)29.75
AuthorsOpenReview ↗

Author Response (3/4)

For GMRel, an ablation using the same oracle supervision with and without the relational component would help isolate the contribution of relationship modeling.

Thank you for this suggestion. Below, we provide results for GMRel without its relational component, compared against GMRel with relational weight τ = 1.01 and τ = 0.30 at r = 50. As shown below, GMRel without the relational component performs worse than GMRel with it and than random selection, highlighting the necessity of the relational component.

MethodSeed 1Seed 2Seed 3Mean
GMRel (no relational component)27.1728.0627.2227.48
GMRel τ = 1.0127.7128.5027.4627.89
GMRel τ = 0.3027.8928.3829.1828.48
Random27.7528.1929.3528.43

The implementation also combines oracle loss changes for audited groups with GradSim scores for unaudited groups. These quantities need not share the same scale or orientation, and the calibration procedure should be specified.

Thank you for pointing this out. We clarify the calibration procedure below and will include it in the new version of the paper pdf. In our experiments, we fit group oracle ≈ a · GradSim + b · GradDiv + c on the audited groups to calibrate the scales.

Recovering high-ranked groups does not necessarily establish that the method preserves the training benefits of full group scoring.

Thank you for this point. To test this directly, we trained on groups selected with GradDiv-guided allocation, using the pre-training pool and training setup from Section 3. We audit using a 20% budget with the group oracle, calibrate the unaudited groups' scores (using the caliration procedure described above), and then select 40 groups with Gumbel-Top-k (temp = 0.5).

We report the results of training using groups selected using GradDiv-guided allocation, versus using full group oracle. Using only 20% of the oracle calls, GradDiv allocation (31.64) outperforms both the individual proxy oracle, and random selection, recovering part of the benefit of full group scoring (31.98).

MethodMean
Random (paper)30.86
Individual proxy oracle (paper)31.20
Group oracle (paper)31.98
GradDiv audit (200) + calibration31.64

Finally, an upper bound increasing with gradient diversity does not establish that high-diversity groups actually have large errors or that auditing them is optimal. The allocation strategy should be presented as an empirically supported heuristic rather than a direct consequence of the bound.

Thank you for this observation. While an upper bound does not guarantee that high-diversity groups have large errors, it does tell us that groups with low gradient diversity cannot have a large gap between the individual proxy and the group oracle. This motivates spending group oracle calls on high-diversity groups as verification. We will clarify this in our paper revision.

Empirically, we show in our paper that this heuristic holds up well. GradDiv correlates with the actual gap between the individual proxy and the group oracle (Spearman 0.58–0.63 across our four candidate pools), and at a 20% oracle budget, auditing the most gradient-diverse groups recovers 75% of the group oracle's top-10% groups, compared to 20% under random allocation (Section 5, Finding 1).

Reviewer_bFwcOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates synthetic data curation, focusing on the role of group-level signals in data selection. It compares group-level and individual-level selection strategies to assess their impact on model performance. The authors also explore how interactions among samples affect data utility and propose a gradient-based diagnostic to identify groups that may benefit from group-level evaluation under limited computational budgets.

Critical strengths

The paper provides a mathematical proof supporting one of its main claims, strengthening the theoretical foundation of the proposed approach.

Critical weaknesses

The paper has organizational issues, with frequent references to methods and concepts adopted "from prior work." This makes it difficult to distinguish the authors' original contributions from existing approaches. Although the paper presents interesting empirical observations, the novelty relative to prior work is not sufficiently clear, which limits the perceived significance of the contribution.

Lines 047–049: This statement requires additional citations and qualification. There are data curation approaches based on other criteria, such as data uncertainty, that do not necessarily rely on utility-based metrics. The authors should acknowledge these alternatives rather than framing data curation primarily in terms of individual-level utility estimation.

Line 066: The authors need to clearly define what they mean by diversity in this context. It is unclear whether they refer to semantic, lexical, gradient-based, or another form of diversity.

Line 080: This section is difficult to follow. The authors could provide a clearer explanation of the central problem addressed in the paper, potentially supported by figures or illustrations to make the motivation more intuitive. Moreover, several statements are insufficiently explained and refer to concepts or findings that are only introduced later in the paper. This makes the progression of the argument difficult to follow.

Line 100: It is unclear whether the metric being discussed is proposed by the authors or adopted from previous work. Later in the paper, it appears to be based on an existing approach. The authors should explicitly clarify its origin and distinguish their contributions from previously established methods.

Figure 1a: The authors should provide a clearer definition of how the rare-word fraction is calculated, including the criterion used to determine whether a word is considered rare.

Figure 1b: This figure is difficult to interpret, and its description is insufficient. The authors should explain more clearly what is being compared and how the figure supports their argument.

Line 153: This is not the conclusion I drew from the previous section. Rather than conclusively establishing the claimed relationship, the preceding discussion appears to make several statements without sufficient supporting evidence. The authors should strengthen the argument and clarify how the previous section supports this conclusion.

In the pre-training description, the authors provide insufficient information about the dataset and its characteristics. A clearer description of its composition, distribution, and relevant properties would help readers understand the experimental setting and assess the generalizability of the results.

Line 200: The details deferred to the appendix should be included, at least in summarized form, in the main text, as they are essential components of the methodology. The paper devotes considerable space to short paragraphs and repetitive explanations, while important methodological details are placed in the appendix. I believe the organization could be improved by prioritizing essential methodological information in the main body and moving less critical material to the appendix.

Table 1b: The results should include confidence intervals or another measure of experimental variability. Changes in group configurations may affect the reported performance, and without uncertainty estimates, it is difficult to assess the robustness of the observed improvements. Additionally, the group construction strategy should be evaluated through an ablation study. In particular, varying the number of clusters, group composition, and grouping strategy would help determine whether the observed improvements arise from group-level influence estimation or are partially driven by the construction procedure itself.

Overall, I find the work insufficiently developed in its current form, particularly regarding the distinction between its original contributions and the methods adopted from prior work. Several methodological components build directly on existing approaches, and the authors should more clearly articulate what is fundamentally new, what constitutes an empirical extension of previous findings, and how the presented evidence supports the claimed contributions.

Line 271: This statement is too strong. The experiments investigate a limited set of data curation techniques, and the conclusions should not be generalized to data curation as a whole without broader empirical support. In particular, demonstrating the limitations of the evaluated individual-level methods does not establish that group-level signals are universally necessary for effective synthetic data curation.

The title of Section 4 should be more specific, such as "Group-Level Signals for Gradient- and Influence-Based Synthetic Data Curation," to better reflect the scope of the methods and experiments discussed.

hroughout the paper, the authors frequently refer to concepts, methods, or findings that are only defined or explained in later sections. This affects readability and makes it unnecessarily difficult to follow the progression of the argument.

The authors compare different data selection methods rather than the broader range of data curation strategies. This distinction is important, as data curation encompasses approaches beyond sample selection. The terminology and conclusions should better reflect the actual scope of the experiments.

The authors should dedicate a section in the main text to explaining what each method in Lines 312–315 does and how the methods differ from one another. Simply listing the methods and referring readers to the appendix makes it difficult to understand the rationale behind the comparisons and interpret the results.

If the authors reduced the excessive fragmentation of the text into short paragraphs, there would be more room to include relevant methodological information in the main body. This would improve both the organization and the technical clarity of the paper.

Section 4.2: This section is somewhat redundant and could be condensed into a more focused discussion. The main message appears to be that different selection methods behave differently as the proportion of synthetic data increases, while group-level approaches generally outperform the evaluated individual-level approaches, particularly on generative tasks. This conclusion could be communicated more directly, avoiding repeated explanations of similar observations.

Section 5: Again, there appears to be some confusion between diversity, group utility, and group-level selection. Selecting groups based on their estimated utility or gradient-related characteristics does not automatically imply that the resulting data is more diverse. In particular, within-group gradient diversity is used as a diagnostic for identifying potential discrepancies between individual-level and group-level influence estimates, but this is not equivalent to demonstrating that the selected training data has greater semantic or distributional diversity. If the authors claim such a relationship, they should explicitly demonstrate it through appropriate definitions and empirical evidence. Otherwise, these concepts should be clearly distinguished throughout the discussion.

Questions for authors

Line 138: What evidence supports this statement? While the claim may hold for the specific dataset used in the experiments, it is not clear whether it can be generalized to synthetic data more broadly. The statement is too strong without further empirical evidence or appropriate qualification.

Line 146: What evidence supports this statement? The authors should provide a clearer justification for why these properties are necessarily invisible to individual-level scoring methods, particularly given the variety of possible individual-level signals.

Line 140: How does this observation necessarily imply homogeneity? The connection is not sufficiently established, and the conclusion appears too strong. The observed behavior could also be influenced by other factors, such as the score computation procedure, dataset characteristics, or model architecture. The authors should clarify this relationship and discuss possible alternative explanations.

If I understood correctly, the approach aims to identify groups of samples that reduce the reference loss. However, given that group utility is estimated using a relatively small reference set, how sensitive is the selection strategy to the composition of this set? Could optimizing group selection against a fixed reference set introduce selection bias or overfitting to the reference distribution? Although downstream evaluations provide some evidence of generalization, an analysis using different reference sets would help establish the robustness of the selection process.

The comparison of techniques is largely centered on gradient-based or influence-based approaches. Since the paper discusses data curation in a broader sense, it would benefit from comparisons with other families of data curation methods, including approaches that explicitly consider diversity, redundancy, or uncertainty without relying on group-level influence estimation. Without these comparisons, it is difficult to determine whether the conclusions extend beyond the specific approaches considered.

Line 199: Again, what exactly do the authors consider diverse? Moreover, how is diversity related to the objective of reducing the reference loss? A group that produces a greater reduction in loss is not necessarily more diverse. These are distinct properties, and their relationship should be explicitly defined and empirically supported rather than implicitly assumed.

12 comments
AuthorsOpenReview ↗

Author Response (1/9)

We appreciate your acknowledgement that our paper presents interesting empirical observations. We address your comments directly below, and we will revise our paper PDF based on your feedback; the updated version will be uploaded.

The paper has organizational issues, with frequent references to methods and concepts adopted "from prior work." This makes it difficult to distinguish the authors' original contributions from existing approaches. Although the paper presents interesting empirical observations, the novelty relative to prior work is not sufficiently clear, which limits the perceived significance of the contribution.

Thank you for this feedback. To clarify, our main contributions are as follows:

  1. We first show that individual-level signals can be weak discriminators for selecting synthetic data: In Section 3, we answer the question, "When groups of synthetic data look equivalent under individual-level signals, do they differ in their actual training value?" We show that even when synthetic data is "diverse" across topics/domains, individual-level signals struggle to discriminate between datasets, and selecting data with them underperforms group-level signals. Our novelty lies in a controlled design (proxy-matched candidate pools) that holds individual-level scores fixed while varying group composition, isolating the effect of sample interactions in both synthetic pre-training and RLVR post-training. We leverage oracle scoring from prior work [1, 2] as the signal for this analysis, as it provides a principled approach to data selection.
  2. Next, we show that group-level signals are important for data curation as organic data becomes scarcer and we increasingly rely on synthetic data: Here, we answer the question, "As training pipelines rely more on recycled synthetic data, do data selection methods need group-level signals to remain effective?", which mirrors the concern that models will exhaust available organic training data [3]. Our novelty lies in a controlled setting that varies recycling intensity (rephrases per source document) at a fixed pool size, within which we leverage existing data curation methods.
  3. Finally, we answer the question, "Under a limited compute budget, which groups should receive expensive group-level scoring?" We provide practitioners with a diagnostic for allocating this compute effectively. Our novelty lies in a bound on the gap between individual- and group-level scores in terms of within-group gradient diversity, and in GradDiv, a cheap diagnostic built on this bound.

To make this clearer, we will update each section with the research question we aim to address, and state clearly what prior tools we use to demonstrate these findings. We will also move a condensed version of our related work section into the main text. We will include these changes in the new version of the paper pdf.

[1] Koh and Liang. Understanding Black-box Predictions via Influence Functions. ICML 2017.

[2] Yu et al. MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models. NeurIPS 2024.

[3] Villalobos et al. Position: Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data. ICML 2024.

AuthorsOpenReview ↗

Author Response (2/9)

Lines 047–049: This statement requires additional citations and qualification. There are data curation approaches based on other criteria, such as data uncertainty, that do not necessarily rely on utility-based metrics. The authors should acknowledge these alternatives rather than framing data curation primarily in terms of individual-level utility estimation.

Thank you for this comment. Below, we provide examples of how data curation approaches based on other criteria largely use individual-level signals.

  1. Data uncertainty approaches [4, 5, 6]: Uncertainty sampling scores each sample by the model's predictive uncertainty on that sample alone (e.g., least-confidence [4], BALD [5], and its application to LLM instruction tuning [6]), so near-identical uncertain samples are all selected.
  2. Quality filtering approaches [7, 8, 9, 10]: Classifiers (e.g., FineWeb-Edu [10], DCLM [9]), LLM ratings (e.g., QuRating [8]), or perplexity (e.g., CCNet [7]) score each document independently and keep those above a threshold.
  3. Verification-based filtering [11]: Rejection sampling keeps a generated solution only if its answer is verified as correct (e.g., STaR [11]), again typically without regard to the rest of the dataset.
  4. LLM-judge and reward-model filtering [12]: A strong model rates each sample and low-rated samples are discarded, again without regard to the rest of the dataset.

While group-level methods exist for LLM data selection (e.g., GREATS [13], Group-MATES [14], and AirRep [15]), data curation for frontier models still relies mainly on individual-level methods (e.g., quality scoring and heuristic filtering in pre-training [9, 10, 16], and verification in post-training [11, 17]), largely because of the compute required to filter huge corpora. Our paper instead challenges this common practice by revealing that group-level signals are needed as synthetic data becomes increasingly prevalent in training pipelines. We will update lines 047–049 in the new version of the paper pdf to reflect these clarifications.

[4] Lewis and Gale. A Sequential Algorithm for Training Text Classifiers. SIGIR 1994.

[5] Gal et al. Deep Bayesian Active Learning with Image Data. ICML 2017.

[6] Kung et al. Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks. EMNLP 2023.

[7] Wenzek et al. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. LREC 2020.

[8] Wettig et al. QuRating: Selecting High-Quality Data for Training Language Models. ICML 2024.

[9] Li et al. DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. NeurIPS 2024.

[10] Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. NeurIPS 2024.

[11] Zelikman et al. STaR: Bootstrapping Reasoning With Reasoning. NeurIPS 2022.

[12] Chen et al. AlpaGasus: Training a Better Alpaca with Fewer Data. ICLR 2024.

[13] Wang et al. GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration. NeurIPS 2024.

[14] Yu et al. Data-Efficient Pretraining with Group-Level Data Influence Modeling. 2025.

[15] Sun et al. Enhancing Training Data Attribution with Representational Optimization. 2025.

[16] Grattafiori et al. The Llama 3 Herd of Models. arXiv:2407.21783, 2024.

[17] Guo et al. DeepSeek-R1 Incentivizes Reasoning in LLMs through Reinforcement Learning. Nature 2025.

Line 066: The authors need to clearly define what they mean by diversity in this context. It is unclear whether they refer to semantic, lexical, gradient-based, or another form of diversity.

Thank you for pointing this out. At line 066, we refer to within-group gradient diversity, i.e., how much the per-sample gradients within a group point in different directions, or equivalently, how differently the samples in a group would update the model. We will clarify this in the new version of the paper pdf.

Line 080: This section is difficult to follow. The authors could provide a clearer explanation of the central problem addressed in the paper, potentially supported by figures or illustrations to make the motivation more intuitive. Moreover, several statements are insufficiently explained and refer to concepts or findings that are only introduced later in the paper. This makes the progression of the argument difficult to follow.

Thank you for this suggestion. We will add an illustrative figure to the introduction showing the central problem: two groups with the same individual proxy score can have different group oracle scores, and lead to different downstream performance. A draft can be found at: https://files.catbox.moe/9xygt5.png

AuthorsOpenReview ↗

Author Response (3/9)

Line 100: It is unclear whether the metric being discussed is proposed by the authors or adopted from previous work. Later in the paper, it appears to be based on an existing approach. The authors should explicitly clarify its origin and distinguish their contributions from previously established methods.

Thank you for this comment. We would like to clarify that the oracle score (Definition 2.1) is adopted from prior work on data influence [1, 2], and we leverage these tools to demonstrate why group-level signals are needed. We will update line 100 in the new version of the paper pdf to clarify this.

[1] Koh and Liang. Understanding Black-box Predictions via Influence Functions. ICML 2017.

[2] Yu et al. MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models. NeurIPS 2024.

Figure 1a: The authors should provide a clearer definition of how the rare-word fraction is calculated, including the criterion used to determine whether a word is considered rare.

Thank you for this comment. We define the rare-word fraction as the share of a document's content tokens (whitespace-split, lowercased, punctuation stripped, and containing at least one letter) that are not among the 20,000 most frequent English words from the python wordfreq library list, averaged over the documents in each sample. We will add this definition, along with the definitions of the other metrics in Figure 1a, to the appendix in the new version of the paper pdf.

Figure 1b: This figure is difficult to interpret, and its description is insufficient. The authors should explain more clearly what is being compared and how the figure supports their argument.

Thank you for this comment. Figure 1b compares, for each of 100 uniformly random groups of 5,120 Repro samples, the group's individual proxy oracle (the mean of its members' individual oracle scores) against its group oracle. If a group's value were simply the average of its members' values, the two would be strongly correlated. Instead, they are only weakly correlated (Spearman ρ = 0.26 with individual scores from the DCLM-400M checkpoint).

Line 146: What evidence supports this statement? The authors should provide a clearer justification for why these properties are necessarily invisible to individual-level scoring methods, particularly given the variety of possible individual-level signals. Line 153: This is not the conclusion I drew from the previous section. Rather than conclusively establishing the claimed relationship, the preceding discussion appears to make several statements without sufficient supporting evidence. The authors should strengthen the argument and clarify how the previous section supports this conclusion.

Thank you for this comment. We would like to clarify (for Line 146) that this follows from the definition of an individual-level score: any score computed on a single sample in isolation cannot depend on which other samples are in the group. For example, two exact duplicates may each receive a high individual score, while the pair adds little beyond a single copy.

This is also consistent with prior work on group influence. Koh et al. [18] and Basu et al. [19] find that summing individual influence estimates systematically underestimates the actual effect of removing a group of training points. Our contribution here is to show that this gap matters for selecting synthetic data for LLM training: unlike these works, which approximate the effect of removing groups from small, mostly convex models, we measure group value directly with the group oracle and show that it changes downstream performance in both pre-training and RLVR post-training. We will update the statement in the main text to make this reasoning explicit, cite this work, and make the conclusion at )(Line 153) clearer in the new version of the paper pdf.

[18] Koh et al. On the Accuracy of Influence Functions for Measuring Group Effects. NeurIPS 2019.

[19] Basu et al. On Second-Order Group Influence Functions for Black-Box Predictions. ICML 2020.

Reviewer_K8viOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether synthetic training data should be selected based on individual sample scores or group-level signals. The authors argue that individual scores cannot capture interactions between samples, and that groups with similar individual scores may have very different training effects.

The authors evaluate this through synthetic pre-training and RLVR post-training experiments. They show that group-level oracle selection can outperform individual-level selection, especially on generative tasks. They also propose using gradient diversity to identify groups where group-level scoring is more useful, with the goal of reducing its computational cost.

Critical strengths
  1. The paper studies an important problem in synthetic data curation. It shows that scoring samples individually may fail to capture their interactions and the actual value of a group of samples. The distinction between individual proxy oracle and group oracle provides a useful perspective on this problem.

  2. The authors evaluate the problem in both synthetic pre-training and RLVR post-training settings, covering a range of language understanding, generation, reasoning, and coding tasks. The controlled experiments, where groups have similar individual proxy scores but different compositions, are particularly interesting. The results show that group-level selection can improve downstream performance, especially on generative tasks. The paper also compares several existing data selection methods, including GREATS and GMRel.

  3. The paper provides a theoretical analysis of the relationship between group-level influence and gradient diversity. It also proposes a gradient-diversity-based diagnostic to identify groups that may benefit from more expensive oracle evaluation. The experiments on gradient subsampling further show the potential to reduce computational costs while retaining much of the benefit of group-level scoring.

Critical weaknesses
  1. The theoretical justification does not fully match the actual implementation. Proposition 5.1 is derived using a standard gradient update and the variance of raw per-sample gradients. However, the actual group-oracle scoring uses one AdamW step, while the practical GradDiv metric is computed from JL-projected and L2-normalized gradients using pairwise cosine similarity. The paper does not explain how the theoretical bound applies to the AdamW update or how the practical GradDiv relates to the raw gradient variance in the proposition. This is important because GradDiv is used to decide which groups should receive expensive oracle evaluation. Without a clear connection, the theoretical justification for this budget allocation strategy remains incomplete.

  2. The experiments do not fully isolate the effect of increasing syntheticity. In Section 4, the total pool size is fixed, but increasing the rephrase factor from 1 to 50 also reduces the number of unique source documents from 716,800 to 14,336. Thus, rephrasing intensity, source coverage, and near-duplicate redundancy change simultaneously. The results show the behavior of group-level selection under this particular data-recycling setup, but do not establish which factor causes the observed trend. Moreover, at r = 50, the default GMRel and GREATS results are both below the baseline in Table 9; the improvement highlighted in Figure 5d requires increasing GMRel's relational weight. Together with the use of only one random seed, this makes the claim that group-level signals become increasingly important for synthetic data less convincing.

Questions for authors
  1. How does Proposition 5.1 apply to the actual AdamW-based group-oracle scoring? In particular, what is the relationship between the raw gradient variance in the theorem and the JL-projected, normalized cosine-based GradDiv used in practice?

  2. Can the authors disentangle the effects of rephrasing intensity, source coverage, and data redundancy in Section 4? Would the observed trend remain if the number of unique source documents were kept fixed? Also, how was the increased relational weight for GMRel at r = 50 chosen, and are the results consistent across multiple random seeds?

  3. Since the paper already compares GREATS and GMRel (from Group-MATES, Yu et al., 2025), what is the main contribution beyond evaluating existing group-aware selection methods on synthetic data? How does the proposed approach differ from related methods that consider interactions among samples, such as G2IS (Zhao et al., 2025), and SPICE (Chang et al., 2026)?

4 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you for this feedback. We first address the theoretical component. While our theoretical analysis was conducted for SGD, below we show that the same bound also applies to AdamW.

Recall the AdamW update:

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2,m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2,

m^t=mt1−β1t,v^t=vt1−β2t,\hat m_t = \frac{m_t}{1-\beta_1^{t}}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^{t}},

θt=θt−1−η(m^tv^t+ϵ+λθt−1).\theta_t = \theta_{t-1} - \eta\left(\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon} + \lambda \theta_{t-1}\right).

Here, θt\theta_t are the model parameters at step tt, gtg_t is the loss gradient, mtm_t and vtv_t are running estimates of the gradient's first and second moments (m^t\hat m_t, v^t\hat v_t are their bias-corrected versions), β1,β2\beta_1, \beta_2 are their decay rates, η\eta is the learning rate, λ\lambda is the weight decay, and ϵ\epsilon is a small constant for numerical stability.

Our oracle creates a fresh optimizer for each group, so m0=v0=0m_0 = v_0 = 0. After one step, bias correction gives

m^1=g,v^1=g2,\hat m_1 = g, \qquad \hat v_1 = g^2,

and the update is

u(g)=−ηg∣g∣+ϵ−ηλθ≈−ηsign(g)−ηλθ.u(g) = -\eta \frac{g}{|g|+\epsilon} - \eta\lambda\theta \approx -\eta \mathrm{sign}(g) - \eta\lambda\theta.

Bound. We apply the same Taylor/Jensen argument as in Proposition 5.1 to the sign updates:

uz=−ηsign(gz),uG=−ηsign(gG),uˉ=1∣G∣∑z∈Guz.u_z = -\eta \mathrm{sign}(g_z), \qquad u_G = -\eta \mathrm{sign}(g_G), \qquad \bar u = \frac{1}{|G|}\sum_{z\in G} u_z.

This gives

∣Iproxy−Iorcl∣≤2η∥∇Φ(uG)∥P+2βη2P,|\mathcal I_{\mathrm{proxy}} - \mathcal I_{\mathrm{orcl}}| \le 2\eta \lVert\nabla\Phi(u_G)\rVert \sqrt{P} + 2\beta\eta^2 P,

where

P=1∣G∣∑z∈Gdz,P = \frac{1}{|G|}\sum_{z\in G} d_z,

and dzd_z is the number of coordinates in which the sign of sample zz's gradient differs from that of the group gradient gGg_G. That is, PP is the average number of coordinates in which a sample's gradient sign disagrees with the group's. The bound is zero when all per-sample gradients agree in sign with the group, and it grows as their directions diverge. This mirrors the effect shown in the theoretical analysis of our paper: the more the samples pull the model in different directions, the more the group oracle can deviate from the individual proxy oracle.

We will include these details in the new version of the paper pdf.

How does the practical GradDiv metric (computed from JL-projected and L2-normalized gradients) compare to the raw gradient variance?

Thank you for this question. First, we compare the two versions of GradDiv empirically: the raw version and the practical version (computed from JL-projected, L2-normalized gradients). The linked figure (https://files.catbox.moe/eyjs2s.png) shows the correlation between the two versions on ~650 groups of size 512 from GooseReason

The two versions are positively correlated (Spearman ρ = 0.61), indicating that the practical version of GradDiv can serve as a strong proxy for raw GradDiv.

Second, we note that JL projection and L2 normalization, the two components of the practical GradDiv, are widely used in practice. For example, LESS [1] and TRAK [2] both use JL-projected and L2-normalized gradients to score and select training data. Together, these results show that our practical GradDiv is not arbitrary, but rather follows procedures commonly adopted in the LLM data influence and broader data curation communities.

[1] Xia et al. LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 2024.

[2] Park et al. TRAK: Attributing Model Behavior at Scale. ICML 2023.

AuthorsOpenReview ↗

Author Response (2/3)

Rephrasing intensity, source coverage, and near-duplicate redundancy change simultaneously. The results show the behavior of group-level selection under this particular data-recycling setup, but do not establish which factor causes the observed trend.

We would like to clarify that synthetic data recycling is a multi-faceted process in which rephrasing intensity, source coverage, and near-duplicate redundancy change simultaneously, as reflected in prior work [3, 4, 5]. The goal of our study is not to isolate the effect of each factor, but to analyze data selection in a realistic synthetic data recycling setting. Specifically, in this setting the data pool contains rephrased low-quality documents (as in RePro [6] and REWIRE [3]), and the source corpus shrinks, mirroring the approaching limit on organic web text [7]. Since this form of synthetic recycling is increasingly adopted in frontier LLMs [5], we believe it is important to study it holistically.

[3] Nguyen et al. Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models. COLM 2025.

[4] Maini et al. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. ACL 2024.

[5] Kimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534, 2025.

[6] Yu and Xiong. RePro: Training Language Models to Faithfully Recycle the Web for Pretraining. arXiv:2510.10681, 2025.

[7] Villalobos et al. Position: Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data. ICML 2024.

At r = 50, the default GMRel and GREATS results are both below the baseline in Table 9; the improvement highlighted in Figure 5d requires increasing GMRel's relational weight.

Thank you for this observation. We would like to clarify that r = 50 is an extreme case in which resisting the collapse caused by heavily synthetic training is much harder than at r = 1 or r = 10. We show that at r = 50, only increasing GMRel's relational weight resists this collapse, highlighting the necessity of capturing relational information among data points.

How was the increased relational weight for GMRel at r = 50 chosen, and are the results consistent across multiple random seeds?

Thank you for this question. By default, GMRel learns its relational temperature during training (τ ≈ 1.01 in our setting), and we use this learned value for all main results. At r = 50, we additionally set τ = 0.30 as a single probe value, which roughly triples the weight on the relational term, to test whether a stronger emphasis on inter-sample relationships helps in this extreme, highly redundant setting.

Regarding random seeds, we have run two additional training seeds for GMRel (τ = 1.01 and τ = 0.30) and random selection at r = 50, and found that the results hold:

MethodPaper (seed 1)Seed 2Seed 3Mean
GMRel τ 1.0127.6727.7127.4627.61
GMRel τ 0.3028.9727.8929.1828.68
Random28.1027.7529.3528.40

Since the paper already compares GREATS and GMRel (from Group-MATES, Yu et al., 2025), what is the main contribution beyond evaluating existing group-aware selection methods on synthetic data?

Thank you for this question. We would like to clarify that our paper is not primarily an evaluation of existing group-aware selection methods. Rather, it asks when and why individual-level scores fail for synthetic data. Our contributions are as follows:

  1. A controlled demonstration of where individual-level scoring fails. Using data influence oracles, we construct groups with nearly identical individual proxy scores but different compositions. Because individual-level scorers aggregate per-sample scores (e.g., by averaging), they cannot distinguish these groups. However, their group oracle scores vary widely, and selecting by group oracle yields better downstream performance in both synthetic pre-training and RLVR post-training.
  2. Evidence that this gap matters for selection methods used in practice. Building on this insight, we show that among widely used data selection methods, only those that use group-level signals (GREATS and GMRel) improve over the baseline on synthetic data, and that group-level signals matter more as synthetic data recycling intensifies. We use GREATS and GMRel to test this hypothesis on methods practitioners already use; they are not the contribution itself.
  3. A diagnostic for when group-level signals are worth their cost. Since group-level scoring is expensive, we show that the gap between individual and group oracles can be characterized by within-group gradient diversity, and propose GradDiv as a method-agnostic diagnostic for deciding which groups should receive expensive group-level evaluation.
AuthorsOpenReview ↗

Author Response (3/3)

How does the proposed approach differ from related methods that consider interactions among samples, such as G2IS (Zhao et al., 2025), and SPICE (Chang et al., 2026)?

We thank the reviewer for pointing out these related works. We would like to clarify that G2IS [8] and SPICE [9] are both selection methods. In Section 5, our goal is not to propose another selector, but to present a diagnostic that identifies which groups are likely to have a large individual-to-group gap and therefore require group-level evaluation. In particular, GradDiv is complementary to these methods and could be used to decide where to apply them. We will add both works to our related work section and discuss this connection in the new version of the paper pdf.

[8] Zhao et al. Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data Selection. ACL 2025.

[9] Chang et al. SPICE: Submodular Penalized Information–Conflict Selection for Efficient Large Language Model Training. ICLR 2026.

AuthorsOpenReview ↗

Paper Revision

Dear Reviewer K8vi,

In addition to the responses above, we have incorporated your feedback and the new experiments in the new paper pdf. Your feedback and engagement is much appreciated!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.