- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes FROST, an online synthetic data filtering framework designed to improve model performance on real-world tasks by selecting useful synthetic training examples. The method estimates sample utility through gradient alignment with an exponential moving average of real-training gradients. It then employs a two-stage filtering strategy: batch-level utility calibration against recent training history determines whether filtering is necessary, while interquartile-range (IQR) thresholds select samples from batches with unusually low or high utility. The framework does not require an external verifier or a held-out validation set. Experiments on CIFAR-100 image classification, Text-to-SQL fine-tuning, and industrial ads re-ranking demonstrate that FROST can improve downstream performance while discarding a portion of the synthetic training data.
Critical strengths
-
The paper addresses an important challenge in synthetic data training, where noisy or distributionally mismatched samples may negatively affect downstream performance. By using real-training gradients to guide online data selection, FROST provides a practical approach to identifying useful synthetic supervision without requiring an external verifier or a separate validation set.
-
FROST combines gradient-based utility estimation with a conditional two-stage filtering strategy. The batch-level calibration determines when filtering is necessary, while sample-level IQR filtering selectively removes potentially harmful examples. The overall framework is straightforward to understand and implement, with a first-order approximation providing theoretical motivation for the utility estimation.
-
The authors evaluate FROST across image classification, LLM fine-tuning, and large-scale industrial ads re-ranking, demonstrating its applicability to different architectures and tasks. The method consistently improves downstream performance over training with all synthetic data while filtering a substantial portion of synthetic samples. The industrial results further highlight its potential practical value
Critical weaknesses
-
Although Appendix B.3 provides a theoretical analysis of the utility computation overhead and reports training throughput reductions of approximately 6.8% and 5.7% for image classification and industrial ads re-ranking, respectively, the paper does not report end-to-end wall-clock training time. More importantly, no runtime measurements are provided for the LLM fine-tuning experiment, where gradient-based utility estimation may introduce non-negligible overhead. Since FROST requires scoring synthetic candidates before filtering, reducing the number of retained samples does not necessarily translate into proportional computational savings. Without direct runtime comparisons against baselines such as GREATS, the practical computational efficiency of FROST remains unclear.
-
Although the authors conduct ablation studies to evaluate the two-stage filtering strategy, the compared variants retain substantially different amounts of synthetic data. In Figure 3, Sample IQR Only removes 32.2% of synthetic samples, whereas full FROST removes only 20.4%. A similar discrepancy appears in Figure 5, where the corresponding removal ratios are 54% and 36%. Since full FROST consistently retains more training data, the observed performance improvements cannot be attributed solely to the batch-level calibration mechanism. The current ablations therefore do not sufficiently isolate the contribution of this key component.
-
A central motivation for the bounded utility interval is the observation that extremely high-utility batches may lead to less stable training. However, the supporting experiment in Figure 2 is limited to 30 continuation steps from a single checkpoint after 20 training epochs. Although the moderate-utility group achieves higher accuracy at the final observed step, the high-utility group reaches a higher early peak. This limited experiment does not establish whether moderate-utility batches consistently outperform high-utility batches across different training stages or longer training horizons. Consequently, the empirical justification for routing extremely high-utility batches to additional filtering remains insufficient.
Questions for authors
-
Could the authors provide end-to-end wall-clock training time and training throughput comparisons between FROST, training with all synthetic data, and GREATS, particularly for the LLM fine-tuning experiment? This would help clarify whether the proposed filtering mechanism introduces additional computational overhead or offers practical training-time benefits despite requiring utility estimation for all candidate samples.
-
Could the authors adjust the filtering thresholds of Sample IQR Only to match the synthetic-data retention ratio of full FROST and report the corresponding performance? Such a controlled comparison would help determine whether batch-level calibration provides additional benefits beyond simply retaining more synthetic training examples.
-
Could the authors extend the analysis in Figure 2 to multiple training checkpoints and longer continuation horizons? Additionally, could the authors evaluate the effect of removing or relaxing the upper utility threshold while keeping the synthetic-data retention ratio comparable? These experiments would help establish whether filtering extremely high-utility batches is necessary for improving training stability and downstream performance.
-
Could the authors provide an experiment comparing the sample utility scores computed using the prediction head with those computed using full-model gradients? Reporting their ranking correlation would help assess the reliability of the partial-gradient approximation used in FROST.
6 comments
Response to Reviewer CPL1 (Part 2/4)
2. Matched-drop-ratio ablations
Could the authors adjust the filtering thresholds of Sample IQR Only to match the synthetic-data retention ratio of full FROST and report the corresponding performance? Such a controlled comparison would help determine whether batch-level calibration provides additional benefits beyond simply retaining more synthetic training examples.
Thank you for pointing this out. We adjusted the filtering thresholds to approximately match Full FROST's synthetic-data drop rate. Results are mean ± standard deviation over 3 seeds for CIFAR-100 and 5 seeds for Qwen2.5-Coder-0.5B text-to-SQL.
CIFAR-100.
| Generator | Method | Synthetic-data drop rate (%) | Accuracy (%) |
|---|---|---|---|
| FLUX | Batch only | 25.84 ± 0.80 | 80.17 ± 0.17 |
| FLUX | Sample only | 25.02 ± 0.61 | 80.55 ± 0.08 |
| FLUX | Full FROST | 26.40 ± 0.57 | 80.86 ± 0.13 |
| SANA | Batch only | 26.02 ± 0.09 | 79.26 ± 0.21 |
| SANA | Sample only | 25.58 ± 1.67 | 79.82 ± 0.19 |
| SANA | Full FROST | 26.96 ± 0.83 | 80.29 ± 0.29 |
| SD1.4 | Batch only | 25.07 ± 0.83 | 79.93 ± 0.12 |
| SD1.4 | Sample only | 25.32 ± 0.04 | 80.64 ± 0.21 |
| SD1.4 | Full FROST | 25.32 ± 0.64 | 80.80 ± 0.18 |
Text-to-SQL.
| Method | Synthetic-data drop rate (%) | EX (%) |
|---|---|---|
| Batch only | 21.71 ± 1.03 | 64.42 ± 0.75 |
| Sample only | 20.54 ± 0.04 | 64.40 ± 0.47 |
| Full FROST | 20.39 ± 0.31 | 65.92 ± 0.59 |
Full FROST achieves higher mean performance than both single-stage variants in all four settings. In image classification, it achieves higher accuracy than Sample only and Batch only across all 3 generators at similar synthetic-data retention rates. In text-to-SQL, Full FROST reaches 65.92% EX versus 64.40% for Sample only at nearly identical drop ratios. These results support the benefit of the two-stage filtering design.
Industrial ads re-ranking. We also adjusted the drop rates of the single-stage variants for our commercial ads re-ranking model. Given the substantial training resources required, we report the available results below. Relative NE change is measured against real-only training; lower is better.
| Method | Synthetic-data drop rate (%) | Relative NE change (%) ↓ |
|---|---|---|
| All Synthetic | 0 | +0.211 |
| Batch only | 35 | +0.097 |
| Sample only | 22 | +0.128 |
| Sample only | 45 | +0.070 |
| Sample only | 51 | +0.057 |
| Full FROST | 36 | -0.096 |
At similar drop rates (35% versus 36%), Batch only yields a +0.097% relative NE change, while Full FROST achieves -0.096%, improving over real-only training. Sample only also underperforms Full FROST at all tested drop rates (22%, 45%, and 51%).
Response to Reviewer CPL1 (Part 3/4)
3. Multiple checkpoints and longer continuation horizons
Could the authors extend the analysis in Figure 2 to multiple training checkpoints and longer continuation horizons? Additionally, could the authors evaluate the effect of removing or relaxing the upper utility threshold while keeping the synthetic-data retention ratio comparable? These experiments would help establish whether filtering extremely high-utility batches is necessary for improving training stability and downstream performance.
Thank you for raising this question about the design of the upper utility bound. Prior work shows that excessively large stochastic gradients can push iterates away from the optimum and cause oscillations [1]. Merad and Gaïffas [2] use rolling gradient-norm quantiles to clip large stochastic gradients and improve stability with corrupted samples. Inspired by this principle, FROST uses an upper utility bound to route unusually high-utility batches to sample-level filtering. To further confirm our design choice, we extended the mid-band analysis to multiple checkpoints and 100 continuation steps, and tested removing the upper bound at comparable drop rates.
Multiple-checkpoint continuation analysis. We use image-model checkpoints at epochs 20, 90, and 160, and Qwen2.5-Coder-0.5B checkpoints at steps 10, 100, 300, and 500. From each checkpoint, we continue training for 100 steps using each of the Low, Mid, and High synthetic-batch utility bands, with 5 repeats per band. To reduce sensitivity to endpoint fluctuations, we first average accuracy or EX over the first 30, 50, or 100 steps within each repeat, then report the mean ± standard deviation of these window averages across the 5 repeats.
Image classification: accuracy (%).
| Checkpoint (epoch) | Averaging window | Low-band training | Mid-band training | High-band training |
|---|---|---|---|---|
| 20 | First 30 steps | 64.08 ± 0.98 | 65.65 ± 0.64 | 64.93 ± 0.97 |
| 20 | First 50 steps | 64.45 ± 0.78 | 64.95 ± 0.53 | 64.87 ± 0.57 |
| 20 | First 100 steps | 64.64 ± 0.46 | 64.69 ± 0.52 | 64.87 ± 0.50 |
| 90 | First 30 steps | 58.75 ± 0.65 | 59.80 ± 0.58 | 59.62 ± 0.35 |
| 90 | First 50 steps | 58.95 ± 1.07 | 59.54 ± 0.25 | 59.11 ± 0.53 |
| 90 | First 100 steps | 58.79 ± 0.59 | 59.24 ± 0.48 | 58.60 ± 0.56 |
| 160 | First 30 steps | 73.44 ± 0.30 | 74.09 ± 0.41 | 73.76 ± 0.52 |
| 160 | First 50 steps | 73.70 ± 0.11 | 74.24 ± 0.16 | 74.03 ± 0.39 |
| 160 | First 100 steps | 73.83 ± 0.28 | 74.09 ± 0.11 | 73.95 ± 0.19 |
Text-to-SQL: EX (%).
| Checkpoint (step) | Averaging window | Low-band training | Mid-band training | High-band training |
|---|---|---|---|---|
| 10 | First 30 steps | 48.83 ± 0.68 | 52.20 ± 0.75 | 50.77 ± 0.64 |
| 10 | First 50 steps | 48.75 ± 0.54 | 51.05 ± 0.61 | 50.47 ± 0.85 |
| 10 | First 100 steps | 49.59 ± 0.31 | 51.22 ± 0.43 | 50.32 ± 0.57 |
| 100 | First 30 steps | 51.84 ± 0.42 | 52.33 ± 1.31 | 52.07 ± 0.31 |
| 100 | First 50 steps | 52.13 ± 0.30 | 52.51 ± 0.33 | 52.94 ± 0.41 |
| 100 | First 100 steps | 53.25 ± 0.36 | 53.90 ± 0.54 | 54.26 ± 0.70 |
| 300 | First 30 steps | 60.86 ± 0.34 | 61.18 ± 0.17 | 60.69 ± 0.45 |
| 300 | First 50 steps | 61.26 ± 0.32 | 61.27 ± 0.06 | 61.49 ± 0.56 |
| 300 | First 100 steps | 62.65 ± 0.32 | 62.56 ± 0.23 | 62.41 ± 0.27 |
| 500 | First 30 steps | 64.66 ± 0.17 | 65.16 ± 0.08 | 63.50 ± 0.17 |
| 500 | First 50 steps | 64.78 ± 0.14 | 65.14 ± 0.06 | 63.51 ± 0.01 |
| 500 | First 100 steps | 64.94 ± 0.16 | 65.08 ± 0.09 | 63.66 ± 0.07 |
Mid has the highest mean in 8 of 9 image comparisons and exceeds High in 9 of 12 text-to-SQL comparisons. Overall, Mid-utility batches generally yield better average performance across the tested checkpoints and horizons, showing that higher current utility does not consistently translate into better subsequent performance.
Reference
[1] Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping. NeurIPS, 2020.
[2] Ibrahim Merad and Stéphane Gaïffas. Robust Stochastic Optimization via Gradient Quantile Clipping. Transactions on Machine Learning Research, 2024.
Response to Reviewer CPL1 (Part 1/4)
We sincerely thank you for the positive feedback and for recognizing the practical value of FROST's real-data-anchored utility estimation, conditional two-stage filtering, and evaluation across three domains. We appreciate your suggestions for strengthening the experimental evidence. We take this opportunity to address your comments in detail below.
1. End-to-end training time and computational overhead
Here we report end-to-end training times for 0.5B and 7B text-to-SQL models. Regular training uses all real and synthetic data without filtering. Within each model setting, all methods use the same dataset and pre-filter batch size, with matched synthetic-data drop rates for FROST and GREATS. Parentheses show the time increase relative to regular training for the same model and GPU.
FROST adds 2.10–2.30% to the end-to-end training time, compared with 5.14–6.67% for GREATS. FROST is faster than GREATS in both configurations.
Selection overhead. Compared with FROST, GREATS [1] additionally computes pairwise candidate-gradient interactions, forming an m×m matrix, where m is the number of synthetic candidates in a batch before filtering. For p-dimensional scoring gradients, direct dense construction costs O(m2p) once gradients are available. FROST aligns candidates with a single real-gradient reference in O(mp), then calibrates the batch mean against recent history. The remaining operations use scalar scores: averaging and threshold comparisons take O(m), while sorting-based IQR computation takes O(mlogm) only for out-of-band batches.
Why both methods take longer than regular training. In our benchmark implementations of FROST and GREATS, filtering zeros the gradient contributions of excluded samples without skipping their dense forward/backward computation. Selection therefore adds computation without converting the drop rate into proportional training-time savings. Selected-sample execution could reduce this cost, but these optimizations are outside our current scope. Even with this dense implementation, FROST adds only around 2% to end-to-end training time in these LLM settings.
Reference
[1] Jiachen T. Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration. NeurIPS, 2024.