- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when a student trained entirely on synthetic data can outperform its teacher in KL divergence to the true distribution. In a Gaussian setting with known zero mean, an MLE teacher, and a scaled covariance estimator for the student, it derives an exact expected KL gap and characterizes a synthetic-sample threshold that scales approximately as . It also presents high-probability guarantees and analyzes successive teacher–student chains. The results are evaluated through Gaussian Monte Carlo simulations, LDA on ResNet-50 features of CIFAR-10, and closed-form linear flow-matching experiments.
The paper is clearly written and well organized. Its exact expected-risk analysis provides a useful quantitative characterization of how estimator scaling and synthetic-sample budgets affect student performance, and the toy experiments illustrate the predicted transition.
However, the interpretation of the student's advantage is limited by the uncorrected MLE teacher baseline. Directly applying the optimal scalar correction to the teacher yields lower expected KL risk than the optimally scaled student for every finite synthetic-sample budget. The reported transition therefore balances estimator-correction benefits against additional sampling costs, rather than demonstrating an advantage over an equally optimized teacher. Furthermore, the PAC proof contains a false variance upper bound and inconsistent calculations, while Theorem 3 includes an incorrect fixed-budget limit claim. The experiments also provide limited evidence beyond the restricted Gaussian and linear settings.
Based on these concerns about the comparison baseline, theoretical correctness, and scope of validation, I recommend Weak Rejection (2/4).
Critical strengths
-
Clear presentation. The paper is well organized and clearly presents the problem formulation, assumptions, and main results. The progression from a single teacher–student pair to successive training rounds is easy to follow.
-
Theoretical contribution. In the Gaussian setting, the paper derives an exact expected KL gap, identifies the optimal scalar covariance correction, and characterizes the synthetic-sample threshold for outperforming an MLE teacher. It also analyzes the optimal expected performance of successive teacher–student chains.
-
Illustrative experiments. The paper provides experiments on toy Gaussian examples, including linear flow matching, to validate the predicted phase transition and illustrate how estimator scaling and synthetic-sample budgets affect student performance.
Critical weaknesses
1. The source of the student's improvement needs clearer interpretation. The teacher uses ordinary MLE, whereas the student is allowed an optimized covariance scaling. In the known-zero-mean Gaussian setting, define and . The optimal student scaling is . These two factors have distinct roles: accounts for the teacher's estimation error under KL loss, while accounts for the additional estimation error introduced by finite synthetic sampling.
The teacher can obtain this correction directly, without generating synthetic data. Consider rescaling its MLE covariance as , with . Since , its expected KL risk satisfies
Differentiating with respect to gives , which is negative for and positive for . Therefore, is the unique optimal deterministic scalar correction.
Having established the optimal scaling for the teacher, we can revisit the optimal student estimator. Its scaling combines two corrections: accounts for the original teacher's estimation error under KL loss, while accounts for the additional estimation error introduced by finite synthetic sampling.
This decomposition explains why the phase transition occurs when comparing the student with an ordinary MLE teacher. The student's expected KL gap can be written as
The first term is the additional synthetic-sampling cost, which decreases to zero as increases. The second term is the benefit of optimally correcting the MLE teacher, which is fixed for a given and . With too few synthetic samples, the sampling cost outweighs the correction benefit, and the student performs worse. Once is sufficiently large, the cost falls below this benefit, and the student outperforms the uncorrected teacher. The transition therefore reflects a balance between estimator-correction benefit and additional sampling cost.
2. The student advantage disappears after correcting the teacher. The paper compares the optimally scaled student against an uncorrected MLE teacher. However, a more appropriate baseline would be a teacher with the same optimal covariance correction, given by .
The paper's original optimally scaled student is obtained by drawing synthetic samples , computing their empirical covariance , and applying the optimal scaling , yielding
Importantly, this student is equivalent in distribution to one trained on synthetic samples from the corrected teacher. To see this, consider , which follows . Its empirical covariance is therefore . Since the teacher-level correction has already been applied, the student only needs the additional correction , giving
Thus, the paper's original optimal student can be equivalently viewed as a student trained on synthetic data from the corrected teacher, allowing us to compare the two under the same correction framework.
Under the analyzed Gaussian setting, the expected KL risk difference between this student and the corrected teacher satisfies
Therefore, even the optimally scaled student has a higher expected KL risk than the corrected teacher for every finite synthetic-sample budget. As increases, the additional sampling error decreases, but the student cannot outperform the corrected teacher:
This suggests that, within the analyzed Gaussian setting and estimator class, the reported student advantage comes from the covariance correction rather than from synthetic-data training itself. Once the same correction is applied to the teacher, the student can at best approach its performance.
3. Errors in the proof of the high-probability guarantees. Appendix F contains a variance bound that is false under its stated conditions. For example, taking , , and , a direct calculation using independent chi-square variables gives
However, substituting the same parameters into Eqs. (66)–(68) gives and , so the purported upper bound in Eq. (65) is only
The actual variance therefore exceeds the claimed upper bound.
4. The progressive-improvement claims require correction and qualification. Theorem 3 states that as . This is false for fixed sample budgets. For example, with , , , and , the prescribed scaling is , independent of , and
Thus, the actual gap does not converge to zero. Only the displayed guaranteed upper bound necessarily tends to zero. In addition, at the exact sample threshold, the last stage can merely tie its predecessor, so strict improvement requires a strict budget inequality. Finally, the exponential budget scaling follows from the proposed construction with fixed and it is not a general necessity result for all progressively improving chains. Indeed, approaches as . These distinctions should be reflected in the theorem statement and the discussion.
5. Limited evidence for applicability beyond the Gaussian setting. The principal guarantees concern Gaussian distributions and a specified class of scaled covariance estimators. Most experiments validate this setting through Gaussian simulations or closed-form linear flow-matching models. The CIFAR-10 experiment provides a useful departure from synthetic Gaussian data, but it fits LDA to fixed ResNet-50 features rather than training a general discriminative or generative model. Its reported improvement is primarily in test log-likelihood, with little change in classification accuracy at sufficiently large synthetic budgets. Consequently, the experiments illustrate the theoretical mechanism but provide limited evidence that the proposed correction or sample-budget predictions remain useful for more general synthetic-data training. The practical claims should be narrowed accordingly or supported by experiments beyond these restricted models.
Questions for authors
-
Can the authors compare the proposed student with an optimally corrected teacher using covariance ?
-
In the corrected teacher setting, the optimal student has strictly higher expected KL risk than this baseline for every finite synthetic-sample budget. How does this affect the interpretation of synthetic-data training as a mechanism for surpassing the teacher?
-
Can the authors address the counterexample to the variance upper bound in Eq. (65) and correct the two sign errors in Eq. (177), which contradict the variance decomposition in Eqs. (163), (174), and (176)? The substitution of in Appendix F also violates the stated condition . Which of the explicit PAC guarantees in Theorem 2 can be rigorously recovered after correcting these steps? Please also clarify the fixed-budget claim in Theorem 3.
-
What evidence supports extending the proposed correction or sample-budget predictions beyond Gaussian covariance estimation and closed-form linear models? Can the authors provide an experiment in a broader model class, including a directly corrected teacher baseline, or clarify the intended scope of the practical conclusions?
Minor comments
-
On page 4, the statement that is convex on is incorrect, since for . Please correct this statement. Its unique minimum at and the relevant branch monotonicity remain valid.
-
At line 78, Fu et al. (2024) is incorrectly categorized as reinforcement learning. The cited paper, “Towards Theoretical Understandings of Self-Consuming Generative Models,” studies recursive training of generative models, including diffusion models and kernel density estimation. Please correct the categorization and clarify its relationship to the present generative analysis.
3 comments
Q3 and W4. Clarification of Theorem 3
We thank the reviewer again for the careful reading, and we note that this was a typo in the
result state in the paper, which should read
(which is what is stated and proven in more detail in the appendix version of the
theorem on Eq. (108)) rather than the expectation converging to .
This change has no impact on the interpretation of the result, as it shows that successive
application of estimators in the chain improve the performance of the estimator
(and indeed, it is actually better if this expectation is strictly negative away from
as it provides a larger improvement in performance with the successive estimators).
After considering the suggestions from R-Y5Di (Q3 and Q4) along with yours:
(i) We have included the exact expression of the mean in the Theorem 3 statement (Eq (9)).
(ii) We derived tighter conditions for sample complexities in the multi-stage students, that is, linear growth w.r.t the length is also sufficient. Our earlier result was correct, but it was not tight.
Q4 and W5. Evidence beyond the Gaussian setting.
Our conclusions extend beyond Gaussian setting in practice. We have conducted the following empirical study to support our stance:
(i) Synthetic data. We now have included evidence of phase-transition for seven non-Gaussian distributions, in dimensions , and . For six of them the median relative error of the boundary w.r.t. the Gaussian prediction is between and , and the boundary grows as (log-log slope --, against for the Gaussian theory). Student- is the exception: its boundary is about lower than the Gaussian prediction, so the student wins with fewer synthetic samples and the Gaussian condition remains sufficient. Please see Appendix C, Table 1 and Figures 7 and 8 for the detailed experimental setup and results.
(ii) Flow matching on real data. We now have Flow matching results on CIFAR 10 dataset with ResNet-50 features that lead us to conclude the same phenomenon. See Figure 3b and updated the experimental description.
As for the corrected teacher baseline we have now included this in Figure 3(a), and Figure (11).
Minor comments All of them are fixed, and none of the conclusions are affected by this.
Conclusion.
We thank the reviewer for the critical review, which made the paper technically stronger and wider in scope. We hope that we have addressed all the concerns of the reviewer, and we are happy to follow up on any remaining point.
Thank you for the detailed response and the substantial revisions. After examining the revised manuscript, I have a few remaining concerns about the interpretation and practical implications of the central result.
The paper investigates when a student trained entirely on synthetic data can outperform its teacher. However, my main concern is the source of the student's improvement. The optimal student scaling decomposes into two factors: one corrects the teacher's covariance estimation error under the expected forward KL criterion, while the other compensates for the additional error introduced by finite synthetic sampling. When the teacher's training sample size is known, the first correction can be applied directly to the fitted teacher. In the zero-mean Gaussian setting, this requires only rescaling the teacher's outputs, without retraining or fitting a separate student. As Proposition 4 now establishes, the optimally scaled student has strictly higher expected KL risk than the corrected teacher for every finite synthetic-sample budget. Thus, the phase transition characterizes when the benefit of correcting an unscaled MLE teacher outweighs the additional cost of synthetic sampling, rather than demonstrating an intrinsic advantage of synthetic-data training over an equally corrected teacher.
This also raises a practical question about how the proposed correction would be selected. The theoretically optimal student scaling depends explicitly on . If this quantity is available, the corresponding teacher-level correction can be applied directly in the analyzed setting. If it is unavailable to the student, the proposed optimal scaling cannot be computed as specified. Generating more samples from a single fitted teacher can improve estimation of the teacher's fitted distribution, but does not by itself identify the size of its original training dataset. The revised analysis therefore leaves unclear how a student with only sampling access to the teacher would select the correction, or whether the predicted advantage would persist under a practically estimated scaling factor.
The CIFAR-10 experiments remain limited in scope. The LDA results show improvements in log-likelihood but negligible changes in classification accuracy. The new flow-matching experiment operates on 32-dimensional ResNet features using a closed-form linear flow, rather than generating images. It therefore provides evidence for the predicted phase transition in feature space, but does not demonstrate improvements in image generation or sample quality. These experiments offer limited evidence for the practical applicability of the proposed approach to more general machine learning models.
Overall, while the revision addresses several of my earlier technical concerns, my main concern regarding the significance of the central result remains. I understand that theoretical analyses often rely on idealized settings and simplifying assumptions to obtain tractable results. Nevertheless, even within the analyzed setting, the student's improvement primarily arises from correcting the unscaled MLE teacher, a correction that can be applied directly without synthetic-data training. Moreover, the dependence on the teacher's training sample size and the limited empirical validation leave the practical relevance of the proposed approach unclear. For these reasons, I maintain my Weak Reject recommendation.
We thank the reviewer for their time and the rigorous review, including the worked derivations. We are encouraged that the reviewer already sees the analysis as a useful characterization, in the reviewer's words:
Below are the detailed comments regarding your comments with W as your indicated weakness, and Q as your indicated question.
Q1 and W1. Comparison with corrected teacher. The standard pipeline to train generative models is to perform MLE, and all the pre-trained models are trained using these principles (see Goodfellow et al. (2016, Sec. 5.5) and Ho et al. (2020, Eq. 3)). Therefore, we consider the uncorrected (MLE) teacher to reflect the pre-trained models used in the practice and characterize the phase transitions for this case.
We agree that if the teacher were corrected (covariance with a0Σ0), then no student estimate that is purely a function of corrected teacher samples is better than the corrected teacher itself, as demonstrated by your arguments (we spelled this out in a proposition in appendix K) along with a numerical simulation (see Figure 11). Please see our response to Reviewer oCjG Q1 and W1 for further detailed explanation.
Q2 and W2. Student-teacher setting when teacher is corrected.
When the teacher is corrected no student in (β,β) class can surpass its teacher, which is a known result from classical paradigms. We argue that corrected teacher case is not practical, specifically,
Therefore, it is more practical to study student-teacher setting with uncorrected teacher. Please see our response to Reviewer oCjG Q1 and W1 for detailed explanation.
Q3 and W3. Proof of Theorem 2
We thank the reviewer for the careful reading. Our proof had a wrong exponent of 2 in the second term of Eq (64) (submitted Eq (65)), which results in the counterexample provided by the reviewer.
Upon similar comments by other reviewers, we have corrected this in the revised version, which we note has small impacts on the final results. Namely, we note that:
(i) Polynomial decay in the dimension d remains intact with some changes to the constants; for the student-wins case the rate is now O(d−2+2ε) instead of O(d−2−2ε).
(ii) The comparison of Monte Carlo with PAC bounds was performed on the exact closed form expressions of mean and variance therefore the conclusions do not change.
(iii) We no longer substitute δ. Instead, we set κ directly in each regime so that κ≤1/4 (see lines 1466–1508).
Please see Table 2 for numerical comparison of our bounds with the theory.