acceptodds
Under review as a conference paper at ICLR 2027

To Bias or Not to Bias: A Phase Transition in Learning from Synthetic Data

Abstract

Real-world dataset curation is expensive. Pre-trained machine learning models provide a powerful means of generating synthetic data to augment limited real data for training downstream models. However, while synthetic-data augmentation has demonstrated empirical success, there is little theoretical understanding of when synthetic data actually helps. Here, we tackle fundamental questions: when can a model trained on synthetic data outperform its teacher, how much synthetic data is required, and how should the student be trained? We study these questions through a student-teacher framework in which both the teacher learns the distribution from the real data, and the student learns from the synthetic data generated by the teacher. When the hypothesis class for both teacher and student is the family of Gaussian distributions, we show if both teacher and student are maximum likelihood estimators (MLE), then the student can never outperform the teacher in terms of Kullback-Leibler (KL) divergence to the true distribution. Crucially, allowing the student to use an appropriately biased estimator changes this picture. We characterize a phase transition in the number of real and synthetic samples: the boundary nearly scales *quadratically* with the number of real samples; below which no student that is scalar multiple of MLE can outperform the teacher, and above the boundary a suitably biased student can achieve strictly lower KL divergence to the true distribution and consequently improved generalization. This phase transition provides exact practical guidance on how to train student models on synthetic data. A natural question is whether we can repeatedly use the student model as a teacher to train a new student model, and so on, to further boost the performance of the student model. We show that achieving an improvement at every stage requires a sample complexity that grows exponentially with the length of the chain. Moreover, we show that a two-stage mechanism can achieve the same asymptotic performance as arbitrarily long chains, revealing a fundamental *No Free Bootstrapping* phenomenon. Finally, we demonstrate the applicability of our theoretical framework in both discriminative and generative models, including linear discriminant analysis and linear flow matching.

Primary area:
generative models
Keywords:
Weak-to-Strong Generalization, Student-Teacher, Self-distillation, Statistical Learning Theory, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ujcSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when a student trained entirely on synthetic data can outperform its teacher in KL divergence to the true distribution. In a Gaussian setting with known zero mean, an MLE teacher, and a scaled covariance estimator for the student, it derives an exact expected KL gap and characterizes a synthetic-sample threshold that scales approximately as N02/dN_0^2/d. It also presents high-probability guarantees and analyzes successive teacher–student chains. The results are evaluated through Gaussian Monte Carlo simulations, LDA on ResNet-50 features of CIFAR-10, and closed-form linear flow-matching experiments.

The paper is clearly written and well organized. Its exact expected-risk analysis provides a useful quantitative characterization of how estimator scaling and synthetic-sample budgets affect student performance, and the toy experiments illustrate the predicted transition.

However, the interpretation of the student's advantage is limited by the uncorrected MLE teacher baseline. Directly applying the optimal scalar correction to the teacher yields lower expected KL risk than the optimally scaled student for every finite synthetic-sample budget. The reported transition therefore balances estimator-correction benefits against additional sampling costs, rather than demonstrating an advantage over an equally optimized teacher. Furthermore, the PAC proof contains a false variance upper bound and inconsistent calculations, while Theorem 3 includes an incorrect fixed-budget limit claim. The experiments also provide limited evidence beyond the restricted Gaussian and linear settings.

Based on these concerns about the comparison baseline, theoretical correctness, and scope of validation, I recommend Weak Rejection (2/4).

Critical strengths
  1. Clear presentation. The paper is well organized and clearly presents the problem formulation, assumptions, and main results. The progression from a single teacher–student pair to successive training rounds is easy to follow.

  2. Theoretical contribution. In the Gaussian setting, the paper derives an exact expected KL gap, identifies the optimal scalar covariance correction, and characterizes the synthetic-sample threshold for outperforming an MLE teacher. It also analyzes the optimal expected performance of successive teacher–student chains.

  3. Illustrative experiments. The paper provides experiments on toy Gaussian examples, including linear flow matching, to validate the predicted phase transition and illustrate how estimator scaling and synthetic-sample budgets affect student performance.

Critical weaknesses

1. The source of the student's improvement needs clearer interpretation. The teacher uses ordinary MLE, whereas the student is allowed an optimized covariance scaling. In the known-zero-mean Gaussian setting, define a0=N0/(N0−d−1)a_0=N_0/(N_0-d-1) and a1=N1/(N1−d−1)a_1=N_1/(N_1-d-1). The optimal student scaling is β∗=a0a1\beta^*=a_0a_1. These two factors have distinct roles: a0a_0 accounts for the teacher's estimation error under KL loss, while a1a_1 accounts for the additional estimation error introduced by finite synthetic sampling.

The teacher can obtain this correction a0a_0 directly, without generating synthetic data. Consider rescaling its MLE covariance as cΣ^0c\widehat\Sigma_0, with c>0c>0. Since E[Σ^0−1]=a0(Σ∗)−1\mathbb E[\widehat\Sigma_0^{-1}]=a_0(\Sigma^*)^{-1}, its expected KL risk satisfies

R(cΣ^0)−R(Σ^0)=d2[log⁡c+a0c−a0].R(c\widehat\Sigma_0)-R(\widehat\Sigma_0)=\frac d2\left[\log c+\frac{a_0}{c}-a_0\right].

Differentiating with respect to cc gives d2(c−a0)/c2\frac d2(c-a_0)/c^2, which is negative for c<a0c<a_0 and positive for c>a0c>a_0. Therefore, c=a0c=a_0 is the unique optimal deterministic scalar correction.

Having established the optimal scaling for the teacher, we can revisit the optimal student estimator. Its scaling β∗=a0a1\beta^*=a_0a_1 combines two corrections: a0a_0 accounts for the original teacher's estimation error under KL loss, while a1a_1 accounts for the additional estimation error introduced by finite synthetic sampling.

This decomposition explains why the phase transition occurs when comparing the student with an ordinary MLE teacher. The student's expected KL gap can be written as

R(β∗S1)−R(Σ^0)=d2[ψd(N1)+log⁡2N1−d−1]−d2(a0−1−log⁡a0).R(\beta^*S_1)-R(\widehat\Sigma_0)=\frac d2\left[\psi_d(N_1)+\log\frac{2}{N_1-d-1}\right]-\frac d2(a_0-1-\log a_0).

The first term is the additional synthetic-sampling cost, which decreases to zero as N1N_1 increases. The second term is the benefit of optimally correcting the MLE teacher, which is fixed for a given N0N_0 and dd. With too few synthetic samples, the sampling cost outweighs the correction benefit, and the student performs worse. Once N1N_1 is sufficiently large, the cost falls below this benefit, and the student outperforms the uncorrected teacher. The transition therefore reflects a balance between estimator-correction benefit and additional sampling cost.

2. The student advantage disappears after correcting the teacher. The paper compares the optimally scaled student against an uncorrected MLE teacher. However, a more appropriate baseline would be a teacher with the same optimal covariance correction, given by Σ~0=a0Σ^0\widetilde\Sigma_0=a_0\widehat\Sigma_0.

The paper's original optimally scaled student is obtained by drawing synthetic samples zj∼N(0,Σ^0)z_j\sim\mathcal N(0,\widehat\Sigma_0), computing their empirical covariance S1S_1, and applying the optimal scaling β∗=a0a1\beta^*=a_0a_1, yielding

Σ1∗=β∗S1=a0a1S1.\Sigma_1^*=\beta^*S_1=a_0a_1S_1.

Importantly, this student is equivalent in distribution to one trained on synthetic samples from the corrected teacher. To see this, consider z~j=a0zj\widetilde z_j=\sqrt{a_0}z_j, which follows N(0,Σ~0)\mathcal N(0,\widetilde\Sigma_0). Its empirical covariance is therefore S~1=a0S1\widetilde S_1=a_0S_1. Since the teacher-level correction a0a_0 has already been applied, the student only needs the additional correction a1a_1, giving Σ~1=a1S~1=a0a1S1=Σ1∗.\widetilde\Sigma_1=a_1\widetilde S_1=a_0a_1S_1=\Sigma_1^*.

Thus, the paper's original optimal student can be equivalently viewed as a student trained on synthetic data from the corrected teacher, allowing us to compare the two under the same correction framework.

Under the analyzed Gaussian setting, the expected KL risk difference between this student and the corrected teacher satisfies R(Σ~1)−R(Σ~0)=d2[ψd(N1)+log⁡2N1−d−1]>0.R(\widetilde\Sigma_1)-R(\widetilde\Sigma_0)=\frac d2\left[\psi_d(N_1)+\log\frac{2}{N_1-d-1}\right]>0.

Therefore, even the optimally scaled student has a higher expected KL risk than the corrected teacher for every finite synthetic-sample budget. As N1N_1 increases, the additional sampling error decreases, but the student cannot outperform the corrected teacher:

lim⁡N1→∞[R(Σ~1)−R(Σ~0)]=0.\lim_{N_1\to\infty}\left[R(\widetilde\Sigma_1)-R(\widetilde\Sigma_0)\right]=0.

This suggests that, within the analyzed Gaussian setting and estimator class, the reported student advantage comes from the covariance correction rather than from synthetic-data training itself. Once the same correction is applied to the teacher, the student can at best approach its performance.

3. Errors in the proof of the high-probability guarantees. Appendix F contains a variance bound that is false under its stated conditions. For example, taking d=1d=1, N0=N1=5N_0=N_1=5, and β=β∗=25/9\beta=\beta^*=25/9, a direct calculation using independent chi-square variables gives

Var⁡(Δ)=14ψ(1)(5/2)+2518≈1.511478.\operatorname{Var}(\Delta)=\frac14\psi^{(1)}(5/2)+\frac{25}{18}\approx1.511478.

However, substituting the same parameters into Eqs. (66)–(68) gives c1=77/64c_1=77/64 and c2=13/4c_2=13/4, so the purported upper bound in Eq. (65) is only

77/64(5/2−1)2+134(2/5)2≈1.054722.\frac{77/64}{(5/2-1)^2}+\frac{13}{4}(2/5)^2\approx1.054722.

The actual variance therefore exceeds the claimed upper bound.

4. The progressive-improvement claims require correction and qualification. Theorem 3 states that E[Δk:k+1]→0−\mathbb E[\Delta_{k:k+1}]\to0^- as ρ→0+\rho\to0^+. This is false for fixed sample budgets. For example, with L=2L=2, d=1d=1, N0=10N_0=10, and N1=100N_1=100, the prescribed scaling is β1=(10/8)(100/98)\beta_1=(10/8)(100/98), independent of ρ\rho, and

E[Δ0:1]=12[ψ(50)+log⁡298−ϕ(0.8)]≈−0.008343537.\mathbb E[\Delta_{0:1}]=\frac12\left[\psi(50)+\log\frac{2}{98}-\phi(0.8)\right]\approx-0.008343537.

Thus, the actual gap does not converge to zero. Only the displayed guaranteed upper bound necessarily tends to zero. In addition, at the exact sample threshold, the last stage can merely tie its predecessor, so strict improvement requires a strict budget inequality. Finally, the exponential budget scaling follows from the proposed construction with fixed ρ>0\rho>0 and it is not a general necessity result for all progressively improving chains. Indeed, SL=((1+ρ)L−1−1)/ρS_L=((1+\rho)^{L-1}-1)/\rho approaches L−1L-1 as ρ→0\rho\to0. These distinctions should be reflected in the theorem statement and the discussion.

5. Limited evidence for applicability beyond the Gaussian setting. The principal guarantees concern Gaussian distributions and a specified class of scaled covariance estimators. Most experiments validate this setting through Gaussian simulations or closed-form linear flow-matching models. The CIFAR-10 experiment provides a useful departure from synthetic Gaussian data, but it fits LDA to fixed ResNet-50 features rather than training a general discriminative or generative model. Its reported improvement is primarily in test log-likelihood, with little change in classification accuracy at sufficiently large synthetic budgets. Consequently, the experiments illustrate the theoretical mechanism but provide limited evidence that the proposed correction or sample-budget predictions remain useful for more general synthetic-data training. The practical claims should be narrowed accordingly or supported by experiments beyond these restricted models.

Questions for authors
  1. Can the authors compare the proposed student with an optimally corrected teacher using covariance a0Σ^0a_0\widehat\Sigma_0?

  2. In the corrected teacher setting, the optimal student has strictly higher expected KL risk than this baseline for every finite synthetic-sample budget. How does this affect the interpretation of synthetic-data training as a mechanism for surpassing the teacher?

  3. Can the authors address the counterexample to the variance upper bound in Eq. (65) and correct the two sign errors in Eq. (177), which contradict the variance decomposition in Eqs. (163), (174), and (176)? The substitution of δˉ=d2+2ε\bar\delta=d^{2+2\varepsilon} in Appendix F also violates the stated condition δˉ∈(0,1]\bar\delta\in(0,1]. Which of the explicit PAC guarantees in Theorem 2 can be rigorously recovered after correcting these steps? Please also clarify the fixed-budget ρ→0\rho\to0 claim in Theorem 3.

  4. What evidence supports extending the proposed correction or sample-budget predictions beyond Gaussian covariance estimation and closed-form linear models? Can the authors provide an experiment in a broader model class, including a directly corrected teacher baseline, or clarify the intended scope of the practical conclusions?

Minor comments

  1. On page 4, the statement that ϕ(x)=log⁡x+1/x−1\phi(x)=\log x+1/x-1 is convex on (0,∞)(0,\infty) is incorrect, since ϕ′′(x)=(2−x)/x3<0\phi''(x)=(2-x)/x^3<0 for x>2x>2. Please correct this statement. Its unique minimum at x=1x=1 and the relevant branch monotonicity remain valid.

  2. At line 78, Fu et al. (2024) is incorrectly categorized as reinforcement learning. The cited paper, “Towards Theoretical Understandings of Self-Consuming Generative Models,” studies recursive training of generative models, including diffusion models and kernel density estimation. Please correct the categorization and clarify its relationship to the present generative analysis.

3 comments

We thank the reviewer for their time and the rigorous review, including the worked derivations. We are encouraged that the reviewer already sees the analysis as a useful characterization, in the reviewer's words:

  • The paper is "clearly written and well organized."
  • The exact expected-risk analysis "provides a useful quantitative characterization of how estimator scaling and synthetic-sample budgets affect student performance."

Below are the detailed comments regarding your comments with W as your indicated weakness, and Q as your indicated question.

Q1 and W1. Comparison with corrected teacher. The standard pipeline to train generative models is to perform MLE, and all the pre-trained models are trained using these principles (see Goodfellow et al. (2016, Sec. 5.5) and Ho et al. (2020, Eq. 3)). Therefore, we consider the uncorrected (MLE) teacher to reflect the pre-trained models used in the practice and characterize the phase transitions for this case.

We agree that if the teacher were corrected (covariance with a0Σ0a_0\Sigma_0), then no student estimate that is purely a function of corrected teacher samples is better than the corrected teacher itself, as demonstrated by your arguments (we spelled this out in a proposition in appendix K) along with a numerical simulation (see Figure 11). Please see our response to Reviewer oCjG Q1 and W1 for further detailed explanation.

Q2 and W2. Student-teacher setting when teacher is corrected.

When the teacher is corrected no student in (β‾,β‾)(\overline{\beta}, \underline{\beta}) class can surpass its teacher, which is a known result from classical paradigms. We argue that corrected teacher case is not practical, specifically,

  • Applying correction to teachers using standard training pipelines is uncommon.
  • With real data and general model class there is no known analogue of correction factor for MLE based training like we have in Gaussian model.

Therefore, it is more practical to study student-teacher setting with uncorrected teacher. Please see our response to Reviewer oCjG Q1 and W1 for detailed explanation.

Q3 and W3. Proof of Theorem 2

We thank the reviewer for the careful reading. Our proof had a wrong exponent of 22 in the second term of Eq (64) (submitted Eq (65)), which results in the counterexample provided by the reviewer.
Upon similar comments by other reviewers, we have corrected this in the revised version, which we note has small impacts on the final results. Namely, we note that:

(i) Polynomial decay in the dimension dd remains intact with some changes to the constants; for the student-wins case the rate is now O(d−2+2ε)\mathcal{O}\left(d^{-2+2\varepsilon}\right) instead of O(d−2−2ε)\mathcal{O}\left(d^{-2-2\varepsilon}\right).

(ii) The comparison of Monte Carlo with PAC bounds was performed on the exact closed form expressions of mean and variance therefore the conclusions do not change.

(iii) We no longer substitute δ‾\overline{\delta}. Instead, we set κ\kappa directly in each regime so that κ≤1/4\kappa \le 1/4 (see lines 1466–1508).

Please see Table 2 for numerical comparison of our bounds with the theory.

Q3 and W4. Clarification of Theorem 3

We thank the reviewer again for the careful reading, and we note that this was a typo in the result state in the paper, which should read E[Δk:k+1]≤0\mathbb{E}[\Delta_{k:k+1}] \leq 0 (which is what is stated and proven in more detail in the appendix version of the theorem on Eq. (108)) rather than the expectation converging to 00.
This change has no impact on the interpretation of the result, as it shows that successive application of estimators in the chain improve the performance of the estimator (and indeed, it is actually better if this expectation is strictly negative away from 00 as it provides a larger improvement in performance with the successive estimators).

After considering the suggestions from R-Y5Di (Q3 and Q4) along with yours:

(i) We have included the exact expression of the mean in the Theorem 3 statement (Eq (9)).

(ii) We derived tighter conditions for sample complexities in the multi-stage students, that is, linear growth w.r.t the length is also sufficient. Our earlier result was correct, but it was not tight.

Q4 and W5. Evidence beyond the Gaussian setting.

Our conclusions extend beyond Gaussian setting in practice. We have conducted the following empirical study to support our stance:

(i) Synthetic data. We now have included evidence of phase-transition for seven non-Gaussian distributions, in dimensions d=128d=128, 256256 and 512512. For six of them the median relative error of the boundary w.r.t. the Gaussian prediction is between −6.1%-6.1\% and +5.0%+5.0\%, and the boundary grows as N02N_0^2 (log-log slope 2.082.08--2.102.10, against 2.102.10 for the Gaussian theory). Student-tt is the exception: its boundary is about 76%76\% lower than the Gaussian prediction, so the student wins with fewer synthetic samples and the Gaussian condition remains sufficient. Please see Appendix C, Table 1 and Figures 7 and 8 for the detailed experimental setup and results.

(ii) Flow matching on real data. We now have Flow matching results on CIFAR 10 dataset with ResNet-50 features that lead us to conclude the same phenomenon. See Figure 3b and updated the experimental description.

As for the corrected teacher baseline we have now included this in Figure 3(a), and Figure (11).

Minor comments All of them are fixed, and none of the conclusions are affected by this.

Conclusion.

We thank the reviewer for the critical review, which made the paper technically stronger and wider in scope. We hope that we have addressed all the concerns of the reviewer, and we are happy to follow up on any remaining point.

Thank you for the detailed response and the substantial revisions. After examining the revised manuscript, I have a few remaining concerns about the interpretation and practical implications of the central result.

The paper investigates when a student trained entirely on synthetic data can outperform its teacher. However, my main concern is the source of the student's improvement. The optimal student scaling decomposes into two factors: one corrects the teacher's covariance estimation error under the expected forward KL criterion, while the other compensates for the additional error introduced by finite synthetic sampling. When the teacher's training sample size N0N_0 is known, the first correction can be applied directly to the fitted teacher. In the zero-mean Gaussian setting, this requires only rescaling the teacher's outputs, without retraining or fitting a separate student. As Proposition 4 now establishes, the optimally scaled student has strictly higher expected KL risk than the corrected teacher for every finite synthetic-sample budget. Thus, the phase transition characterizes when the benefit of correcting an unscaled MLE teacher outweighs the additional cost of synthetic sampling, rather than demonstrating an intrinsic advantage of synthetic-data training over an equally corrected teacher.

This also raises a practical question about how the proposed correction would be selected. The theoretically optimal student scaling depends explicitly on N0N_0. If this quantity is available, the corresponding teacher-level correction can be applied directly in the analyzed setting. If it is unavailable to the student, the proposed optimal scaling cannot be computed as specified. Generating more samples from a single fitted teacher can improve estimation of the teacher's fitted distribution, but does not by itself identify the size of its original training dataset. The revised analysis therefore leaves unclear how a student with only sampling access to the teacher would select the correction, or whether the predicted advantage would persist under a practically estimated scaling factor.

The CIFAR-10 experiments remain limited in scope. The LDA results show improvements in log-likelihood but negligible changes in classification accuracy. The new flow-matching experiment operates on 32-dimensional ResNet features using a closed-form linear flow, rather than generating images. It therefore provides evidence for the predicted phase transition in feature space, but does not demonstrate improvements in image generation or sample quality. These experiments offer limited evidence for the practical applicability of the proposed approach to more general machine learning models.

Overall, while the revision addresses several of my earlier technical concerns, my main concern regarding the significance of the central result remains. I understand that theoretical analyses often rely on idealized settings and simplifying assumptions to obtain tractable results. Nevertheless, even within the analyzed setting, the student's improvement primarily arises from correcting the unscaled MLE teacher, a correction that can be applied directly without synthetic-data training. Moreover, the dependence on the teacher's training sample size and the limited empirical validation leave the practical relevance of the proposed approach unclear. For these reasons, I maintain my Weak Reject recommendation.

Reviewer_oCjGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies a generative student-teacher pipeline where the true distribution is an unknown dd-dimensional Gaussian p∗={N}(0,Σ\*)p_* = \mathcal\{N\}(0, \Sigma_\*) (Σ\*∈{S}{++}d\Sigma_\* \in \mathbb\{S\}_\{++\}^d), a teacher learns pθ0=N(0,Σ0)p_{\theta_0} = \mathcal{N}(0, \Sigma_0) from N0N_0 i.i.d. real samples X0∈Rd×N0X_0 \in \mathbb{R}^{d \times N_0} via the standard maximum likelihood estimator (MLE) Σ0=1N0X0X0⊤\Sigma_0 = \frac{1}{N_0} X_0 X_0^\top, and a student learns pθ1=N(0,Σ1)p_{\theta_1} = \mathcal{N}(0, \Sigma_1) purely from N1N_1 i.i.d. synthetic samples X1∈Rd×N1X_1 \in \mathbb{R}^{d \times N_1} drawn from pθ0p_{\theta_0}.

Performance is evaluated by the forward KL divergence gap Δ0:1:=KL(p∗∥pθ1)−KL(p∗∥pθ0)\Delta_{0:1} := \mathrm{KL}(p_* \parallel p_{\theta_1}) - \mathrm{KL}(p_* \parallel p_{\theta_0}) (equivalently, expected test log-likelihood on real data). Under standard baseline training where the student is also an unbiased MLE (Σ1=1N1X1X1⊤\Sigma_1 = \frac{1}{N_1} X_1 X_1^\top), synthetic retraining suffers from two compounding finite-sample distortions in 2KL(p∗∥N(0,Σ))=ln⁡det⁡Σdet⁡Σ∗−d+⟨Σ−1,Σ∗⟩2\mathrm{KL}(p_* \parallel \mathcal{N}(0, \Sigma)) = \ln\frac{\det \Sigma}{\det \Sigma_*} - d + \langle \Sigma^{-1}, \Sigma_* \rangle:

  1. Precision upward bias: By Jensen's inequality on the inverse-Wishart distribution, E[ΣML(N,Σ)−1]=NN−d−1Σ−1≈(1+dN)Σ−1\mathbb{E}[\Sigma_{\mathrm{ML}}(N, \Sigma)^{-1}] = \frac{N}{N-d-1}\Sigma^{-1} \approx (1 + \frac{d}{N})\Sigma^{-1}, which inflates the trace penalty ⟨Σ1−1,Σ∗⟩\langle \Sigma_1^{-1}, \Sigma_* \rangle.
  2. Log-determinant shrinkage: Eigenvalue dispersion causes E[ln⁡det⁡ΣML(N,Σ)]−ln⁡det⁡Σ=dln⁡(2/N)+dψd(N)≈−d22N\mathbb{E}[\ln\det \Sigma_{\mathrm{ML}}(N, \Sigma)] - \ln\det\Sigma = d\ln(2/N) + d\psi_d(N) \approx -\frac{d^2}{2N}, so an unbiased MLE student is strictly worse than the teacher (E[Δ0:1]>0\mathbb{E}[\Delta_{0:1}] > 0) for every (d,N0,N1)(d, N_0, N_1).

The authors propose two partial solutions to the above issues (along with their caveats):

  1. Biased Student Estimators (Section 2): The student is chosen from the class of (β‾,β‾)(\overline{\beta}, \underline{\beta})-MLEs satisfying β‾N1X1X1⊤⪯Σ1⪯β‾N1X1X1⊤\frac{\underline{\beta}}{N_1} X_1 X_1^\top \preceq \Sigma_1 \preceq \frac{\overline{\beta}}{N_1} X_1 X_1^\top. Inflating the student's empirical covariance by β0:1∗=N0N1(N0−d−1)(N1−d−1)≈(1−dN0)−1(1−dN1)−1>1\beta_{0:1}^* = \frac{N_0 N_1}{(N_0 - d - 1)(N_1 - d - 1)} \approx (1 - \frac{d}{N_0})^{-1}(1 - \frac{d}{N_1})^{-1} > 1 eliminates the compounded inverse-Wishart precision bias - recovering the unscaled teacher's Θ(d2/N02)\Theta(d^2 / N_0^2) KL suboptimality- at the expense of an unavoidable Θ(d2/N1)\Theta(d^2 / N_1) Wishart log-determinant variance penalty from synthetic sampling. Comparing these two terms yields an exact phase transition at N1=hd(N0)=Θ(N02/d)N_1 = h_d(N_0) = \Theta(N_0^2 / d), which holds both in expectation (Theorem 1, Corollary 1) and with probability 1−O(d−2∓2ε)1 - \mathcal{O}(d^{-2 \mp 2\varepsilon}) via fourth-order inverse-Wishart variance bounds and Cantelli's inequality (Theorem 2).
  2. Multi-Stage Synthetic Chains (Section 3): Extending the pipeline to an (L−1)(L-1)-generation chain where learner kk trains on NkN_k synthetic samples from learner k−1k-1, the authors construct a telescoping sequence of stage-wise scalings βk\beta_k that guarantees strict improvement at every generation (E[Δk:k+1]<0\mathbb{E}[\Delta_{k:k+1}] < 0), but show that the required per-stage sample size grows exponentially in chain length as NL∗=Θ((1+ρ)L−1−1ρN02d)N_L^* = \Theta(\frac{(1+\rho)^{L-1}-1}{\rho} \frac{N_0^2}{d}) (Theorem 3). Moreover, every additional intermediate generation strictly degrades the best achievable end-to-end KL divergence by d2(ψd(Nm)+ln⁡2Nm−d−1)=Θ(d2/Nm)>0\frac{d}{2}(\psi_d(N_m) + \ln\frac{2}{N_m-d-1}) = \Theta(d^2 / N_m) > 0, proving that a single teacher-to-student stage is globally optimal (Theorem 4).
Critical strengths

The Wishart / inverse-Wishart moment derivations (Theorems 1-5, Corollary 1, Proposition 1) are exact, non-asymptotic, and tight across all N0,N1≥d+2N_0, N_1 \ge d+2.

The multi-stage "No Free Bootstrapping" dichotomy (stepwise improvement is possible at exponential sample cost, yet end-to-end optimal performance strictly degrades with chain length) and the Cochran decomposition for unknown class means in multi-class LDA (Theorem 5, Lemma 1) are mathematically elegant.

I believe that the technical merits of this paper are significant and would be open to increasing my score once several of the gaps (discussed below) are addressed.

Critical weaknesses

First, conceptually, the pipeline X0→Σ0→X1→Σ1X_0 \to \Sigma_0 \to X_1 \to \Sigma_1 is a Markov chain, so X1X_1 contains strictly less information about Σ∗\Sigma_* than Σ0\Sigma_0. The student beats the teacher as N1≳N02/dN_1 \gtrsim N_0^2/d only because the teacher is restricted to the unscaled (1,1)(1,1)-MLE (which is suboptimal under forward KL due to its own inverse-Wishart bias γ0=N0N0−d−1\gamma_0 = \frac{N_0}{N_0-d-1}), whereas the student is permitted post-hoc scaling β0:1∗=γ0γ1\beta_{0:1}^* = \gamma_0 \gamma_1 that corrects the teacher's unscaled error Θ(d2/N02)\Theta(d^2/N_0^2) once synthetic sampling noise Θ(d2/N1)\Theta(d^2/N_1) is small enough. If the teacher itself were scaled by γ0\gamma_0 (or if its output samples were directly rescaled by γ0\sqrt{\gamma_0}), no student could ever beat the teacher for any finite N1N_1.

Second, in multi-class LDA with balanced classes (CIFAR-10), multiplying the shared covariance {^Σ}{{LDA}}\hat\{\Sigma\}_\{\mathrm\{LDA\}\} by any scalar β>0\beta > 0 leaves the Bayes classification rule {^y}θ(x)\hat\{y\}_\theta(x) strictly invariant (improving generative log-likelihood rather than test accuracy, which remains below the teacher's in Figure 3a), and Flow Matching is only evaluated on synthetic 5D Gaussian data.

Third, the proof of Theorem 4 is omitted from the appendices, alongside several copy-editing defects on Page 6 and in Appendices E, F, and H.

Questions for authors
  1. Clarifying the role of teacher bias and the direct-rescaling baseline:
    • Can you add a remark after Theorem 1 explicitly noting that the (1,1)(1,1)-MLE teacher Σ0\Sigma_0 is suboptimal under forward KL due to inverse-Wishart precision bias (E[Σ0−1]=γ0Σ∗−1\mathbb{E}[\Sigma_0^{-1}] = \gamma_0 \Sigma_*^{-1}), and compare the student Σ1=β0:1∗ΣML(N1,Σ0)\Sigma_1 = \beta_{0:1}^* \Sigma_{\mathrm{ML}}(N_1, \Sigma_0) against the directly rescaled teacher γ0Σ0=N0N0−d−1Σ0\gamma_0 \Sigma_0 = \frac{N_0}{N_0-d-1}\Sigma_0?
    • In what practical scenarios does a learner have access to N0N_0 and dd (required to compute β0:1∗\beta_{0:1}^*) and the ability to rescale covariance/samples, yet cannot apply the γ0\gamma_0 correction directly to the teacher's output?
  2. LDA accuracy invariance under balanced classes:
    • Can you clarify in Contribution 4 (p. 3) and Section 2.2 (p. 6-7) that for balanced class priors πk=1/K\pi_k = 1/K, the LDA classification boundary {^y}θ(x)\hat\{y\}_\theta(x) in Eq. (16) is invariant to any scalar multiplier β>0\beta > 0 on {^Σ}{{LDA}}\hat\{\Sigma\}_\{\mathrm\{LDA\}\}, so the student improves generative log-likelihood (and probability calibration) rather than classification accuracy?
  3. Completing the proof of Theorem 4 and strengthening its statement:
    • Please include the full proof of Theorem 4 (both the backward-induction conditional Wishart proof of Eq. 12 and the minimization yielding Eq. 13) in Appendix G, and consider noting that by Eq. (88), min⁡β(1),…,β(m)>0E[Δ0:m]=min⁡β>0Δ~0:m(β)\min_{\beta^{(1)}, \dots, \beta^{(m)} > 0} \mathbb{E}[\Delta_{0:m}] = \min_{\beta > 0} \tilde{\Delta}_{0:m}(\beta), so Theorem 4 holds even when each stage l∈[m]l \in [m] uses a distinct scaling factor β(l)\beta^{(l)}.
  4. Clarifying the c2c_2 bound in Appendix F (Eqs. 65-69):
    • Can you clean up Eqs. (65)-(69) by keeping the first term of Eq. (64) linear in d+1N1\frac{d+1}{N_1} (since g1g2β∗−N1N1−d−1=O(dN0+dN1)\frac{g_1 g_2}{\beta^*} - \frac{N_1}{N_1-d-1} = \mathcal{O}(\frac{d}{N_0} + \frac{d}{N_1})) rather than factoring out [d+1N1]2\big[\frac{d+1}{N_1}\big]^2 in Eq. (65) and switching back to [d+1N1]\big[\frac{d+1}{N_1}\big] in Eq. (69)?
  5. Minor corrections:
    • Page 3, Definition 1 (line 153 & 161): Clarify the tuple ordering in (β‾,β‾)(\overline{\beta}, \underline{\beta})-MLE (upper bound first, lower bound second, β‾≥β‾\overline{\beta} \ge \underline{\beta}) so that the minimax (NN−d+1,NN+d−1)\big(\frac{N}{N-d+1}, \frac{N}{N+d-1}\big)-MLE and Haff (1+T,1)(1+T, 1)-MLE examples are unambiguous.
    • Page 5, Theorem 1 (line 235): E[Δ0,1]<0\mathbb{E}[\Delta_{0,1}] < 0 →\to E[Δ0:1]<0\mathbb{E}[\Delta_{0:1}] < 0.
    • Page 5, Corollary 1 (Eq. 6): Note that for ζ>1\zeta > 1, g(1/ζ)=ln⁡(1/ζ)+1/ζ−1<0g(1/\zeta) = \ln(1/\zeta) + 1/\zeta - 1 < 0, so the mis-scaling term +2g(1/ζ)+2g(1/\zeta) in the lower bound is negative.
    • Page 6, lines 302-315: Remove the duplicate draft paragraph (lines 302-312) and fix "unscaled student" (line 311) to "rescaled student".
    • Page 8, Figure 4 caption (line 394): "all student improves" →\to "every student improves".
    • Page 9, line 447: "albeight" →\to "albeit".
    • Page 17, Theorem 5 (lines 867-884): Unify the class-count notation between N{kl}N_\{kl\} and Nl{(k)}N_l^\{(k)\}, and add the missing {E}\mathbb\{E\} on the right-hand side of line 903 ({E}[{KL}(p{θ∗}∥p{θ1})]≤{E}[{KL}(p{θ∗}∥p{θ0})]\mathbb\{E\}[\mathrm\{KL\}(p_\{\theta^*\} \parallel p_\{\theta_1\})] \le \mathbb\{E\}[\mathrm\{KL\}(p_\{\theta^*\} \parallel p_\{\theta_0\})]).
    • Page 18, Corollary 2 (line 948): "on MM such samples" →\to "on N1N_1 such samples".
    • Page 34, line 1818: Move the variance lemmas (Theorem 6, Lemmas 2-4, Proposition 1) into Appendix F or a dedicated section header, and fix the self-contradictory phrasing "avoiding the loose concentration-based approach of Theorem 2".
4 comments

We thank the reviewer for their time and the detailed review. We are encouraged that the reviewer already finds the technical merits significant, in the reviewer's words:

  • The derivations are "exact, non-asymptotic, and tight."
  • The no-free-bootstrapping result and the LDA decomposition are "mathematically elegant."
  • The technical merits of the paper are "significant."

Below are our detailed replies (W for weakness, Q for question).

Q1 and W1. Uncorrected teacher and rescaled-teacher baseline.

We agree that clearly if the teacher derived its estimate using the optimal scaling then it would always outperform the student. However, the main point of our paper is to consider when standard estimators (such as MLE, which is arguably the most widely used estimator) can be outperformed in the student-teacher setting. This is why we focus the main claims of the paper in comparison to an uncorrected teacher. To address the additional details of your questions:

Part 1. For the sake of completeness we add a remark after the Theorem 1 with reference to a new proposition (see remark after Theorem 1 (Sec. 2.1), Proposition 4 and Figure 11 in Appendix K).

Part 2. The teacher knows N0N_0 and dd, yet does not apply correction (γ0\gamma_0), for two reasons:

  • The correction isn't part of the standard pipeline: Pre-trained models are trained by maximum likelihood and then released. The objective has no step for the finite-sample correction factor γ0\gamma_0 (Goodfellow et al., 2016, section 5.5; Ho et al., 2020, Eq. 3). The teacher has the access to N0N_0 and dd but nothing in its training procedure uses them.

  • No known correction outside the Gaussian model: Since the MLE has no closed form for real data and general model classes, we know of no analogue of γ0\gamma_0.

W2 and Q2. LDA contributions.

Our claim here is regarding the log-likelihood of the estimators, not the classification accuracy.
The reviewer is correct that the decision boundary of the LDA classifier (under uniform class priors) does not change between the student and the teacher as the bias of the student estimator affects the scaling of the estimator, which is the primary factor in improving the log-likelihood claimed in the result (see Sec. 2.2, LDA paragraph).

W2. Flow matching on real data.

We performed Flow Matching only on synthetic Gaussian data purely for the reasons that the closed form flow solution is available. For the sake of completeness,

  • We performed and added an experiment with flow matching on CIFAR-10 image features extracted from ResNet-50 (see our reply to Reviewer Y5Di Q1). The theoretical predictions again stay close to the observed experimental results despite the non-Gaussianity of the real features: the median relative error w.r.t. Gaussian of the boundary is about −8%-8\% at the final stopping time (t=1t=1) (see the updated Figure 3(b) and section 2.2 FM paragraph).

  • We also added synthetic experiments with true distribution belonging to seven known non-Gaussian truth distributions (Student-t, Gaussian mixture, Laplace, uniform, exponential, Gamma and Weibull), in dimensions 128,256128, 256 and 512512 (see Appendix C, Table 1 and Figures 7 and 8). For six of the seven non-Gaussian distributions, the median relative error of the boundary lies between −6.1%-6.1\% and +5.0%+5.0\%. Student-t is the only exception: its boundary is about 76%76\% lower than the Gaussian theory prediction, so for this case the Gaussian prediction is a sufficient condition.

Q3 and W3. Missing proof.

The proof of Theorem 4 is a straight-forward extension of Theorem 3. For completeness, we now spell it out in appendix (see Appendix J (Proof of Theorem 4)). Upon considering the suggestion provided by the reviewer, we were able to generalize the Theorem 4 for students with different scaling factors βk\beta_k with the earlier conclusions still standing correct.

Q4. The c2c_2 bound.

We thank the reviewer for this suggestion, we cleaned up the proof in appendix F (see from Eq (64)) In Eq (65) (revised version Eq (64)), the 2nd term was meant to be c2[d+1N1]c_2\left[\frac{d+1}{N_1}\right] there was a typo in the exponent. Along with yours and similar comments from Reviewer ujcS W3 we rectified the following proof (see revised Theorem 2). We note that the conclusions of the theorem remain unchanged that is PAC guarantees get stronger for high-dimensional data.

For d=1d=1, N0=N1=5N_0 = N_1= 5, now the upper bound in Eq (64) (submitted version Eq (65)) evaluates to ≈1.67\approx 1.67 which is more than the exact value computed by you (that is ≈1.51\approx 1.51). Please see Table 2 for further numerical comparisons between exact variance and its upper bound for further sanity check.

Q5. Minor corrections. All the points are fixed in the updated draft.

Conclusion.

We thank the reviewer for the critical review, which made the paper technically stronger and wider in scope: we compare with the corrected teacher explicitly, the proofs the reviewer asked for are spelled out for completeness, and the experiments include real features. We have addressed each point above and are happy to clarify further.

Thank you for the thorough rebuttal and extensive revisions. I have verified the updated manuscript and confirmed that all of my technical and empirical concerns have been addressed. Accordingly, I have raised my Soundness score from 2 to 3 and maintain my positive recommendation.

Note that I am I am keeping my overall rating at 3 (Weak acceptance) because the two reasons given in the rebuttal for why the student can apply β0:1∗=γ0γ1\beta_{0:1}^* = \gamma_0 \gamma_1 while the teacher remains unscaled do not fully resolve the baseline asymmetry: (i) even if a pre-trained teacher is released via standard MLE, a downstream user who knows (N0,d)(N_0, d) and can rescale teacher samples by \sqrt\{\beta_{0:1}^{*}} (as in Corollary 2, Eq. 28) can simply rescale the teacher's outputs by γ0\sqrt{\gamma_0} directly at inference time without training a student, and (ii) the lack of a closed-form analogue of γ0\gamma_0 outside Gaussian models applies equally to the student's factor β0:1∗=γ0γ1\beta_{0:1}^* = \gamma_0 \gamma_1.

We thank the reviewer for checking the revised manuscript and for raising the soundness score.

(i) We agree with the reviewer's point; we will position our contribution accordingly in the final version.

(ii) We agree that our theory does not cover non-Gaussian models. Empirically, we report the best student scaling β^\hat\beta when the true distribution is non-Gaussian (see Appendix C). This is the β\beta that minimises the student's expected KL, selected on 500 trials. The table below compares it with β0:1∗=N0N1/((N0−d−1)(N1−d−1))\beta^*_{0:1} = N_0N_1/((N_0-d-1)(N_1-d-1)).

Median relative error of the best student scaling β^\hat{\beta} from β0:1∗\beta^*_{0:1} over all (N0,N1)(N_0, N_1) cells (in %).

Truthd=128d=128d=256d=256d=512d=512
Gaussian−0.0047-0.0047%−0.00071-0.00071%−0.00059-0.00059%
GMM+0.0026+0.0026%+0.0046+0.0046%+0.011+0.011%
Laplace+0.093+0.093%+0.042+0.042%+0.027+0.027%
uniform−0.047-0.047%−0.028-0.028%−0.014-0.014%
exponential+0.31+0.31%+0.085+0.085%+0.061+0.061%
Gamma+0.15+0.15%+0.078+0.078%+0.032+0.032%
Weibull+0.039+0.039%+0.012+0.012%+0.014+0.014%
Student-tt (ν=5\nu=5)+7.5+7.5%+7.0+7.0%+6.5+6.5%

The Gaussian row is 00 in theory; its value shows the Monte Carlo error of the selection. Therefore, empirically, β0:1∗\beta^*_{0:1} is within 0.31%0.31\% of the best scaling for six of the seven non-Gaussian truths, while for heavy-tailed truths such as Student-tt it underestimates the best scaling.

We hope this clarifies how well our theoretical predictions hold outside our assumptions.

Reviewer_Y5DiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how training student models on synthetic data generated by previous teacher models affects learning quality. The paper uses Gaussian covariance estimation to show that carefully choosing sample sizes and a correct biased covariance can allow students to outperform their teachers. The authors show that multi stage training can result in continuous improvements. But longer synthetic data training chains do not necessarily produce better final models.

Critical strengths
  1. The paper addresses an important topic of using synthetic data and possible collapse of the student models but take a different approach compare to the literature by biasing the estimators such that it compensates for the negative effect that was captured in previous papers.

  2. The theoretical framework is nice and intuitive, some of the results are sound and helpful for the practical audience as well as theoretical audience.

  3. The paper has lots of potential and I think if the authors address the critical weaknesses, they can have a positive impact on the topic.

Critical weaknesses
  1. The last section of the paper compares the muti stage with the better final student. There are some discussions and conclusion around this, however, the theorems do not establish the same optimization problem. It is false that the authors stated in the same setup of Theorem 3 in Theorem 4 as the optimization problem and details are completely different. This is misleading. To be clear, the issue is whether the second theorem genuinely rules out better performance for the same class of stage-dependent constructions and under a fair resource comparison. There is also a presentation problem in my opinion. I could locate Appendix G's proof of Theorem 3 and Appendix H's proof of Theorem 5, but could not locate a separately identified proof of Theorem 4 in the supplied manuscript. Equation (13) may be derivable from the preceding formulas, but the authors should show that derivation, state the optimization domain, and justify the strict inequality.

  2. The function ϕ\phi is not globally convex. It decreases on (0,1)(0,1), increases on (1,∞)(1,\infty), and has a unique global minimum ϕ(1)=0\phi(1)=0. So the paper's claim about the unique minimum and the existence of two inverse branches remains correct, but the convexity claim does not. I suggest the authors rephrasing their claim,

  3. The paper is lacking some recent literature in the related works:

  • HIGH-DIMENSIONAL ANALYSIS OF SYNTHETIC DATASELECTION, Rezaei et al, 2026
  • Beyond Real Data: Synthetic Data through the Lens of Regularization, Shidani et al, 2026
Questions for authors
  1. Authors have used real data in the LDA setup but not the flow matching setup. They only used the synthetic Gaussian data. It is good to explain the results but does not show effectiveness of the results in practice. Additionally, the plots shown are difficlut to understand, the caption was not helpful to understand the details of the plots and they were also hard to see the details. I suggest authors to use zoom-in versions to show the boundaries and maybe they can add the current plots in the appendix for completeness.

  2. How do authors approximate d in the practical setup? For example if they were to use the flow matching with real data, how would they do this? If d is the embedding dimension, how does the model's performance and the theoretical results change as we change this dimension? How sensitive is the results? As it clearly changes the quadratic size of the synthetic data.

  3. In the setup of Theorem 3, every stage can improve in expectation if and only if (L−1)y(N)<c(L−1)y(N) < c. Necessity follows because ϕ≥ 0\phi \geq 0. Sufficiency follows by spending c in equal decrements across the stages. This gives N≈(L−1)N02/(d+1)N \approx (L−1) N_0^2/(d+1), which is linear in L. But authors say it is exponential. Can you please help me understand the result better?

  4. It seems to me E[Δk:k+1]→0−E[\Delta_{k:k+1}] \to 0^{-} as ρ→0+\rho \to 0^+. This is false at fixed N. Each gap converges to (d/2)[y(N)−c/(L−1)](d/2)[y(N) − c/(L−1)], which is strictly negative. Is this correct?

2 comments

We thank the reviewer for their time and the detailed review. We are particularly pleased that the reviewer acknowledges the significance of our core contribution, characterizing it as:

  • The theoretical framework is "nice and intuitive."
  • Results that are "helpful for the practical audience as well as theoretical audience."
  • The paper "has lots of potential."

Below are the detailed comments regarding your comments with W as your indicated weakness, and Q as your indicated question.

W1. Clarification on Theorem 3, and 4

We argue that both Theorem 3 and 4 tackle the same problem, and that proof of Theorem 4 is a straight forward extension of Theorem 3:

(i) The confusion between setting of Theorem 3 and 4 is understandable, and we clarify it:

  • Theorem 3 asks when every stage of a chain can improve, with a separate scaling factor (βk\beta_k) for each stage.

  • Theorem 4 asks how good the whole chain can be at its best. Our sentence "in the setting of Theorem 3" is meant to imply that all aspects are the same as Theorem 3, but in the original version of the paper was restricted to a special case of Theorem 3, where all βk\beta_k scalings take the same value.

Nevertheless, based on this comment and similar comments from the Reviewer oCjG (Q3 and W3), we have now extended the result in Theorem 4 to include the more general setting where all students have general scaling factors βk\beta_k (the exact setting of Theorem 3).

(ii) The proof of Theorem 4 is derivable from proof of Theorem 3, using the argument from previous point and minimizing over the scalars βk\beta_k. For completeness, we spell out the proof in appendix (see Appendix J).

W2. Properties of function ϕ\phi

This is a minor typo we meant to say "continuous" but not "convex". The entire paper only relies on the fact that ϕ\phi is decreasing in (0,1](0, 1] , increasing in [1,∞)[1, \infty) and never the convexity (see section 2.1). We corrected the sentence.

W3. Related works After reviewing the two papers, their problem posits stronger assumptions than our work. We categorize them as analyses of linear regression with real plus synthetic data; a category in which we already cited other works (Loureiro et al., 2021; Xue et al., 2025; Lecoiu et al., 2026; see Appendix A). Our setting is generative and is more general in the following sense:

  • The student has no real inputs and trains only on synthetic data.
  • The student uses nothing from the teacher except the synthetic samples that the teacher generates.
  • In these two papers the learner also has real data: partial real inputs and partial access to the real labels.

In comparison, our results are exact and non-asymptotic: the exact phase transition in the numbers of real and synthetic samples, and a sufficient condition for improvement at every stage of a chain. We now cite both papers in the same category (line 075 and Appendix A).

Q1. Clarification on experiments

Our new experiments suggest that our conclusions hold beyond Gaussian distributions, specifically, we conducted the following experiments:

(i) Synthetic Non-Gaussian family: When the truth is one of eight distributions (Gaussian, Student-tt, Gaussian mixture, Laplace, uniform, centered exponential, Gamma, Weibull), in dimensions d∈{128,256,512}d \in \{128, 256, 512\}, the phase-transition boundary predicted for the Gaussian still holds (Appendix C, Table 1 and Figures 7 and 8). Specifically, - For the Gaussian control, the simulated boundary has the slope of the Gaussian theory (2.092.09--2.102.10 against 2.102.10) and a median relative error of +3.2%+3.2\% to +3.6%+3.6\%; since the truth is Gaussian here, this error comes from the simulation. - For six of the seven non-Gaussian distributions, the simulated boundary is close to the Gaussian prediction: - The median relative error, pooled over the six distributions, is +0.6%+0.6\% for d=128d=128, +1.9%+1.9\% for d=256d=256 and +2.4%+2.4\% for d=512d=512. - For each single distribution, the median relative error lies between −6.1%-6.1\% and +5.0%+5.0\% for d=128d=128, between −1.5%-1.5\% and +4.1%+4.1\% for d=256d=256, and between +1.0%+1.0\% and +3.7%+3.7\% for d=512d=512. - The boundary grows as N02N_0^2 (log-log slope 2.082.08--2.102.10), as for the Gaussian. - The heavy-tailed Student-tt (five degrees of freedom, so its fourth moment is barely finite) behaves differently: - The median relative error is about −76%-76\% in all three dimensions: the simulated boundary lies at about a quarter of the Gaussian prediction (median ratio 0.240.24, and as low as 0.170.17), with slope 1.891.89. - For this distribution the Gaussian prediction is a stronger requirement than needed: the phase transition occurs earlier, with about four times fewer synthetic samples (up to six times).

(ii) Flow matching on CIFAR-10 images: The truth is the ResNet-50 features of CIFAR-10 (all ten classes), projected by PCA to d=32d=32, and the teacher is a linear flow-matching model. We sweep the flow time t∈{0.6,0.8,0.9,0.95,0.975,1}t \in \{0.6, 0.8, 0.9, 0.95, 0.975, 1\} (Figure 3b). As tt approaches 11, the simulated boundary approaches the Gaussian prediction. At t=1t=1: - The boundary grows as N02N_0^2 (log-log slope 2.082.08, the same as for the Gaussian theory). - The median relative error with respect to the Gaussian prediction is −8.2%-8.2\%: the student wins with slightly fewer synthetic samples than predicted.

The revised plots have larger fonts for better readability.

Q2. Sensitivity of feature dimension

We provide characterization of phase transition in student-teacher setting for a given dimension dd, and that features follow Gaussian distribution. Since we study under-parameterized regime (Ni≥d+2N_i \geq d+2), naturally we expect to have more data if the dimension increases. However, our phase-transition results when the number of data points is normalized with d+1d+1, the sufficient conditions for student to outperform the teacher are unaffected.

In the revised version, we use ResNet-50 features of CIFAR-10 data to empirically verify our phase transition results (see Figure 3(b)), although the features are not exactly Gaussian but we hypothesize that they are closer to Gaussian than raw pixels.

Q3. Improvement in Theorem 3

After considering your suggestions and other reviewers we re-derived the sample complexity, we proved that linear growth w.r.t. the chain length of synthetic data size is also sufficient (see Theorem 3). The exponential growth result is correct but it is only sufficient and not tight. We have modified our claims accordingly (see abstract (line 32-33), and contributions (line 111-113)).

Q4. Upper bound in Theorem 3

Yes, in the original version only upper bound of Eq (9) converges to 0−0^{-}, therefore, the expected forward KL gap between consecutive students w.r.t. truth is strictly negative which is better as it shows strict improvements. For better presentation Theorem 3 now states the exact gap (Eq. (9)), which is strictly negative for fixed NN, as you note. Please see our response to Reviewer ujcS (Q3 and W4) for detailed response.

Comments on the paper

1 comment
Area_Chair_dAC5OpenReview ↗

Discussion phase coming to an end

Dear reviewers Y5Di and ujcS,

As we are approaching the deadline of the discussion phase, please read the authors’ responses and share your thoughts in the discussion as soon as possible. In particular, please clarify whether the responses have addressed your main concerns. Please also update your review and scores if appropriate.

Your timely input will help us reach a well-informed decision. Thank you for your time and effort!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.