- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SERF adds a self-distillation regularizer to continuous flow language models. An EMA teacher sees a cleaner point on the same interpolation path as the student and provides an auxiliary target. Two variants are studied: SERF-L aligns an early student block with a deeper teacher block via a projection head and cosine loss; SERF-P matches the student's output token distribution to the teacher's. The method is applied to FLM (one-hot), S-FLM (hypersphere), and ELF (frozen T5 latents). At a matched step budget, both variants improve MAUVE on OpenWebText; SERF-L trained for 300K steps exceeds the released 1M-step FLM at matched entropy, and SERF-L improves GSM8K accuracy of S-FLM by 4.6 points.
Critical strengths
- Presentation: The method and figures are easy to follow, and the design is well motivated for the text.
- The regularizer is validated on three representation types (one-hot, spherical, latent), and comparisons are made along quality–entropy fronts rather than at a single temperature.
- Experimentally, SERF-L-300K beats FLM-1M at every sampling budget on matched-entropy fronts.
Critical weaknesses
- Positioning.
- SERF-L is essentially the objective of SRA (Jiang et al., ICLR 2026) and Self-Flow (Chefer et al., ICML 2026): EMA teacher at the lower noise level, early student block mapped through an MLP head and aligned to a deeper teacher block with a cosine loss, EMA decay. Self-Flow already evaluates the teacher at min(t, s) for two independently drawn times, which is exactly the "random offset" of Sec. 3.3.
- SERF-P, in turn, is consistency training (Song et al., 2023) applied to the denoiser output: shared noise, an EMA stop-gradient target at the cleaner point, and a CE anchor. The paper cites SRA and Self-Flow as inspiration but never states this correspondence, and the consistency-model connection is absent.
- Suggestion. I do not consider the transfer to text disqualifying; the empirical findings are useful. But please add a short paragraph or table that contrasts SERF with SRA, Self-Flow, LayerSync, and consistency training point by point, and state precisely what is new.
Jiang, Dengyang, et al. "Representation alignment for diffusion transformers without external components." The Fourteenth International Conference on Learning Representations. 2026.
Song, Yang, and Prafulla Dhariwal. "Improved techniques for training consistency models." International Conference on Learning Representations. Vol. 2024. 2024.
-
Under shared noise, Eq. (4) gives , so for the pair determines . The tower property 4E[E[x_1|x_t]|x_s] = E[x_1|x_s]$ therefore does not hold, and the minimizer of L_FLM + λ·L_P at fixed x_s is biased toward sharper predictions. The "soft target that reflects plausible alternatives" narrative ( in p. 6) is therefore incomplete.
-
The random offset also reweights the main loss in time, and the ablation cannot separate the two effects. Algorithms 1–2 evaluate L_FLM at s = min(µ, κ), i.e. s ~ Beta(1, 2) with mean 1/3 rather than U(0, 1). SERF therefore changes the time distribution of the main loss in addition to adding a distillation term. In Fig. 5 (Left), SERF-P with the fixed offset (uniform s) is indistinguishable from FLM, while the random offset helps; time reweighting explains this as well. The fixed-offset control is itself degenerate (with δ = 0.8, 80% of teacher inputs are fully clean), so it does not isolate the offset choice either.
- Suggestion. If time and computation allows, I would suggest: Add (i) FLM with s ~ Beta(1, 2) and λ = 0, (ii) SERF with L_FLM evaluated at an independent uniform s, and (iii) an EMA self-distillation control with t = s. These would make the attribution convincing.
-
The 128-step entropy-matched Gen. PPL is an extrapolation. At 128 steps, both FLM and SERF-L have sampled entropies above 5.44 at all three temperatures, so "interpolating" to H⋆ = 5.44 extrapolates outside the sweep. The result is sensitive to the fitting function; a quadratic fit in log space would give SERF-L a higher value than FLM.
- Suggestion. Compare within the observed entropy overlap (or extend the τ sweep until the fronts bracket 5.44); show Gen. PPL–entropy Pareto fronts. And it would be better to also show FLM results at 1024 steps as presented in their paper.
Questions for authors
- Does the EMA-teacher alignment remain effective when the token embedding space is learned jointly with the denoiser (e.g. LangFlow/RePlaid)? In SERF both FLM (one-hot) and ELF (frozen T5) fix the input space; with learned embeddings the teacher's embedding table lags the student's, and the SERF-L target may drift.
RePlaid: Yang, Zhihan, et al. "Continuous diffusion scales competitively with discrete diffusion for language." arXiv preprint arXiv:2605.18530 (2026).
Langflow: Chen, Yuxin, et al. "Langflow: Continuous diffusion rivals discrete in language modeling." arXiv preprint arXiv:2604.11748 (2026).
-
Missing controls: The introduction argues that text lacks a suitable external representation encoder. Why are BERT-style encoders or hidden states of an autoregressive LM ruled out as REPA-style teachers? Additionally, it would be interesting to see controls on the EMA teacher against a frozen pretrained teacher, a stop-gradient copy of the student, or shared versus independent noise. One or two of these at 100K steps would suffice.
-
Implementation details: Are samples drawn from EMA or raw weights, for SERF and for the baselines? How is the shared-noise pair (x_s, x_t) constructed for S-FLM and for ELF's native schedule, where Eq. (4) does not apply directly?
Additional comments and suggestions are given under Weaknesses.
10 comments
W2. Tower property and the soft-target motivation
We thank the reviewer for this analysis. The reviewer is correct that, under shared noise, the pair determines , so the teacher's prediction does not satisfy the tower property. Below we show that SERF-P is a form of label smoothing, which supports our soft-target claim: every per-sample target places probability on plausible alternative tokens rather than only on . We further show that the resulting deviation from the posterior is bounded by λ, and that the shared-noise teacher is nonetheless the better design.
Label-smoothing view. Up to the constant factor , is cross-entropy against the target , with , i.e., label smoothing with the teacher's prediction in place of the uniform distribution. Each per-sample target is softer than the one-hot label and assigns probability to alternative tokens that the teacher considers plausible given a less corrupted context. Unlike the uniform distribution, however, is correlated with through the shared noise, which is the source of the bias.
The bias is bounded. At a fixed , the minimizer is , so its deviation from the posterior is at most at .
The teacher's target is what helps. Two ablations show that the gains come from the shared-noise teacher rather than from smoothing alone. First, replacing with the uniform distribution, i.e., standard label smoothing with the same , performs worse than SERF-P, and even below FLM. Second, building the teacher's input from independent noise, , under which the pair no longer determines , also performs worse. This supports our choice of a teacher on the same interpolation path.
| Method | Smoothing distribution | MAUVE (↑) |
|---|---|---|
| FLM | – | 0.237 ± 0.009 |
| + Label smoothing | Uniform | 0.212 ± 0.014 |
| + SERF-P, independent noise | Teacher , independent noise | 0.291 ± 0.017 |
| + SERF-P | Teacher , shared noise | 0.325 ± 0.016 |
OWT, 50K training steps, 1024 sampling steps, same for all variants. Best MAUVE over , ± std over 5 MAUVE seeds.
We have added these controls to Appendix B.
W3. Time reweighting vs. distillation
We thank the reviewer for this observation, which is correct: with , SERF also shifts the time distribution of to . We ran the suggested controls (SERF-L, OWT, 50K steps). For (ii), instead of evaluating in a separate forward pass at an independent , we sample the student time uniformly, , and the teacher time as . This removes the Beta(1, 2) reweighting from both losses while keeping the same conditional teacher time given as the random offset, so it isolates exactly the effect in question without requiring an extra forward pass.
| Run | time | Teacher time | Best MAUVE (↑) |
|---|---|---|---|
| FLM | – | 0.237 ± 0.009 | |
| (i) FLM, λ = 0 | Beta(1, 2) | – | 0.238 ± 0.022 |
| (ii) SERF-L, uniform student time | 0.268 ± 0.016 | ||
| (iii) EMA self-distillation | Beta(1, 2) | 0.246 ± 0.010 | |
| SERF-L | Beta(1, 2) | 0.270 ± 0.017 |
OWT, 50K training steps, 1024 sampling steps. Best MAUVE over , ± std over 5 MAUVE seeds.
Time reweighting alone (i) does not improve MAUVE over FLM. Conversely, removing the reweighting (ii) retains nearly the full gain of SERF-L, while removing the noise gap (iii) loses most of it. The gains of SERF-L therefore come from distillation with a cleaner teacher, not from the change in the time distribution of .
We agree that the fixed offset with δ = 0.8 is a degenerate control for SERF-P, since most teacher inputs are fully clean. For SERF-L, however, the fixed offset uses a uniform student time and already performs on par with the random offset (Fig. 5, left), consistent with control (ii). We have added these controls to Appendix B, and now state this reweighting explicitly in Sec. 3.3.
W4. Extrapolated Gen. PPL at 128 steps
The reviewer is correct: at 128 steps our sweep gave coldest entropies (FLM) and (SERF-L), both above , so the value was extrapolated. We have run , giving and : both sweeps now bracket .
To reassure the reviewer that our conclusion does not depend on the choice of interpolation, we read Gen. PPL at three ways: piecewise linear between the two straddling temperatures (the paper's scheme), a least-squares line over the sweep, and a least-squares quadratic .
| Steps | sweep | Piecewise linear (paper's scheme) | Linear | Quadratic |
|---|---|---|---|---|
| 64 | 0.85, 0.9, 0.95, 1.0 | 86.5 / 82.3 | 88.3 / 80.8 | 85.3 / 81.2 |
| 128 | 0.9, 0.95, 1.0, 1.05 | 78.8 / 75.4 | 80.1 / 74.3 | 77.8 / 74.1 |
| 256 | 0.95, 1.0, 1.05 | 75.7 / 72.6 | 76.3 / 72.5 | 75.1 / 72.0 |
| 512 | 0.95, 1.0, 1.05 | 74.7 / 71.7 | 75.7 / 73.2 | 74.4 / 71.7 |
| 1024 | 0.8 to 1.05 | 74.9 / 72.9 | 72.0 / 70.9 | 74.5 / 72.5 |
Gen. PPL (↓) as FLM-1M / SERF-L-300K at , same scheme and same temperatures for both models in every cell, 5000 samples per temperature.
Under all three schemes, SERF-L achieves lower Gen. PPL than FLM at every sampling budget.
On the last point, Fig. 6 shows the full MAUVE– and Gen. PPL–entropy fronts. The FLM Gen. PPL values in Fig. 6 at match those reported in the FLM paper, and we will add the raw values behind Fig. 6 as a table.
We thank the reviewer for the detailed and constructive review, and for recognizing that the method is clear and well motivated for text, that it is validated across three representation types with comparisons along quality–entropy fronts, and that SERF-L-300K outperforms FLM-1M at every sampling budget. We also appreciate and share the reviewer's view that SERF's empirical findings are useful. Following the reviewer's suggestions, we report new experiments and analyses:
We address each point below, and would kindly ask the reviewer to reconsider their assessment in light of these results.
W1 Positioning/novelty
(This reply is identical to our response to Reviewer K6iV.)
We thank the reviewer for highlighting these connections. The original submission explicitly builds on image-domain self-distillation, citing SRA and Self-Flow in Secs. 1, 2, and 4.4. We have now expanded this discussion with a point-by-point comparison with SRA, Self-Flow, LayerSync, and consistency training in Appendix A.4, and revised the remark in Sec. 3.2.2.
SERF-L and transfer to language. SERF-L follows SRA and Self-Flow’s principle of aligning an early student layer with a deeper EMA-teacher layer evaluated at lower noise. Our random noise-level pairing likewise follows the cleaner-teacher principle of Self-Flow. We do not claim these principles as new.
Our contribution is to establish their effectiveness for continuous diffusion language models, which is not guaranteed by their success in images. For example, a motivation for image representation alignment is to improve semantic features learned through denoising pixels or VAE latents, whereas token prediction already directly supervises discrete linguistic units. Whether intermediate alignment still provides useful additional supervision, therefore, merits investigation. We demonstrate benefits across three substantially different representations—one-hot vectors, jointly learned hyperspherical embeddings, and frozen T5 latents—and two tasks, unconditional generation and mathematical reasoning.
SERF-P and consistency training. We agree that SERF-P shares the noisy-student/cleaner-teacher construction of consistency training (CT). The quantities being learned, however, have different mathematical roles. Consistency training learns an endpoint mapping that is invariant along a probability-flow ODE trajectory, enabling direct few-step generation. SERF-P instead distills categorical token distributions to regularize the standard denoiser, retaining clean-token cross-entropy and the original sampler.
Our focus is on the effect of this loss as an auxiliary regularizer on training efficiency and generation quality, while retaining the original denoising objective and sampler. We do not claim that SERF-P learns a consistency function. We address the bias introduced by its teacher targets separately in our response on the training target.
What is new. Our contribution is the systematic investigation of noisy-to-clean self-distillation for continuous diffusion language models, establishing its benefits for training efficiency and generation quality across three representations and two tasks. SERF-P additionally studies distillation of categorical token distributions, requiring neither a projection head nor the selection of student and teacher layers. The revised discussion distinguishes these contributions from the self-distillation principles inherited from prior work.