- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors apply time-step optimization to continuous-diffusion based language models. They identify how directly applying image-diffusion metrics for this optimization is not optimal for language and propose alternative metrics that improve optimization performance. They experiment across two different latent-space diffusion language models and show gains on metrics such as generative perplexity and bleu score.
Critical strengths
The main innovation of the paper is a detailed analysis of the right metric to optimize for with timestep placement in particular for diffusion-language models. The authors show how naive metrics are suboptimal and conduct ablations to support their claims.
Critical weaknesses
First, the motivation of the paper and the central novelty is not strong enough. While the paper is motivated as an augmentation to existing timestep optimization methods tailored to continuous diffusion language models, the authors do not clarify benefits compared to other methods such as consistency models/meanflow/rectified flow/flow map. This is particularly important as the continuous language community increasingly moves towards few step generation. Moreover, the continuous-diffusion language space is pretty broad but their proposed augmentation seems to be scoped for only a particular class of language models. It may be helpful to answer the questions in the next section to strengthen the paper's motivation and novelty.
Second, the analysis of the paper's method could be stronger. At minimum, some empirical samples with 64 NFE vs 16 NFE vs 16 optimized NFE (all from the same initial gaussian noise) should be shown to allow readers to visually identify the degradation/improvement in quality when timesteps are suboptimal/optimized. Moreover, the actual optimized timesteps should also be shown for one to understand how significant the change in timepoints is. Finally, correlations of optimized timesteps with heuristics for flow curvature/straightness may help build some intuition around why timestep optimization can improve quality.
Finally, the continuous diffusion for language literature has been poorly presented. The overall motivation for continuous diffusion for language instead of the prevalent discrete diffusion and why it is important is weak. This paper is an augmentation on top of continuous diffusion for language and hence it would strengthen the paper's position if the authors can elaborate on why continuous diffusion should be optimized in the first place. Several key citations are missing. For example, several key papers from around 20222 such as diffuseq and variants are missing. The recent wave of continuous diffusion for language began with three key (and concurrent) papers - categorical flow maps, langflow and flow map language models. All of these citations are missing.
Questions for authors
- Why is a timestep optimization strategy needed given the increasing adoption of meanflow/flow map for few step distillation? How do gains compare with timestep optimization versus such distillation methods? Can the two methods be combined for compounded gains? (I understand if the authors can't find enough time to empirically conduct these in time for the rebuttals but would appreciate it if such comparisons were added to the final version of the paper)
- How would the proposed augmentations to timestep optimization be applied to other continuous diffusion language models, such as FMLM (over one-hot vectors), RePlaid (over learned embeddings) and DiffuSeq (over pre-trained embeddings). Both considered methods in the paper use an autoencoder, but there are multiple variants of continuous diffusion and it is not clear which one is the best. So robustness to the paradigm will help support the case for the paper.
- Are there toy examples that can help build intuition over why/when timestep optimization is effective?
3 comments
Response to Reviewer AzAH (2/3)
W2. Paired generation examples
For ELF, we compare Native16, StepTune16, and Native64 using the same input and initial Gaussian noise. The examples below illustrate changes in fluency and source fidelity.
WMT14. The source states that a traffic-light system cost approximately EUR 15,000 and quotes Arnold describing it as the most modern system currently available.
| Method | Generated text |
|---|---|
| Native16 | The traffic lights system cost about 15 000 euros itself is the "most modern thing there is in the market today", Arnold explained. |
| StepTune16 | The traffic lights system itself, which costs around EUR 15 000, is the "most modern thing on the market currently", explained Arnold. |
| Native64 | The traffic lights system which costs about EUR 15 000 itself is the "modernest thing in the market now", Arnold explained. |
StepTune produces a grammatical relative clause and avoids Native64's malformed superlative.
XSum. The source reports a fatal collision involving two motorbikes, the death of a 19-year-old rider, and the arrest of a 29-year-old man.
| Method | Generated text |
|---|---|
| Native16 | A man has been arrested after a teenage motorcyclist collided to the death with two bikes. |
| StepTune16 | A man has been arrested after a teenager was killed in a collision involving two motorbikes. |
| Native64 | A man has been arrested after a teenage cyclist was killed in a collision between two motorbike vehicles. |
StepTune expresses the supported events fluently, improving Native16's collision description and avoiding Native64's substitution of "cyclist." We have included these selected examples and additional matched cases in the supplementary materials.
W2. Optimized timesteps
Appendix C summarizes the 11 deployed schedules in revised Table 27 (original Table 6); the reproducibility package submitted with the original paper contains their exact grids. For clarity, the XSum16 grid is reproduced below, rounded to six decimals:
[0.000000, 0.004751, 0.009502, 0.014253, 0.019004,
0.023755, 0.028506, 0.033257, 0.038008, 0.042759,
0.047510, 0.052261, 0.057012, 0.061763, 0.068221,
0.229057, 1.000000]
Its four regions have boundaries approximately and interval allocations . Compared with Uniform16, , it allocates substantially more resolution to the beginning of the trajectory. Appendix C of the original submission analyzes this pattern across tasks and sampling budgets.
W2 / Q3. From timestep placement to token probabilities
Consider fixed dynamics , , with initial state and two Euler evaluations over . The grids and yield endpoints and , respectively; the exact endpoint is . The dynamics, initialization, total time, and NFE are identical.
Apply the same illustrative two-token readout to both endpoints:
The uniform grid gives ; the nonuniform grid gives . The selected token therefore changes from B to A, matching the token preferred at the exact endpoint. Although the nonuniform endpoint has larger Euclidean latent error (0.52950 versus 0.51253), it has smaller forward KL from the exact-endpoint readout (0.53972 versus 1.03340).
This toy example illustrates Appendix A of the original submission. First, placement changes the terminal state. Appendix A.2 (Theorem 1) expresses terminal error as a sum of propagated interval defects. Placement changes both the defects and their propagation, explaining the different numerical endpoints even for the straight-line component.
Second, decoder geometry determines which endpoint changes matter to tokens. Appendix A.5 (Theorem 3) expresses local KL as a quadratic form weighted by decoder sensitivity; Proposition 5 gives an exact decomposition along a finite latent path. Here, the readout responds to but not , so a larger latent error can yield a better token distribution. Appendix A.6 connects transported errors and decoder geometry to the joint objective. We have included the full toy calculation in Section 3 of the revised manuscript.
W2. Brief trajectory check
Following your suggestion to examine flow curvature and straightness, we conducted a separate trajectory check using Uniform256 reference trajectories for 16 XSum inputs with two initializations each. The mean path-length-to-chord ratio is 1.005, and the first of normalized time accounts for 48.5% of the accumulated turning angle.
Response to Reviewer AzAH (3/3)
W1. Novelty: systematic study and use of placement in continuous DLMs
We systematically investigate an underexplored aspect of continuous DLMs: how timestep placement shapes terminal representations and decoded token distributions. This analysis informs a language-aware calibration framework that improves generation with the model, solver, and NFE fixed. Our contribution has three connected parts:
Measuring the effect. At fixed model, solver, and NFE, we show that placement alone can substantially alter terminal representations and analyze how these changes affect decoded token distributions (Sec. 3).
Explaining what matters for language. We identify two regimes: latent distance correlates strongly with decoder KL at large deviations, but this relationship weakens near the reference, where the decoder's sensitivity to error direction becomes important. This finding motivates measuring both terminal-state and token-distribution discrepancies.
Using the findings. StepTune combines latent error with actual decoder KL, searches a compressed schedule space or candidate bank under quality constraints, and freezes the selected schedule for reuse across inputs (Secs. 3.3–4).
The original ELF and AURORA experiments evaluate this design. The new FMLM and DiffuSeq results broaden its empirical support; the cross-model geometry analysis below and the toy example in Part 2 further clarify its underlying reasoning.
Direct WMT14→XSum and 32→16-NFE transfers improve every metric over Native in all four tests without a new search (Reviewer 6mEv, Part 4, Q2).
The additional paired analysis also supports the joint objective. For the WMT14 ablation in revised Table 4 (original Table 3), we retain the default case-sensitive SacreBLEU configuration documented in revised Appendix B.5. Full improves BLEU and chrF over A-only in 9/10 prespecified calibration panels, with positive 95% CIs for both mean gains. The detailed statistics appear in our response to Reviewer 6mEv, Part 2, W2/Q3.
W1 / W2. Broader evidence for the language-specific mechanism
We repeat the terminal-readout analysis on AURORA and distilled FMLM, using six scales and 96 schedule-induced directions per model. All three models show the same reversal. Near the reference (), the readout-aware quadratic tracks actual KL better than latent distance (pooled Pearson 0.861–0.975 versus 0.081–0.250). At the full gap (), latent distance is more predictive (0.832–0.976 versus 0.009–0.147 for the quadratic).
This supports the language-specific finding across distinct readouts, including FMLM without a separate decoder. StepTune uses actual KL; the quadratic diagnoses local geometry. Full correlations are reported in revised Appendix B.8, Table 20. Construction and aggregation details are also discussed in our response to Reviewer 3m1w, Part 2 (W2/Q2 and the cross-model analysis).
W3. Positioning within continuous language modeling
Thank you for suggesting a broader discussion of continuous language modeling. We have expanded Section 2 (Related Work) of the revised manuscript to cover early embedding-space diffusion and recent continuous one-hot/simplex and learned-embedding approaches, including Diffusion-LM, DiffuSeq, Categorical Flow Maps, LangFlow, FMLM, and RePlaid. Diffusion-LM, DiffuSeq, Categorical Flow Maps.
Continuous language modeling is worth optimizing because it already supports competitive generation and offers effective routes to few-step inference. LangFlow and RePlaid demonstrate competitive continuous approaches, while FMLM and Categorical Flow Maps demonstrate the potential of continuous flow maps for few-step generation. Our work asks how to improve the use of such models after training, through their finite inference grid. LangFlow, RePlaid, FMLM.
Thank you again for the constructive suggestions. They have helped us broaden the empirical support and make the paper's motivation and scope more explicit. We hope these results address your concerns and would appreciate your updated assessment in light of the additional evidence. We welcome further discussion of any remaining questions.
Best regards, The authors
Response to Reviewer AzAH (1/3)
Dear Reviewer AzAH,
Thank you for recognizing the metric analysis and its ablations.
We understand your central concerns to be the motivation and novelty of timestep calibration as continuous DLMs move toward few-step generation, and its applicability across model families. We address these through experiments on distilled FMLM and DiffuSeq (Part 1), concrete examples explaining placement effects (Part 2), and a clearer account of the language-specific findings and calibration design (Part 3).
W1 / Q1. Timestep calibration remains useful after distillation
Distillation and timestep calibration can be combined. Consistency, MeanFlow, rectified-flow, and flow-map methods train or distill the generator, including direct training as in MeanFlow. StepTune adjusts its inference grid with weights, solver, and NFE fixed. As long as inference still uses multiple refinement steps, timestep placement remains a variable, regardless of how the model was trained. Even at low NFE, changing timestep placement may also have a substantial effect on the terminal state and generation quality.
We test this on the released FMLM-B-OWT, distilled from FLM and operating over continuous one-hot states (FMLM). Each of the four settings below uses 3,000 unconditional generations, with the same checkpoint and matched NFE within each comparison.
Relative reductions from Native, with 95% confidence intervals:
The 2-NFE experiment is an extreme few-step illustration: placement still affects generation with only two evaluations. The 16-NFE experiment provides the main quantitative support, with a larger PPL reduction while largely preserving unigram entropy. Both results require no additional training or distillation, directly demonstrating the combination you suggested.
W1 / Q2. Applicability across representations and readouts
Regarding the statement that both models use an autoencoder, AURORA-LM uses a separately trained, frozen autoencoder, whereas ELF shares weights between denoising and final token decoding, without a separate decoder. Thus, the original evaluation covers two distinct architectural designs.
The joint objective uses the model's continuous state z and native token readout q(⋅∣z): terminal-state error against a reference and KL between their readout distributions, with baseline normalization. It does not require a separate autoencoder decoder.
DiffuSeq results cover the full 2,500-example official QQP test set, with the same frozen checkpoint and matched single-decode evaluation. Scores use the 0–100 scale.
StepTune16 exceeds Native2000 on all three metrics, with the following improvements and 95% confidence intervals:
This uses 99.2% fewer denoiser calls (16 versus 2,000). BLEU follows the official NLTK method-4 smoothed sentence-level average. All rows use single decoding rather than the original paper's multi-candidate MBR setting (official implementation).
These results extend the evidence to continuous one-hot and embedding-space models. We will add the results and model-specific calibration protocols to the revision.