acceptodds
Under review as a conference paper at ICLR 2027

A Step in the Right Place: Understanding and Optimizing Timestep Placement in Continuous Diffusion Language Models

17 views

Abstract

Continuous diffusion language models represent an emerging paradigm for language modeling. They generate text in parallel by iteratively refining continuous representations and mapping them back to tokens through a decoder. However, realizing the efficiency advantage of parallel generation requires maintaining generation quality with only a few denoising steps. Even when the number of steps is fixed, their placement along the denoising trajectory remains flexible. In this paper, we demonstrate that changing timestep placement alone can substantially alter the final latent representations and thereby affect the decoded token distributions. Building on these findings, we introduce StepTune, an algorithm that searches for timestep placements that improve generation quality. The search is conducted on a small calibration set and guided by a language-aware objective, with the model and sampling budget fixed. Once calibrated, the timestep placement is fixed and reused across inputs. Experiments demonstrate consistent improvements in unconditional generation perplexity and conditional task metrics across all evaluated sampling budgets. In particular, StepTune enables 16-step sampling with a frozen model to outperform its native 64-step baseline on both XSum summarization and WMT14 translation and reduces end-to-end latency by approximately 3.8×. These results establish timestep placement as a practical means of improving the quality–efficiency trade-off in continuous diffusion language models.

Primary area:
generative models
Keywords:
Continuous Diffusion Language Models, Timestep Scheduling, Few-Step Diffusion Sampling, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AzAHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors apply time-step optimization to continuous-diffusion based language models. They identify how directly applying image-diffusion metrics for this optimization is not optimal for language and propose alternative metrics that improve optimization performance. They experiment across two different latent-space diffusion language models and show gains on metrics such as generative perplexity and bleu score.

Critical strengths

The main innovation of the paper is a detailed analysis of the right metric to optimize for with timestep placement in particular for diffusion-language models. The authors show how naive metrics are suboptimal and conduct ablations to support their claims.

Critical weaknesses

First, the motivation of the paper and the central novelty is not strong enough. While the paper is motivated as an augmentation to existing timestep optimization methods tailored to continuous diffusion language models, the authors do not clarify benefits compared to other methods such as consistency models/meanflow/rectified flow/flow map. This is particularly important as the continuous language community increasingly moves towards few step generation. Moreover, the continuous-diffusion language space is pretty broad but their proposed augmentation seems to be scoped for only a particular class of language models. It may be helpful to answer the questions in the next section to strengthen the paper's motivation and novelty.

Second, the analysis of the paper's method could be stronger. At minimum, some empirical samples with 64 NFE vs 16 NFE vs 16 optimized NFE (all from the same initial gaussian noise) should be shown to allow readers to visually identify the degradation/improvement in quality when timesteps are suboptimal/optimized. Moreover, the actual optimized timesteps should also be shown for one to understand how significant the change in timepoints is. Finally, correlations of optimized timesteps with heuristics for flow curvature/straightness may help build some intuition around why timestep optimization can improve quality.

Finally, the continuous diffusion for language literature has been poorly presented. The overall motivation for continuous diffusion for language instead of the prevalent discrete diffusion and why it is important is weak. This paper is an augmentation on top of continuous diffusion for language and hence it would strengthen the paper's position if the authors can elaborate on why continuous diffusion should be optimized in the first place. Several key citations are missing. For example, several key papers from around 20222 such as diffuseq and variants are missing. The recent wave of continuous diffusion for language began with three key (and concurrent) papers - categorical flow maps, langflow and flow map language models. All of these citations are missing.

Questions for authors
  1. Why is a timestep optimization strategy needed given the increasing adoption of meanflow/flow map for few step distillation? How do gains compare with timestep optimization versus such distillation methods? Can the two methods be combined for compounded gains? (I understand if the authors can't find enough time to empirically conduct these in time for the rebuttals but would appreciate it if such comparisons were added to the final version of the paper)
  2. How would the proposed augmentations to timestep optimization be applied to other continuous diffusion language models, such as FMLM (over one-hot vectors), RePlaid (over learned embeddings) and DiffuSeq (over pre-trained embeddings). Both considered methods in the paper use an autoencoder, but there are multiple variants of continuous diffusion and it is not clear which one is the best. So robustness to the paradigm will help support the case for the paper.
  3. Are there toy examples that can help build intuition over why/when timestep optimization is effective?
3 comments
AuthorsOpenReview ↗

Response to Reviewer AzAH (1/3)

Dear Reviewer AzAH,

Thank you for recognizing the metric analysis and its ablations.

We understand your central concerns to be the motivation and novelty of timestep calibration as continuous DLMs move toward few-step generation, and its applicability across model families. We address these through experiments on distilled FMLM and DiffuSeq (Part 1), concrete examples explaining placement effects (Part 2), and a clearer account of the language-specific findings and calibration design (Part 3).

W1 / Q1. Timestep calibration remains useful after distillation

Distillation and timestep calibration can be combined. Consistency, MeanFlow, rectified-flow, and flow-map methods train or distill the generator, including direct training as in MeanFlow. StepTune adjusts its inference grid with weights, solver, and NFE fixed. As long as inference still uses multiple refinement steps, timestep placement remains a variable, regardless of how the model was trained. Even at low NFE, changing timestep placement may also have a substantial effect on the terminal state and generation quality.

We test this on the released FMLM-B-OWT, distilled from FLM and operating over continuous one-hot states (FMLM). Each of the four settings below uses 3,000 unconditional generations, with the same checkpoint and matched NFE within each comparison.

MethodNFEGenerative PPLUnigram entropy
Native2135.63595.29311
StepTune2120.38085.02009
Native1663.51795.29152
StepTune1649.31195.15347

Relative reductions from Native, with 95% confidence intervals:

NFEPPL reductionEntropy reduction
211.25% [10.46%, 12.03%]5.16% [5.08%, 5.23%]
1622.37% [21.36%, 23.37%]2.61% [2.47%, 2.75%]

The 2-NFE experiment is an extreme few-step illustration: placement still affects generation with only two evaluations. The 16-NFE experiment provides the main quantitative support, with a larger PPL reduction while largely preserving unigram entropy. Both results require no additional training or distillation, directly demonstrating the combination you suggested.

W1 / Q2. Applicability across representations and readouts

Regarding the statement that both models use an autoencoder, AURORA-LM uses a separately trained, frozen autoencoder, whereas ELF shares weights between denoising and final token decoding, without a separate decoder. Thus, the original evaluation covers two distinct architectural designs.

The joint objective uses the model's continuous state zz and native token readout q(⋅∣z)q(\cdot\mid z): terminal-state error against a reference and KL between their readout distributions, with baseline normalization. It does not require a separate autoencoder decoder.

  • FMLM: continuous one-hot states and its native two-time-conditioned readout.
  • DiffuSeq: the official QQP checkpoint, embedding-space respaced DDIM sampler, and native token-probability head. We preserve source-token anchoring and count actual denoiser calls. Calibration uses the full FF-and-QQ selection rule.
  • RePlaid: the formulation would use its learned embeddings and native token readout under its own sampler. Public pretrained weights are currently unavailable, so we leave implementation and empirical evaluation to future work.

DiffuSeq results cover the full 2,500-example official QQP test set, with the same frozen checkpoint and matched single-decode evaluation. Scores use the 0–100 scale.

MethodNFEBLEUROUGE-LBERTScore F1
Native200015.284448.254375.4090
Native1615.146147.479573.8097
StepTune1617.998353.023076.2959

StepTune16 exceeds Native2000 on all three metrics, with the following improvements and 95% confidence intervals:

MetricImprovement95% confidence interval
BLEU+2.7138[+2.2639, +3.1639]
ROUGE-L+4.7687[+4.2703, +5.2823]
BERTScore F1+0.8869[+0.5775, +1.2019]

This uses 99.2% fewer denoiser calls (16 versus 2,000). BLEU follows the official NLTK method-4 smoothed sentence-level average. All rows use single decoding rather than the original paper's multi-candidate MBR setting (official implementation).

These results extend the evidence to continuous one-hot and embedding-space models. We will add the results and model-specific calibration protocols to the revision.

AuthorsOpenReview ↗

Response to Reviewer AzAH (2/3)

W2. Paired generation examples

For ELF, we compare Native16, StepTune16, and Native64 using the same input and initial Gaussian noise. The examples below illustrate changes in fluency and source fidelity.

WMT14. The source states that a traffic-light system cost approximately EUR 15,000 and quotes Arnold describing it as the most modern system currently available.

MethodGenerated text
Native16The traffic lights system cost about 15 000 euros itself is the "most modern thing there is in the market today", Arnold explained.
StepTune16The traffic lights system itself, which costs around EUR 15 000, is the "most modern thing on the market currently", explained Arnold.
Native64The traffic lights system which costs about EUR 15 000 itself is the "modernest thing in the market now", Arnold explained.

StepTune produces a grammatical relative clause and avoids Native64's malformed superlative.

XSum. The source reports a fatal collision involving two motorbikes, the death of a 19-year-old rider, and the arrest of a 29-year-old man.

MethodGenerated text
Native16A man has been arrested after a teenage motorcyclist collided to the death with two bikes.
StepTune16A man has been arrested after a teenager was killed in a collision involving two motorbikes.
Native64A man has been arrested after a teenage cyclist was killed in a collision between two motorbike vehicles.

StepTune expresses the supported events fluently, improving Native16's collision description and avoiding Native64's substitution of "cyclist." We have included these selected examples and additional matched cases in the supplementary materials.

W2. Optimized timesteps

Appendix C summarizes the 11 deployed schedules in revised Table 27 (original Table 6); the reproducibility package submitted with the original paper contains their exact grids. For clarity, the XSum16 grid is reproduced below, rounded to six decimals:

[0.000000, 0.004751, 0.009502, 0.014253, 0.019004,
 0.023755, 0.028506, 0.033257, 0.038008, 0.042759,
 0.047510, 0.052261, 0.057012, 0.061763, 0.068221,
 0.229057, 1.000000]

Its four regions have boundaries approximately (0,0.061763,0.068221,0.229057,1)(0,0.061763,0.068221,0.229057,1) and interval allocations (13,1,1,1)(13,1,1,1). Compared with Uniform16, tk=k/16t_k=k/16, it allocates substantially more resolution to the beginning of the trajectory. Appendix C of the original submission analyzes this pattern across tasks and sampling budgets.

W2 / Q3. From timestep placement to token probabilities

Consider fixed dynamics x˙=x\dot x=x, y˙=t2\dot y=t^2, with initial state (1,0)(1,0) and two Euler evaluations over [0,1][0,1]. The grids (0,1/2,1)(0,1/2,1) and (0,2/3,1)(0,2/3,1) yield endpoints (9/4,1/8)(9/4,1/8) and (20/9,4/27)(20/9,4/27), respectively; the exact endpoint is (e,1/3)(\mathrm e,1/3). The dynamics, initialization, total time, and NFE are identical.

Apply the same illustrative two-token readout to both endpoints:

q(A∣x,y)=sigmoid⁡(40(y−0.14)),q(B∣x,y)=1−q(A∣x,y).q(A\mid x,y)=\operatorname{sigmoid}(40(y-0.14)),\qquad q(B\mid x,y)=1-q(A\mid x,y).

The uniform grid gives q(A)=0.354q(A)=0.354; the nonuniform grid gives q(A)=0.581q(A)=0.581. The selected token therefore changes from B to A, matching the token preferred at the exact endpoint. Although the nonuniform endpoint has larger Euclidean latent error (0.52950 versus 0.51253), it has smaller forward KL from the exact-endpoint readout (0.53972 versus 1.03340).

This toy example illustrates Appendix A of the original submission. First, placement changes the terminal state. Appendix A.2 (Theorem 1) expresses terminal error as a sum of propagated interval defects. Placement changes both the defects and their propagation, explaining the different numerical endpoints even for the straight-line xx component.

Second, decoder geometry determines which endpoint changes matter to tokens. Appendix A.5 (Theorem 3) expresses local KL as a quadratic form weighted by decoder sensitivity; Proposition 5 gives an exact decomposition along a finite latent path. Here, the readout responds to yy but not xx, so a larger latent error can yield a better token distribution. Appendix A.6 connects transported errors and decoder geometry to the joint objective. We have included the full toy calculation in Section 3 of the revised manuscript.

W2. Brief trajectory check

Following your suggestion to examine flow curvature and straightness, we conducted a separate trajectory check using Uniform256 reference trajectories for 16 XSum inputs with two initializations each. The mean path-length-to-chord ratio is 1.005, and the first 1/161/16 of normalized time accounts for 48.5% of the accumulated turning angle.

AuthorsOpenReview ↗

Response to Reviewer AzAH (3/3)

W1. Novelty: systematic study and use of placement in continuous DLMs

We systematically investigate an underexplored aspect of continuous DLMs: how timestep placement shapes terminal representations and decoded token distributions. This analysis informs a language-aware calibration framework that improves generation with the model, solver, and NFE fixed. Our contribution has three connected parts:

Measuring the effect. At fixed model, solver, and NFE, we show that placement alone can substantially alter terminal representations and analyze how these changes affect decoded token distributions (Sec. 3).

Explaining what matters for language. We identify two regimes: latent distance correlates strongly with decoder KL at large deviations, but this relationship weakens near the reference, where the decoder's sensitivity to error direction becomes important. This finding motivates measuring both terminal-state and token-distribution discrepancies.

Using the findings. StepTune combines latent error with actual decoder KL, searches a compressed schedule space or candidate bank under quality constraints, and freezes the selected schedule for reuse across inputs (Secs. 3.3–4).

The original ELF and AURORA experiments evaluate this design. The new FMLM and DiffuSeq results broaden its empirical support; the cross-model geometry analysis below and the toy example in Part 2 further clarify its underlying reasoning.

Direct WMT14→XSum and 32→16-NFE transfers improve every metric over Native in all four tests without a new search (Reviewer 6mEv, Part 4, Q2).

The additional paired analysis also supports the joint objective. For the WMT14 ablation in revised Table 4 (original Table 3), we retain the default case-sensitive SacreBLEU configuration documented in revised Appendix B.5. Full improves BLEU and chrF over A-only in 9/10 prespecified calibration panels, with positive 95% CIs for both mean gains. The detailed statistics appear in our response to Reviewer 6mEv, Part 2, W2/Q3.

W1 / W2. Broader evidence for the language-specific mechanism

We repeat the terminal-readout analysis on AURORA and distilled FMLM, using six scales and 96 schedule-induced directions per model. All three models show the same reversal. Near the reference (β=0.05\beta=0.05), the readout-aware quadratic tracks actual KL better than latent distance (pooled Pearson 0.861–0.975 versus 0.081–0.250). At the full gap (β=1\beta=1), latent distance is more predictive (0.832–0.976 versus 0.009–0.147 for the quadratic).

This supports the language-specific finding across distinct readouts, including FMLM without a separate decoder. StepTune uses actual KL; the quadratic diagnoses local geometry. Full correlations are reported in revised Appendix B.8, Table 20. Construction and aggregation details are also discussed in our response to Reviewer 3m1w, Part 2 (W2/Q2 and the cross-model analysis).

W3. Positioning within continuous language modeling

Thank you for suggesting a broader discussion of continuous language modeling. We have expanded Section 2 (Related Work) of the revised manuscript to cover early embedding-space diffusion and recent continuous one-hot/simplex and learned-embedding approaches, including Diffusion-LM, DiffuSeq, Categorical Flow Maps, LangFlow, FMLM, and RePlaid. Diffusion-LM, DiffuSeq, Categorical Flow Maps.

Continuous language modeling is worth optimizing because it already supports competitive generation and offers effective routes to few-step inference. LangFlow and RePlaid demonstrate competitive continuous approaches, while FMLM and Categorical Flow Maps demonstrate the potential of continuous flow maps for few-step generation. Our work asks how to improve the use of such models after training, through their finite inference grid. LangFlow, RePlaid, FMLM.

Thank you again for the constructive suggestions. They have helped us broaden the empirical support and make the paper's motivation and scope more explicit. We hope these results address your concerns and would appreciate your updated assessment in light of the additional evidence. We welcome further discussion of any remaining questions.

Best regards, The authors

Reviewer_3m1wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

I read this paper as a study of how timestep placement affects generation quality in continuous diffusion language models when the number of sampling steps is fixed. The authors examine changes in terminal latent representations and decoded token distributions, finding that latent distance alone does not adequately capture differences in token probabilities near a reference representation. This observation motivates StepTune, which searches for timestep placements using both latent error and decoder KL divergence. The search uses a small calibration set and quality constraints, after which the selected placement is reused across inputs without updating the model. The paper reports that, on ELF, 16-step sampling exceeds the native 64-step baseline on XSum summarization and WMT14 translation while achieving an approximately 3.8x speedup. It also reports lower perplexity in unconditional generation and improved generation quality on AURORA-LM-S.

Critical strengths

S1. I find the improvements in quality and latency from changing timestep placement without updating the model to be a substantial strength. On ELF, the reported 16-step StepTune results exceed the native 64-step baseline on all reported XSum and WMT14 quality metrics. For example, XSum ROUGE-2 increases from 11.878 to 12.589, and WMT14 BLEU increases from 25.5353 to 26.8390. The measured latency corresponds to speedups of 3.82x and 3.84x on the two tasks. The selected schedule can also be reused across inputs without model updates or per-example search, and the authors also report improvements on AURORA-LM-S for XSum.

S2. I find the comparison between latent distance and decoder KL useful for understanding the choice of calibration objective. In the XSum perturbation experiment, the authors evaluate the decoder along 96 terminal-error directions using a 64-step reference. At perturbation scales of 0.05 and 0.1, the Pearson correlation between normalized squared latent distance and decoder KL drops to about 0.11, while the decoder-aware quadratic approximation achieves correlations of 0.975 and 0.925. For the perturbations tested here, this suggests to me that latent distance alone misses differences in token probabilities that decoder KL captures.

Critical weaknesses

W1. I find it difficult to assess the robustness of the main quality improvements in Tables 1 and 2 because they don't report uncertainty. This is particularly relevant to the AURORA-LM-S XSum comparison, where ROUGE-2 increases from 13.479 to 13.635, and the table reports only the mean across five runs. The reported means favor StepTune, but I cannot tell how consistently it outperforms the native schedule across runs.

W2. I find the Figure 4 setup insufficiently clear to judge how broadly its correlation results apply. Section 3.2 describes 96 terminal-error directions from XSum schedules, but does not explain how these are distributed across inputs and schedules or how the correlations are aggregated. Since this analysis motivates the calibration objective, these details matter for assessing whether the observed pattern holds across inputs.

Questions for authors

Q1. In Section 3.1, which nonuniform timestep placement is used for Figure 3, and how are the timesteps interpolated as \lambda changes? The reported terminal displacement seems to depend on this choice, so a brief specification would help me interpret its magnitude.

Q2. In Section 3.2, how are the 96 terminal-error directions distributed across XSum inputs and timestep schedules? Are the correlations in Figure 4 computed separately for each input and then averaged, or computed after pooling the observations across inputs?

Q3. Could the authors report the variability across the five runs for the AURORA-LM-S XSum results in Table 2(b)? The mean ROUGE scores favor 16-step StepTune over the native 32-step schedule, but I cannot tell from the table how consistent this improvement is across runs.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3m1w (1/2)

Dear Reviewer 3m1w,

Thank you for your careful review and for recognizing the practical quality–latency gains and the value of the latent-distance versus decoder-KL analysis.

We understand your main concerns to be the reliability of the quality gains and how broadly the diagnostic findings hold across inputs. Part 1 provides paired uncertainty estimates and the exact Figure 3 schedule. Part 2 details the Figure 4 direction construction and aggregation, tests the finding within individual inputs, and extends the evidence across models.

W1 / Q3. Reliability of the quality improvements

For the AURORA-LM-S XSum comparison in Table 2(b), StepTune at 16 NFE improves ROUGE-1 in four of five paired runs and ROUGE-2 and ROUGE-L in all five, compared with the native 32-NFE schedule, while halving the number of model evaluations. Against the native 16-NFE schedule, it improves all three ROUGE metrics in every run.

We recompute statistics from the original five XSum generation runs (1,000 generations per run, with one frozen StepTune schedule shared across runs). We report each method's sample standard deviation (SD) and paired differences for StepTune16 minus Native32 on the 0–100 ROUGE scale.

MetricNative32 SDStepTune16 SDMean differenceRuns with improvement95% paired CI
ROUGE-10.1330.226+0.1724/5[−0.071, +0.415]
ROUGE-20.3260.358+0.1555/5[+0.035, +0.276]
ROUGE-L0.2230.291+0.2085/5[−0.015, +0.431]

These are two-sided paired t intervals over the five run-level differences (4 degrees of freedom). The ROUGE-2 gain is positive in every run and its interval excludes zero. The ROUGE-1 and ROUGE-L intervals span zero.

For ELF, we regenerated native and StepTune outputs at NFE 16, 32, and 64 on the complete WMT14 and XSum test sets (3,003 and 11,334 examples), keeping the deployed StepTune schedules unchanged. The estimates below come from these fresh outputs, explaining the small numerical differences from Table 1. WMT14 BLEU retains ELF's lowercase SacreBLEU corpus configuration for comparability with Table 1 and ELF; the settings and rationale are documented in Appendix B.5, "WMT14 BLEU conventions and aligned rescoring," of the revised manuscript. Using 8,192 paired bootstrap resamples with common example indices for both methods, StepTune16 minus Native64 gives:

TaskMetricDifference95% paired bootstrap CI
WMT14BLEU+1.284[+0.980, +1.626]
WMT14chrF+0.974[+0.735, +1.221]
XSumROUGE-1+0.578[+0.454, +0.701]
XSumROUGE-2+0.730[+0.629, +0.830]
XSumROUGE-L+0.749[+0.640, +0.855]

All five intervals are strictly positive, supporting the quality improvement with one-quarter of the model evaluations. Corpus BLEU and chrF are recomputed on each resample. These intervals measure uncertainty over test examples conditional on the frozen schedules and generated outputs; the AURORA analysis separately measures variation across runs. Confidence intervals for Native and StepTune on ELF conditional generation at NFE 16/32/64 are provided in revised Appendix B.7, Tables 16–18.

Run-level CIs for ELF unconditional generation at 8/16/32 NFE and AURORA at 16 NFE are reported in our response to Reviewer 6mEv, Part 5, W3/Q4.

Q1. Exact schedule and interpolation in Figure 3

The target is the frozen ELF-B XSum 16-NFE schedule used by StepTune. Its four regions have boundaries

(0,;0.0617626682,;0.0682209507,;0.2290565073,;1)(0,\\;0.0617626682,\\;0.0682209507,\\;0.2290565073,\\;1)

and interval allocations (13,1,1,1)(13,1,1,1), with equally spaced boundaries within each region. We linearly interpolate corresponding time boundaries from Uniform16 to this target:

tk(λ)=(1−λ)k16+λtktarget,k=0,…,16,t_k(\lambda)=(1-\lambda)\frac{k}{16}+\lambda t_k^{\mathrm{target}},\qquad k=0,\ldots,16,

for λ∈0,0.05,…,1\lambda\in\\{0,0.05,\ldots,1\\}. The endpoints, strict ordering, and 16-NFE budget are preserved. For each of the 128 XSum validation inputs, initial noise, model, Euler solver, conditioning, and guidance settings remain identical throughout the interpolation.

The relative displacement is

Dx(λ)=∣Zx(λ)−Zx(0)∣2∣Zx(0)∣2,D_x(\lambda)=\frac{\\|Z_x(\lambda)-Z_x(0)\\|_2}{\\|Z_x(0)\\|_2},

computed over the 64 generated positions. The reported 79.6% is the median of Dx(1)D_x(1) across inputs. The target grid and interpolation are detailed in revised Appendix B.8, referenced in Section 3.1.

AuthorsOpenReview ↗

Response to Reviewer 3m1w (2/2)

W2 / Q2. Construction of the 96 directions and robustness to aggregation

The original Figure 4 uses 16 XSum inputs × 2 initialization seeds × 3 fixed 16-NFE schedules = 96 terminal-error directions. The schedules are Uniform16 and two fixed nonuniform grids, shared across inputs. For input xx, seed ss, and schedule jj, the direction is

ex,s,j=Zx,s,j16−Zx,s64,∗,e_{x,s,j}=Z^{16}_{x,s,j}-Z^{64,*}_{x,s},

where the uniform 64-NFE reference uses the same input and initialization. We evaluate the decoder at Zx,s64,∗+βex,s,jZ^{64,*}_{x,s}+\beta e_{x,s,j} for the six original scales β∈0.05,0.10,0.25,0.50,0.75,1.00\beta\in\\{0.05,0.10,0.25,0.50,0.75,1.00\\}. At β=1\beta=1, this is the actual 16-NFE endpoint; smaller scales probe local geometry along the same schedule-induced error direction.

The original figure pools all 96 observations at each scale to compute Pearson and Spearman correlations. It does not average per-input correlations. To directly test whether the pattern holds across inputs, we additionally compute both correlations separately within each input, using its six directions:

  • At β=0.05\beta=0.05 and 0.100.10, the decoder-aware quadratic has higher correlation with actual KL than latent distance, under both Pearson and Spearman, in all 16 inputs.
  • At β=0.75\beta=0.75 and 1.001.00, latent distance has higher correlation with actual KL under both measures, again in all 16 inputs.

Thus, the reversal between the local and finite-gap regimes is reproduced within individual inputs as well as in the pooled analysis. We will add the input–seed–schedule breakdown, the aggregation rule, and these within-input results to the experimental description.

Additional evidence across AURORA and distilled FMLM

To further assess the scope of this finding, we repeat the diagnostic at the same six scales on two additional models. AURORA uses 16 XSum inputs × 2 initializations × 3 fixed schedules. Unconditional FMLM uses 32 initializations × 3 fixed schedules. Each experiment therefore provides 96 directions. Representative pooled Pearson correlations with actual readout KL are:

ModelQuadratic, β=0.05Latent distance, β=0.05Quadratic, β=1.00Latent distance, β=1.00
ELF0.9750.1090.1180.976
AURORA0.9090.2500.1470.832
FMLM0.8610.0810.0090.832

Here, “Quadratic” denotes the decoder/readout-aware Fisher quadratic approximation to KL. In both additional models, it tracks KL more closely near the reference, while latent distance is more predictive at the full terminal gap. The same qualitative reversal appears in all three models.

For FMLM, the diagnostic uses its native continuous state and token-probability readout. We keep the final readout time conditioning fixed to isolate the effect of state perturbations, and use the official 32-step grid for the reference.

These results strengthen the empirical motivation for accounting for both latent error and token-distribution divergence. The quadratic approximation diagnoses the local geometry; StepTune's calibration objective evaluates actual KL. We will incorporate the additional-model results and their protocols into the revision.

Broader generation evidence

We also extend generation evaluation to two additional model families:

  • Distilled FMLM-B-OWT: over 3,000 unconditional generations per setting, 16-NFE calibration reduces PPL from 63.5179 to 49.3119 (22.37%, 95% CI [21.36%, 23.37%]), while largely preserving unigram entropy. This demonstrates an additional benefit after distillation.
  • Embedding-space DiffuSeq: on the full 2,500-example QQP test set, StepTune16 exceeds Native2000 by 2.7138 BLEU, 4.7687 ROUGE-L, and 0.8869 BERTScore F1, with positive 95% CIs for all three gains under a matched single-decode protocol.

Both evaluations keep the official model weights fixed. Together with the cross-model mechanism analysis above, they broaden the practical and explanatory support for timestep calibration. Full generation tables, confidence intervals, and model-specific protocols appear in our response to Reviewer AzAH, Part 1, W1/Q1–Q2.

Direct WMT14→XSum and 32→16-NFE transfers improve every metric over Native in all four tests without a new search (Reviewer 6mEv, Part 4, Q2).

We appreciate your suggestions, which have helped us make the statistical evidence and mechanism analysis more complete. We hope the clarifications and additional results address your concerns, and would appreciate your updated assessment of the paper's empirical support and broader contribution. We welcome further discussion.

Best regards, The authors

Thank you for the detailed responses and additional experiments. I appreciate the effort to address my concerns.

  • The additional statistical results help clarify the reliability of the reported improvements.
  • The clarification of the Figure 4 setup also addresses my concern.
  • The additional results on AURORA and FMLM further strengthen the empirical evidence for the observed relationship between latent distance and decoder KL.
  • The explanation of the timestep schedule and interpolation used in Figure 3 is also sufficient.

I think the authors have adequately addressed my original concerns. Since my original rating was already positive, I will maintain it.

Dear Reviewer 3m1w,

Thank you for your thoughtful follow-up. We are glad the additional statistical analyses and the clarifications for Figures 3 and 4 addressed your concerns. Your feedback helped us present the experimental setup and the reliability of our results more clearly in the revised manuscript.

Sincerely, The Authors

Reviewer_6mEvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies inference timestep placement in continuous diffusion LMs (ELF, AURORA) at a fixed NFE. It shows that placement alone moves the terminal latents (median 79.6% relative displacement). It also shows that latent distance tracks decoder KL far from the reference but not near it (Pearson 0.11, versus 0.975 for a decoder-aware approximation). This motivates a joint objective F_α, which combines latent error and decoder KL.

Critical strengths
  • Useful analysis tied to the method. The two-regime finding (latent distance vs. KL) motivates F_α. Frozen schedules cut both latent error (59.8–81.5%) and decoder KL (71.6–78.2%) on unseen validation trajectories across ELF-B/M/L.
  • Practical and cheap. It needs no model updates and calibration costs 0.57–6.79 GPU-minutes. Gains hold on full test sets and on two architectures, and the schedules are released.
  • Careful protocol. Calibration, validation and test data are disjoint. It uses bootstrap CIs, quality gates with a fallback to the baseline, and a search-space ablation (Mixed6 reaches the target in 20 queries vs. 153 for Full15).
Critical weaknesses
  • Weak or untuned baselines. The native baseline is an untuned LogitNormal that redraws its grid per batch. StepTune is tuned per task and per NFE using task-metric gates on calibration data. There is no simple tuned baseline, such as a sweep of the LogitNormal (μ, σ) on the same calibration set with the same gates. The LD3 adaptation uses only ℓ2 and drops LPIPS. The gain at NFE 64 (BLEU 25.5→27.1) and 16 NFE beating 64 NFE suggest part of the gain comes from a poor native schedule, not from reducing discretization error. The “3.8× speedup” is just the 64/16 NFE ratio.
  • The joint objective is weakly supported. In Table 3, Full vs. A-only differs by ~0.36 BLEU, with Full’s std at 0.38 (A-only 0.15). This was run on one task with 16-example panels, and α=½ is never ablated. The correlation study uses artificial perturbations βe, not schedule-induced errors, and covers one model and task. The Appendix A theory is mostly standard and is not used in the search.
  • Limited scope and evaluation. Only ELF and AURORA are tested, and the conditional tasks use ELF-B with 64-token outputs. Metrics are ROUGE/BLEU and GPT-2 perplexity. Entropy drops slightly on ELF (5.157→5.111), there is no human or LLM evaluation, and MAUVE appears only for AURORA. AURORA gains are small (ROUGE-1 +0.17) with no CIs. Table 1 and Table 2 report no variance. Each model, task and NFE needs its own calibration, and cross-scale transfer is shown only for F, not for task quality.
Questions for authors
  • (Clarification) How does StepTune compare to a tuned LogitNormal (μ, σ) baseline, or to Pure-Q search, on the main tasks (XSum, WMT14) with the same calibration data and gates?
  • (Clarification) Do schedules transfer across tasks or NFEs (e.g., a WMT14 schedule used on XSum)? Does a schedule calibrated on ELF-M keep its task-quality gains on ELF-B and ELF-L?
  • (Clarification) Is Full vs. A-only statistically significant? How sensitive is the result to α and to calibration size?
  • (Clarification) Can you report CIs for Tables 1–2 and add diversity metrics (e.g., self-BLEU) or an LLM judge for unconditional generation?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 6mEv (1/5)

Dear Reviewer 6mEv,

Thank you for recognizing the analysis and practical calibration.

We understand your main concern to be the source and robustness of the gains: the roles of the schedule space and joint objective, and generalization across settings and models. Parts 1–2 address the calibration controls and paired objective evidence; Part 3 covers the mechanism and model coverage; Part 4 reports task/NFE transfer and conditional-generation uncertainty; Part 5 presents unconditional-generation statistics, diversity, and text assessment.

W1 / Q1. Timestep calibration within the StepTune framework

Motivation and calibration framework.

You suggest that part of the gain comes from a "poor native schedule." Indeed, the headroom in the native schedule is precisely what motivates our study. We systematically investigate how timestep placement can improve generation quality at fixed model weights, solver, and NFE.

StepTune combines a language-aware signal, quality constraints, calibration on a small set, and a frozen schedule reused across inputs; search is one component of this framework.

Candidate-bank comparison.

Regarding your requested (μ,σ)(\mu,\sigma) comparison, the original submission did not include this scan for XSum and WMT14. The finite-bank approach itself is explicit in Section 4 and was used for ELF unconditional generation with 80 native-family grids (Appendix B.4). We now evaluate this formulation on both conditional tasks and will include these comparisons in the revised manuscript.

Our bank crosses seven μ\mu values from −3.0 to 0.0 with seven σ\sigma values from 0.2 to 1.6, including ELF's native (μ,σ)=(−1.5,0.8)(\mu,\sigma)=(-1.5,0.8) configuration. Three seeds, shared across the 49 settings, yield 147 candidate grids per NFE, varying timestep location, dispersion, and stochastic realization. We use the same calibration panels and language-quality gates as regional search (16 examples for WMT14; 64 for XSum), with matched initial noise and unchanged model and sampler settings. The 147 evaluations are comparable to the regional search's 160-query budget. Revised Appendix B.5, "LogitNormal candidate bank," provides the parameter values and grid construction.

WMT14 BLEU conventions. We follow the two protocols documented in Appendix B.5, "WMT14 BLEU conventions and aligned rescoring," of the revised manuscript. The bank/Pure-Q comparisons in Part 1 and the transfer and main-result statistics in Part 4 use ELF's lowercase SacreBLEU corpus configuration (lowercase=True, use_effective_order=True) for comparability with Table 1 and ELF. The objective ablation and sensitivity analyses in Part 2 retain the default case-sensitive configuration (both options False), following revised Table 4 (original Table 3) for these within-experiment comparisons. The appendix also reports aligned rescoring of the original ablation, with its mean-score ordering unchanged.

WMT14 scores are BLEU/chrF; XSum scores are ROUGE-1/2/L, all on the 0–100 scale. The regional column contains Table 1 results from the original manuscript; Native and bank scores are from the additional comparison. Tests contain all 3,003 WMT14 and 11,334 XSum examples.

TaskNFENativeLogitNormal bankStepTune regional
WMT141622.993 / 50.97926.490 / 53.48326.839 / 53.688
WMT143224.691 / 52.16426.643 / 53.52727.092 / 53.792
WMT146425.529 / 52.70526.760 / 53.66827.127 / 53.853
XSum1634.625 / 11.002 / 26.34536.194 / 12.472 / 27.85636.304 / 12.589 / 28.015
XSum3235.353 / 11.617 / 26.99036.262 / 12.518 / 27.90636.577 / 12.947 / 28.263
XSum6435.720 / 11.851 / 27.26536.399 / 12.579 / 27.98536.607 / 12.988 / 28.277

Gains from timestep placement.

ELF's native LogitNormal family supplies an informed prior for the bank. The bank's gains support the motivation above, while regional allocation further improves every reported metric at a comparable budget. Both are candidate-space options within the original framework.

Pure-Q comparison.

The original ablation compares Pure-Q and Full across ten WMT14 calibration panels (Section 5.3; revised Table 4, original Table 3). Building on this evidence, our additional comparison for Q1 focuses on NFE–16 on WMT14 and XSum. We match the six-dimensional space, calibration examples, 160-query budget, and quality gates; Pure-Q uses task quality in acquisition and selection. Full-test results, with the StepTune column retaining the original Table 1 values, are:

TaskPure-QStepTune
WMT1426.350 / 53.34326.839 / 53.688
XSum36.105 / 12.408 / 27.82736.304 / 12.589 / 28.015

StepTune improves every reported metric, supporting the original ablation's conclusion.

AuthorsOpenReview ↗

Response to Reviewer 6mEv (2/5)

W1 / Q1, continued. Generation quality and the role of FF

Regarding your distinction between improving the native schedule and reducing discretization error, our goal is generation quality (Section 3.3). A high-NFE reference provides a useful quality anchor, making its neighborhood in terminal-state and token-distribution space a promising region for calibration. FF guides search toward this region; the original Pure-F ablation supports its usefulness (revised Table 4, original Table 3).

Generation quality need not improve monotonically with NFE or numerical accuracy, nor be maximized at the exact solution of the learned dynamics. We therefore combine FF with measured quality QQ: quality constraints and near-best-QQ filtering determine eligible candidates, and FF selects within that set (Section 4).

LD3 adaptation.

Regarding the omitted LPIPS term, LPIPS is defined on image features and does not directly apply to language-model embeddings. Our adaptation therefore matches the student's terminal latent representations to the high-NFE teacher using ℓ2\ell_2 distance over generated tokens, preserving conditioning and the generation mask (Appendix B.5). This retains LD3's reference-matching principle in the representation space used by the language model.

Measured speedup.

Regarding the comment that the speedup is "just the 64/16 NFE ratio," Table 1 reports measured end-to-end latency. On a B200 at batch size one, Native64 versus StepTune16 takes 541.11 vs. 141.07 ms on WMT14 (3.84×) and 543.15 vs. 142.20 ms on XSum (3.82×). Better placement enables higher quality at this lower measured inference cost.

W2 / Q3. Paired evidence for the objective and sensitivity to calibration choices

Full versus A-only. We analyze the same ten prespecified calibration panels as revised Table 4 (original Table 3), retaining its default case-sensitive BLEU convention. Full improves both BLEU and chrF in 9/10 panels. The mean paired gains are +0.360 BLEU and +0.266 chrF, with paired-difference SDs of 0.347 and 0.232. Pairing accounts for the fact that both methods use the same panel; the marginal SD of Full alone does not measure uncertainty in their difference.

MetricMean Full − A-only95% paired CITwo-sided pHolm-adjusted p
BLEU+0.360[+0.112, +0.608]0.00950.0111
chrF+0.266[+0.100, +0.432]0.00550.0111

These are two-sided paired t intervals and tests (9 degrees of freedom), with Holm correction across the two tests. Both gains remain significant at 0.05 after correction.

Mixture weight. We rerun the full method with Fα=αA/Ascale+(1−α)K/KscaleF_\alpha=\alpha A/A_{\mathrm{scale}}+(1-\alpha)K/K_{\mathrm{scale}} at α∈0.25,0.50,0.75\alpha\in\\{0.25,0.50,0.75\\} on the same ten panels. Every frozen schedule is evaluated on all 3,003 WMT14 test examples; values are mean ± sample SD.

αBLEUchrF
0.2524.618 ± 0.49352.841 ± 0.322
0.5024.721 ± 0.38352.918 ± 0.247
0.7524.402 ± 0.42452.693 ± 0.270

The reported mean scores vary by 0.319 BLEU and 0.225 chrF across these weights. The default equal weighting achieves the highest mean, with nearby choices producing similar scores.

Calibration size. A controlled nested-panel experiment uses m∈4,8,16,32,64m\in\\{4,8,16,32,64\\}, holding the six-dimensional space, 160-query budget, teacher, initialization, and search seed fixed. The resulting schedules are evaluated on the full WMT14 test set:

Calibration examplesBLEUchrF
424.99753.077
825.12953.302
1625.52753.521
3225.70953.669
6425.63553.626

Quality improves as the calibration set grows from 4 to 32 examples, with similar scores at 64. This is one nested-panel comparison in the six-dimensional space; revised Table 4 (original Table 3) averages ten panels in Region3.

AuthorsOpenReview ↗

Response to Reviewer 6mEv (3/5)

W2. Schedule-induced directions, cross-model geometry, and the role of theory

The directions in Figure 4 come from actual terminal differences between a 16-NFE schedule and its matched 64-NFE reference, sharing the input and initial noise. Scaling by β\beta probes the decoder along that segment: β=1\beta=1 is the actual student endpoint, while smaller values examine its local geometry near the reference. The original six scales are 0.05,0.10,0.25,0.50,0.75,1.00\\{0.05,0.10,0.25,0.50,0.75,1.00\\}.

For ELF, the 96 directions comprise 16 inputs × 2 initializations × 3 schedules. Computing correlations separately within each input, the Fisher approximation exceeds latent distance in both Pearson and Spearman correlation for all 16 inputs at β=0.05,0.10\beta=0.05,0.10; the ordering reverses for every input at β=1\beta=1.

We repeat the diagnostic on AURORA (16 inputs × 2 initializations × 3 schedules) and FMLM (32 initializations × 3 schedules), using each model's native token readout. FMLM's readout time conditioning is held fixed. Pooled Pearson correlations with actual readout KL are:

ModelQuadratic, β=0.05Latent distance, β=0.05Quadratic, β=1.00Latent distance, β=1.00
ELF0.9750.1090.1180.976
AURORA0.9090.2500.1470.832
FMLM0.8610.0810.0090.832

Here, "Quadratic" denotes the readout-aware Fisher approximation to KL. The reversal appears in all three models. The full direction-construction and aggregation protocol is given in our response to Reviewer 3m1w, Part 2, W2/Q2 and the cross-model analysis.

Appendix A connects this observation to calibration: A.2, Theorem 1, describes how interval defects propagate to the terminal state; A.5, Theorem 3 and Proposition 5, relate terminal changes to readout KL locally and along a finite path. A.6 connects these quantities to the joint objective. StepTune evaluates terminal latent error and actual decoder KL; the quadratic approximation diagnoses the local geometry. These analyses explain the complementary signals used in the search.

W3. Generation evidence beyond ELF and AURORA

To broaden the model coverage requested in W3, we add distilled continuous one-hot FMLM and embedding-space DiffuSeq, keeping their official weights fixed.

FMLM-B-OWT at 16 NFE. With 3,000 generations per method, StepTune reduces PPL by 22.37% (95% CI [21.36%, 23.37%]), while largely preserving unigram entropy.

MethodGenerative PPLUnigram entropy
Native1663.51795.29152
StepTune1649.31195.15347

DiffuSeq. On all 2,500 QQP test examples, we use full FF-and-QQ calibration and matched single decoding. Scores are on the 0–100 scale.

MethodNFEBLEUROUGE-LBERTScore F1
Native200015.284448.254375.4090
Native1615.146147.479573.8097
StepTune1617.998353.023076.2959

StepTune16 exceeds Native2000 on all three metrics, by 2.7138 BLEU, 4.7687 ROUGE-L, and 0.8869 BERTScore F1, with positive paired 95% CIs for all three gains. BERTScore adds semantic similarity evidence beyond lexical overlap.

Detailed confidence intervals and model-specific protocols, including the 2-NFE FMLM experiment, appear in our response to Reviewer AzAH, Part 1, W1/Q1–Q2. We will include these results in the revision.

Q2. What transfers, and what is calibrated separately?

Task-specific checkpoints. The official ELF release provides ELF-B checkpoints for XSum and WMT14, with no ELF-M/L checkpoints for these tasks. This is why our conditional experiments use ELF-B and its official 64-token output setting. The released M/L checkpoints are unconditional models.

Generalization across trajectories. Section 3.3 asks whether the joint objective's gains persist beyond calibration. At each NFE (16/32), we calibrate on 16 ELF-M trajectories, freeze the schedule, and evaluate it on 160 unseen trajectories per model in ELF-B/M/L (Appendix B.3). Both latent error and decoder KL decrease across the six settings. This supports timestep placement as a reusable control variable whose benefits extend beyond the calibration samples.

Transfer within the ELF family. The same ELF-M schedule also works on ELF-B/L without recalibration, suggesting shared schedule preferences within the family. These results measure teacher-relative fidelity. The new task/NFE transfer experiments in Part 4 directly evaluate generation quality. Our main benchmarks use schedules calibrated for each checkpoint, task, solver, and NFE, with a one-time cost of 0.57–6.79 GPU-minutes, followed by reuse across inputs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We thank all three reviewers for their constructive feedback and for recognizing the value of our metric analysis, practical quality–efficiency gains, and careful calibration protocol. StepTune connects a systematic study of timestep placement in continuous DLMs with a language-aware calibration framework whose schedules can be reused across inputs. We have strengthened the evidence in three directions that address the reviewers’ central concerns.

1. Broader applicability, including after distillation.

The new FMLM and DiffuSeq experiments extend the evaluation to four model families, using fixed official checkpoints. On distilled FMLM, calibration reduces generative PPL by 22.37% at 16 NFE, while largely preserving unigram entropy. On DiffuSeq, 16-NFE StepTune exceeds native 2,000-NFE sampling on BLEU, ROUGE-L, and BERTScore on the full QQP test set. These results demonstrate additional gains after distillation and broaden the evidence across representations and readouts. See our responses to AzAH, Q1–Q2, and 6mEv, W3.

We also test direct WMT14→XSum transfer at 16/32 NFE and 32→16-NFE transfer within both tasks. All four settings improve every metric over Native without a new search, while target-specific calibration gives the highest scores. See 6mEv, Part 4, Q2.

2. Stronger evidence for the language-specific finding.

We extend the latent-distance/readout-KL analysis from ELF to AURORA and distilled FMLM. All three models exhibit the same qualitative reversal: the readout-aware quadratic approximation tracks KL more closely near the reference, whereas latent distance is more predictive at the full terminal gap. The ELF finding also holds in within-input analyses. This supports accounting for both terminal-state and token-distribution discrepancies in calibration. Paired text examples, optimized grids, and a toy example linked to Appendix A further illustrate how placement affects decoded tokens. See our responses to AzAH, W2/Q3, and 3m1w, W2/Q2.

3. Controlled comparisons and statistical reliability.

The added LogitNormal parameter scan instantiates the candidate-bank option in the original framework and produces substantial gains on ELF; regional allocation further improves every reported metric at a comparable query budget. Matched Pure-Q comparisons on both conditional tasks also favor the full method. On the original ten calibration panels, Full improves both BLEU and chrF over latent-only calibration in 9/10 panels, with positive 95% paired confidence intervals for both mean gains.

In fresh ELF generations with unchanged schedules, all five 95% paired bootstrap intervals for StepTune16 minus Native64 are positive across WMT14 and XSum. We also provide five-run AURORA statistics and sensitivity analyses for calibration-set size and objective weight. See our responses to 6mEv, Q1/Q3/Q4, and 3m1w, W1/Q3.

We will incorporate these results, the accompanying analyses, and the clarified protocols into the revised manuscript and supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.