acceptodds
Under review as a conference paper at ICLR 2027

Calibrating Noise Schedules for Continuous Language Diffusion with Token Uncertainty

Abstract

Continuous language diffusion models use noise schedules to allocate training capacity and guide inference across varying corruption levels. Existing schedules often adapt heuristics from image diffusion, rely on costly denoiser queries, or assume uniform token distributions, overlooking how embedding geometry and token frequency affect recoverability under Gaussian noise. We present **kNN Information Scheduling** (KIS), an approach that bases noise scheduling on token frequencies and distances between embeddings without querying a denoiser. By modeling token uncertainty under Gaussian noise, KIS tracks conditional entropy across log signal to noise ratios (log-SNRs) and expresses its decrease as a normalized entropy reduction. Instead of replacing schedules without constraints, KIS applies ***constrained calibration***: it shifts the training distribution's median while keeping its standard deviation fixed and applies a positive affine correction to the interior sampling levels in log-SNR. Across various architectures (ELF, FLM) and datasets, KIS significantly accelerates inference and enhances generation quality. On FLM-B, KIS improves generative perplexity by up to 25.6% on OpenWebText and 47.1% on LM1B under limited budgets. Notably, a calibrated FLM at 256 steps outperforms its uncalibrated version at 1024 steps, using four times fewer sampling steps. Ablation studies confirm that affine calibration outperforms direct quantile placement at 8–32 steps in the reported comparisons, offering key insights into the relationship between discrete priors, continuous representations, and diffusion dynamics.

Primary area:
generative models
Keywords:
Continuous Language Diffusion, Noise Scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Sx98OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KIS, which uses a corpus unigram prior and fixed token-embedding geometry to compute the conditional entropy of a single-token Gaussian corruption channel, then normalizes entropy reduction into an information coordinate. During training, KIS shifts the median of the noise distribution to the information midpoint; during sampling, it fits interior log-SNR nodes to information quantiles using a positive affine transformation. The method does not query the denoiser. Experiments cover pretraining calibration for ELF-B and sampling calibration for frozen ELF, FLM, and LangFlow models. Several settings improve substantially—for example, ELF-B's LM1B GenPPL drops from 121.2 to 90.1—but some ELF-L and LangFlow results regress.

Critical strengths
  1. KIS can be applied before training or to a frozen model at sampling time without repeated denoiser evaluations. Sampling calibration preserves monotonicity and uses a constrained transformation of the original grid.
  2. The experiments cover both training and inference, several model representations, and different sampling budgets. The paper reports substantial GenPPL improvements in low-step settings such as FLM-B.
Critical weaknesses
  1. H(Y∣Zλ)H(Y\mid Z_\lambda) is computed from a fixed-embedding, single-token Gaussian channel and does not include context. For ELF, calibration additionally relies on a T5-small lookup-embedding proxy with a global RMS scale adjustment; this does not establish that the proxy matches the information curve of the actual contextual latents.
  2. Improvements are not universal: some ELF-L and LangFlow results regress. The main metrics are GPT-2 Large GenPPL and unigram token entropy, and the main results do not provide enough seed or confidence-interval information to assess stability.
Questions for authors
  1. For ELF, can you quantify how the T5 lookup-embedding proxy relates to predictive uncertainty or denoising loss on actual contextual latents? This would affect my assessment of the central proxy assumption.
  2. How many independent random seeds were used for each training and sampling comparison? Is the corpus token prior estimated only from the training split?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Sx98 (1/4)

Q1 — Quantifying the T5 Lookup Proxy vs. Contextual Latents in ELF

Reviewer Sx98 (Q1): For ELF, can you quantify how the T5 lookup-embedding proxy relates to predictive uncertainty or denoising loss on actual contextual latents? This would affect my assessment of the central proxy assumption.

Answer: Static T5 lookup captures the slope of the denoiser's SNR-weighted loss profile, and contextual centroids close the remaining scale offset.

  • Post-hoc denoising-loss target: Following Gaussian I-MMSE [3], we measure empirical latent difficulty via τdenoiser(λ)\tau_{\mathrm{denoiser}}(\lambda), integrating the trained ELF-B [2] denoiser's SNR-weighted MSE fθ(λ)=12exp⁡(λ)MSE⁡θ(exp⁡(λ))f_\theta(\lambda) = \frac{1}{2}\exp(\lambda)\operatorname{MSE} _ \theta(\exp(\lambda)) across 49 noise levels on 512 validation sequences (Qden(1/2)=−3.02,−2.91,−2.84Q_{\mathrm{den}}(1/2) = -3.02, -2.91, -2.84 across λc∈{2,4,6}\lambda_{\mathrm{c}} \in \lbrace 2, 4, 6 \rbrace) rather than discrete H(Yi∣Zλ,i,context)H(Y_i \mid Z_{\lambda,i}, \text{context}).
  • Lookup vs. contextual centroids (Overall Sec. 3, Table 6 & Table 10 below): Static lookup KIS matches b∗b^{\ast} (0.650.65 vs. 0.670.67), cuts midpoint error by 43%, and recovers 74% of the N=128N=128 LM1B GenPPL reduction (121.2→90.1121.2 \to 90.1). Under xi=eˉyi+δix_i = \bar e_{y_i} + \delta_i (r=0.80,rproto=0.50,sδ≈0.62r=0.80, r_{\mathrm{proto}}=0.50, s_\delta \approx 0.62), SNReff(λ)=eλrproto2/(1+eλsδ2)\mathrm{SNR} _ {\mathrm{eff}}(\lambda) = e^\lambda r_{\mathrm{proto}}^2 / (1 + e^\lambda s_\delta^2) has eλsδ2≈0.024≪1e^\lambda s_\delta^2 \approx 0.024 \ll 1 near λ≈−2.8\lambda \approx -2.8, explaining why unit-RMS lookup (−4.18-4.18) rescaled by rproto=0.50r_{\mathrm{proto}}=0.50 gives −4.18−2log⁡0.50=−2.79-4.18 - 2\log 0.50 = -2.79 vs. measured −2.78-2.78. Contextual centroids and M=4M=4 mixtures match τdenoiser\tau_{\mathrm{denoiser}}'s training GenPPL (79.379.3 and 78.578.5 vs. 79.079.0) and lower GenPPL over an online EMA schedule (80.480.4) across all five sampling step counts NN from step 0.
Calibration ReferenceTraining Schedule VariantPmeanP_{\mathrm{mean}}PstdP_{\mathrm{std}}N=4N=4 GenPPLN=8N=8 GenPPLN=16N=16 GenPPLN=32N=32 GenPPLN=128N=128 GenPPL
None, uncalibratedDefault schedule−1.500-1.5000.8000.800189.6180.9146.7113.4121.2
Online denoiser MSEOnline adaptive center via EMA lossonline0.8000.800183.1155.4121.094.680.4
Top-level contextual centroidsKIS: shift center only−0.700-0.7000.8000.800181.8153.2119.393.879.3
Top-level contextual centroidsShift center, narrow spread−0.700-0.7000.6000.600190.1164.6129.6100.185.4
Top-level contextual centroidsDirect KIS density qKISq_{\mathrm{KIS}}, unweightedn/an/a266.8202.3148.6114.2122.8
Top-level contextual centroidsDirect KIS + IS weight, q0/qKISq_0 / q_{\mathrm{KIS}} reweightn/an/a192.4161.4126.897.982.1
Lookup table proxyKIS: shift center only−1.174-1.1740.8000.800183.4178.5129.3104.190.1
Lookup table proxyShift center, narrow spread−1.174-1.1740.5550.555185.4196.9171.6139.6112.4

↳ Manuscript updates: Sec. 4.4 (Table 6), App. E.2 (Table 10, Fig. 5), and App. E.3 (Fig. 7).

AuthorsOpenReview ↗

Response to Reviewer Sx98 (2/4)

Q2 — ELF-L Algorithm 2 Audit & LangFlow Scope Boundary

Reviewer Sx98 (Q2): Improvements are not universal: how do the ELF-L and LangFlow results affect the main conclusions?

Answer: Applying Algorithm 2 uniformly lowers ELF-L GenPPL at both small and full step budgets, while LangFlow reflects train-test schedule shift when overriding a co-trained schedule post hoc.

  • ELF-L implementation audit: We audited all five pretrained models across LM1B and OWT (Tables 2, 4–6, 8, 9, 12): FLM-B [4], ELF-B/M [2], and LangFlow [1] already used affine-fit (using direct-quantiles only in w/o linear fit ablations), whereas ELF-L's evaluation config had used direct-quantiles (unified configs are in the supplementary code). Applying Algorithm 2 to ELF-L lowers GenPPL at both small (−15.6-15.6 and −18.4-18.4 at N=4N=4) and full step budgets (shown alongside raw quantiles below), and matched ELF-L training also lowers GenPPL across all four settings:
ELF-L Frozen-Sampling ScheduleLM1B (N=128N=128) GenPPLOWT (N=64N=64) GenPPL
Default schedule108.923.3
Direct quantiles Q(i/N)Q(i/N) (initial submission)110.324.1
Positive affine fit (a∗,b∗)(a^{\ast}, b^{\ast}) (Algorithm 2, corrected)107.122.8
  • LangFlow post-hoc boundary: Online schedules cannot calibrate frozen fixed-schedule checkpoints post hoc and incur early-training warmup lag (80.480.4 GenPPL for online EMA vs. 79.379.3 for KIS in Table 10); on LangFlow [1], whose velocity network already co-adapted with its Gumbel schedule during training, post-hoc replacement shifts the midpoint 0.65–0.700.65\text{--}0.70 log-SNR noisier relative to the training distribution:
ModelCorpusMode and Step Budget NNDefault GenPPLKIS GenPPLΔGenPPL\Delta\mathrm{GenPPL}
ELF-L without encoderLM1BMatched training decoded at N=128N=128109.1104.3−4.8-4.8
ELF-L with encoderLM1BMatched training decoded at N=128N=128108.9104.8−4.1-4.1
ELF-L without encoderOWTMatched training decoded at N=32N=3230.929.2−1.7-1.7
ELF-L with encoderOWTMatched training decoded at N=32N=3231.128.8−2.3-2.3
ELF-L with encoderLM1BPaired frozen sampling at small budget N=4N=4180.8165.2−15.6-15.6
ELF-L with encoderLM1BPaired frozen sampling at full budget N=128N=128108.9107.1−1.8-1.8
ELF-L with encoderOWTPaired frozen sampling at small budget N=4N=4194.8176.4−18.4-18.4
ELF-L with encoderOWTPaired frozen sampling at full budget N=64N=6423.322.8−0.5-0.5
LangFlow [1]LM1BPaired frozen sampling at small budget N=8N=8238.4244.1+5.7+5.7
LangFlow [1]LM1BPaired frozen sampling at full budget N=128N=12892.294.1+1.9+1.9
LangFlow [1]OWTPaired frozen sampling at small budget N=8N=8192.5198.2+5.7+5.7
LangFlow [1]OWTPaired frozen sampling at full budget N=1024N=102436.537.3+0.8+0.8

↳ Manuscript updates: Sec. 1, Secs. 4.1–4.2, Sec. 5, and App. D.3 (Table 8).

AuthorsOpenReview ↗

Response to Reviewer Sx98 (3/4)

Q3 — Diversity Metrics, Temperature Matching, & Llama-3-8B Evaluation

Reviewer Sx98 (Q3): Scope of GPT-2 Large GenPPL and unigram token entropy HtokH_{\mathrm{tok}} as quality and diversity metrics.

Answer: GenPPL reductions persist after matching corpus-level diversity via logit temperature and under independent Llama-3-8B evaluation.

  • Training & ELF sampling diversity: In Table 9 below, KIS training on ELF-B LM1B lowers GenPPL while moving token and nn-gram diversity closer to held-out LM1B text, and across ELF-B/M/L sampling Distinct-2/3\text{Distinct-2/3} and H4-gramH_{4\text{-gram}} remain within 0.010.01 of default.
  • Diversity-matched frontier & Llama-3-8B: On FLM-B [4] OWT sampling, matching baseline diversity at categorical logit temperature T=1.03T=1.03 lowers both GPT-2 Large (62.2→48.962.2 \to 48.9) and Llama-3-8B (48.4→39.648.4 \to 39.6) GenPPL, and Llama-3-8B preserves all OWT rankings.

↳ Manuscript updates: Secs. 4.1–4.2 and App. D.3 (Table 9).

Model, Corpus, and Evaluation ModeGenPPL GPT-2 LargeGenPPL Llama-3-8BHtokH_{\mathrm{tok}} natsDistinct-2Distinct-3H4-gramH_{4\text{-gram}} nats
Held-out LM1B reference text, T4=64kT_4=64\text{k}n/an/a4.30.620.8810.96
Held-out OWT reference text, T4=523kT_4=523\text{k}n/an/a5.30.480.8112.93
ELF-B on LM1B Training, N=128N=128, Table 1 Default →\to KIS121.2 →\to 79.3n/a3.4 →\to 3.60.52 →\to 0.550.78 →\to 0.8110.74 →\to 10.80
FLM-B on LM1B Training, N=1024N=1024, Table 1 Default →\to KIS96.9 →\to 92.5n/a4.3 →\to 4.30.61 →\to 0.610.87 →\to 0.8710.92 →\to 10.92
FLM-B on LM1B Sampling, N=1024N=1024, Default →\to KIS98.0 →\to 94.0n/a4.3 →\to 4.30.61 →\to 0.610.87 →\to 0.8710.92 →\to 10.91
FLM-B on OWT Sampling, N=1024N=1024, Default →\to KIS at T=1.00T=1.0062.2 →\to 46.348.4 →\to 37.95.3 →\to 5.10.47 →\to 0.450.80 →\to 0.7812.89 →\to 12.85
FLM-B on OWT Sampling, N=1024N=1024, KIS at T=1.02T=1.0262.2 →\to 47.948.4 →\to 39.05.3 →\to 5.20.47 →\to 0.460.80 →\to 0.7912.89 →\to 12.88
FLM-B on OWT Sampling, N=1024N=1024, KIS diversity-matched (T=1.03T=1.03)62.2 →\to 48.948.4 →\to 39.65.3 →\to 5.30.47 →\to 0.470.80 →\to 0.8012.89 →\to 12.89
FLM-B on OWT Sampling, N=1024N=1024, KIS at T=1.05T=1.0562.2 →\to 51.448.4 →\to 41.25.3 →\to 5.40.47 →\to 0.480.80 →\to 0.8112.89 →\to 12.91
ELF-B on LM1B Sampling, N=128N=128, Default →\to KIS121.2 →\to 115.1n/a3.4 →\to 3.40.52 →\to 0.520.78 →\to 0.7810.74 →\to 10.74
ELF-B on OWT Sampling, N=32N=32, Default →\to KIS23.6 →\to 21.821.4 →\to 19.85.2 →\to 5.20.46 →\to 0.460.79 →\to 0.7912.88 →\to 12.87
ELF-M on OWT Sampling, N=64N=64, Default →\to KIS21.7 →\to 20.619.9 →\to 19.05.2 →\to 5.20.46 →\to 0.460.79 →\to 0.7912.88 →\to 12.88
ELF-L on OWT Sampling, N=64N=64, Default →\to KIS23.3 →\to 22.821.1 →\to 20.75.3 →\to 5.30.47 →\to 0.470.80 →\to 0.8012.89 →\to 12.89
LangFlow on OWT Sampling, N=1024N=1024, Default →\to KIS36.5 →\to 37.331.2 →\to 31.85.3 →\to 5.20.47 →\to 0.460.80 →\to 0.7912.89 →\to 12.87
Reviewer_X8odOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies an overlooked issue on noise schedule on continuous language diffusion models: how embedding geometry and token frequency affect recoverability under Gaussian noise? It proposes kNN Information Scheduling (KIS) to design noise schedule. Experiments across various architectures (ELF, FLM) and datasets show that KIS significantly accelerates inference and enhances generation quality.

Critical strengths

This paper identifies overlooked factors that affect noise schedule: embedding geometry and token frequency, and proposed kNN Information Scheduling (KIS) that bases noise scheduling on token frequencies and distances between embeddings without querying a denoiser. KIS significantly accelerates inference and enhances generation quality across various architectures (ELF, FLM) and datasets.

Critical weaknesses
  1. KIS doesn't show superior performance compared to other global noise schedule methods. For instance, in Table 2, LangFlow is better than KIS.

  2. The paper does not present the results obtained from simple hyperparameter tuning. For instance, in Eq. (9), would simple grid search for a and b achieve the same or better performance?

  3. Whether the difficulty curve calculated by KIS accurately represents the difficulty of the actual model requires further verification. Generally, the exact meaning of a word depends on its context, if the context provides enough information, a certain could also be identified.

Questions for authors

Why is KIS inferior to LangFlow’s original scheduling?

4 comments
AuthorsOpenReview ↗

Response to Reviewer X8od (1/3)

We thank Reviewer X8od for your constructive review and questions on cross-model performance, grid searches over aa,bb and PmeanP_{mean}, how KIS relates to contextual model difficulty, and why post-hoc replacement raises GenPPL on LangFlow. Below we address each question directly.

Q1 — Cross-Model Comparisons vs. Same-Checkpoint Calibration in Table 2

Reviewer X8od (Q1): KIS does not show superior performance compared to all global noise schedule methods; for instance, in Table 2, LangFlow is better than KIS.

Answer: Table 2 evaluates same-checkpoint post-hoc calibration rather than cross-architecture rankings: KIS lowers GenPPL across all fixed-schedule checkpoints, whereas overriding LangFlow's co-trained schedule at inference shifts away from its training distribution.

  • Why online schedules do not replace KIS: Online schedules cannot calibrate already-trained fixed-schedule models post hoc (FLM [3], ELF [2]) and incur early-training warmup lag during pretraining, where step-0 KIS lowers GenPPL over an online EMA baseline across all five sampling step counts NN (79.379.3 vs. 80.480.4 in Table 10).
  • Same-checkpoint paired results: Averaged over 3 paired 512-sample seeds on released FLM-B, LangFlow, and OWT ELF checkpoints and our Table 1 default LM1B ELF checkpoints (as Hu et al. [2] release only OWT checkpoints), KIS lowers GenPPL across every FLM and ELF setting below, while LangFlow [1] was already co-trained with its Gumbel schedule.

↳ Manuscript updates: Sec. 1, Sec. 4.2 (Table 2), Sec. 5, and App. D.2–D.3 (Table 8).

ModelCorpusNative Schedule DesignSmall NNSmall NN Default →\to KIS GenPPLLarge NNLarge NN Default →\to KIS GenPPL
FLM-B [3]LM1BAnalytical uniform argmax error8252.9 →\to 133.7102498.0 →\to 94.0
FLM-B [3]OWTAnalytical uniform argmax error8205.8 →\to 153.1102462.2 →\to 46.3
ELF-B [2]LM1BHeuristic logit-normal4189.6 →\to 166.4128121.2 →\to 115.1
ELF-M [2]LM1BHeuristic logit-normal4180.1 →\to 156.4128112.1 →\to 108.8
ELF-L [2]LM1BHeuristic logit-normal4180.8 →\to 165.2128108.9 →\to 107.1
ELF-B [2]OWTHeuristic logit-normal4138.7 →\to 116.43223.6 →\to 21.8
ELF-M [2]OWTHeuristic logit-normal4297.1 →\to 230.56421.7 →\to 20.6
ELF-L [2]OWTHeuristic logit-normal4194.8 →\to 176.46423.3 →\to 22.8
LangFlow [1]LM1BLearned Gumbel from denoiser8238.4 →\to 244.112892.2 →\to 94.1
LangFlow [1]OWTLearned Gumbel from denoiser8192.5 →\to 198.2102436.5 →\to 37.3
AuthorsOpenReview ↗

Response to Reviewer X8od (2/3)

Q2 — Validation Grid Search over the (a,b)(a, b) Affine Family & Pre-Training Sweep

Reviewer X8od (Q2): The paper does not present results from simple hyperparameter tuning; in Equation 9, would a simple grid search for aa and bb achieve the same or better performance?

Answer: Proposition 1 derives the 2-parameter step-ratio-preserving affine family (a,b)(a, b) (Equation 9) that underpins this grid search; closed-form KIS beats a 9-rollout 3×3 grid and achieves GenPPL within 0.2–0.8 points (less than 1% relative difference) of the best 25-rollout 5×5 grid results without validation rollouts.

  • Grid search builds on our parameterization: Proposition 1 establishes that an affine map (a,b)(a, b) on interior log-SNR nodes preserves solver step ratios while adjusting location and scale; arbitrary rescaling such as b=1.50b=1.50 raises ELF-B GenPPL above the default schedule (23.6→24.423.6 \to 24.4).
  • Zero-rollout closed form vs. 25-rollout grid: We benchmark λi′=m0+Δm+b(xi−m0)\lambda_i' = m_0 + \Delta m + b(x_i - m_0) over Δm∈[−1.2,+1.2]\Delta m \in [-1.2, +1.2] and b∈[0.50,1.50]b \in [0.50, 1.50] on 256 validation samples (~50 min per NN for 5×55 \times 5 on one A100). Without validation rollouts, closed-form KIS beats the 9-rollout 3×33 \times 3 grid, achieves GenPPL within 0.2–0.80.2\text{--}0.8 points (0.4%–0.9% relative difference across all four settings) of the best 25-rollout 5×55 \times 5 grid, and analytically predicts the opposite shift directions across architectures: noisier on FLM-B (Δm=−0.6\Delta m = -0.6 from skewed token priors) vs. cleaner on ELF-B (Δm=+0.6\Delta m = +0.6 from embedding RMS scale):
Schedule Calibration MethodValidation Sweeps / NNCenter & Slope m,bm, b on FLM-B LM1BCenter & Slope m,bm, b on ELF-B OWTFLM-B LM1B N=8N=8FLM-B LM1B N=32N=32ELF-B OWT N=8N=8ELF-B OWT N=32N=32
Default schedule, a=0,b=1a=0, b=10−3.02,1.00-3.02, 1.00−4.39,1.00-4.39, 1.00252.9153.154.423.6
Fixed midpoint Q(1/2)Q(1/2), unscaled b=1.00b=1.000−3.68,1.00-3.68, 1.00−3.82,1.00-3.82, 1.00146.2107.248.322.5
Fixed midpoint Q(1/2)Q(1/2), widened b=1.50b=1.500−3.68,1.50-3.68, 1.50−3.82,1.50-3.82, 1.50178.5121.052.924.4
Coarse 3×33 \times 3 validation grid search9−3.02,0.50-3.02, 0.50 [Δm=0.0\Delta m = 0.0]−3.19,1.00-3.19, 1.00 [Δm=+1.2\Delta m = +1.2]136.4104.947.222.1
Dense 5×55 \times 5 validation grid search25−3.62,0.75-3.62, 0.75 [Δm=−0.6\Delta m = -0.6]−3.79,0.75-3.79, 0.75 [Δm=+0.6\Delta m = +0.6]132.9102.846.321.6
KIS closed-form fit (a∗,b∗)(a^{\ast}, b^{\ast}) via Equation 90mfit=−3.64,b∗=0.73m_{\mathrm{fit}}=-3.64, b^{\ast}=0.73 [Q1/2=−3.68Q_{1/2}=-3.68]mfit=−3.86,b∗=0.65m_{\mathrm{fit}}=-3.86, b^{\ast}=0.65 [Q1/2=−3.82Q_{1/2}=-3.82]133.7103.246.521.8
  • Eliminating 8x pretraining sweeps: Before training, each grid point costs a full pretraining run; across Table 11's 8-run PmeanP_{\mathrm{mean}} sweep on ELF-B LM1B below, offline KIS lands directly inside the empirical optimum [−0.800,−0.600][-0.800, -0.600] (79.379.3 vs. 79.8–80.179.8\text{--}80.1 GenPPL) before step 0.

↳ Manuscript updates: Sec. 4.3 (Table 5), App. D.3, and App. E.2 (Table 11).

ELF-B LM1B Training PmeanP_{\mathrm{mean}}−1.500-1.500 Default−1.174-1.174 Lookup KIS−0.900-0.900 Grid−0.800-0.800 Grid−0.700-0.700 Contextual KIS−0.600-0.600 Grid−0.400-0.400 Grid0.0000.000 Grid
Induced Log-SNR Median λ1/2\lambda_{1/2}−4.39-4.39−3.73-3.73−3.19-3.19−2.99-2.99−2.79\mathbf{-2.79}−2.59-2.59−2.19-2.19−1.39-1.39
Test GenPPL at N=32N=32113.4104.196.494.593.894.297.5118.2
Test GenPPL at N=128N=128121.290.182.480.179.379.884.1109.5
AuthorsOpenReview ↗

Response to Reviewer X8od (3/3)

Q3 — Does KIS Track Actual Denoising Difficulty Under Context?

Reviewer X8od (Q3): Whether KIS's difficulty curve represents actual model difficulty when token identification depends on context.

Answer: Not in the sense of discrete contextual token entropy; measured against the trained denoiser's SNR-weighted loss profile, contextual-centroid KIS tracks actual denoising difficulty closely.

  • Continuous denoising-loss target: We measure empirical denoising difficulty via τdenoiser(λ)\tau_{\mathrm{denoiser}}(\lambda), integrating ELF-B's continuous-latent loss 12eλMSE⁡θ\frac{1}{2}e^\lambda\operatorname{MSE} _ \theta across 49 noise levels on 512 validation sequences (Qden(1/2)=−3.02,−2.91,−2.84Q_{\mathrm{den}}(1/2) = -3.02, -2.91, -2.84 across λc∈{2,4,6}\lambda_{\mathrm{c}} \in \lbrace 2, 4, 6 \rbrace). Whereas discrete H(Yi∣Zλ,i,context)H(Y_i \mid Z_{\lambda,i}, \text{context}) would shift noisier under context disambiguation, τdenoiser(λ)\tau_{\mathrm{denoiser}}(\lambda) sits cleaner than static lookup (−3.73-3.73) because continuous MSE reconstructs contextual residuals δi\delta_i and incurs finite-network regression error, remaining consistent across default (−2.84-2.84) and KIS (−2.81-2.81) ELF-B and matching KIS on one-hot FLM-B (−3.59-3.59 vs. −3.68-3.68).
  • Empirical match (Overall Sec. 3, Table 6): Static lookup already matches b∗b^{\ast} (0.650.65 vs. 0.670.67), cuts midpoint error by 43%, and recovers 74% of the N=128N=128 LM1B GenPPL reduction. Under xi=eˉyi+δix_i = \bar e_{y_i} + \delta_i (r=0.80,rproto=0.50,sδ≈0.62r=0.80, r_{\mathrm{proto}}=0.50, s_\delta \approx 0.62), SNReff(λ)=eλrproto2/(1+eλsδ2)\mathrm{SNR} _ {\mathrm{eff}}(\lambda) = e^\lambda r_{\mathrm{proto}}^2 / (1 + e^\lambda s_\delta^2) has eλsδ2≈0.024≪1e^\lambda s_\delta^2 \approx 0.024 \ll 1 near λ≈−2.8\lambda \approx -2.8, explaining the +0.95+0.95 shift (−4.18−2log⁡0.50=−2.79-4.18 - 2\log 0.50 = -2.79 vs. measured −2.78-2.78) so contextual centroids and M=4M=4 mixtures match τdenoiser\tau_{\mathrm{denoiser}}'s training GenPPL offline (79.379.3 and 78.578.5 vs. 79.079.0).

↳ Manuscript updates: Sec. 4.4 (Table 6), App. E.2 (Table 10), and App. E.3.

Q4 — Analysis of Post-Hoc Schedule Substitution on LangFlow

Reviewer X8od (Q4): Why is KIS inferior to LangFlow's original scheduling?

Answer: On a frozen LangFlow checkpoint co-trained with its online Gumbel schedule, single-token KIS sits 0.65–0.70 log-SNR noisier than the co-adapted schedule, creating a train-test schedule shift when substituted post hoc.

  • Ruling out shape mismatch: A Gumbel quantile curve QGumbel(u)=μλ+βlog⁡[−log⁡(1−u)]Q_{\mathrm{Gumbel}}(u) = \mu_\lambda + \beta\log[-\log(1-u)] fits KIS on LangFlow's embedding table within 0.17%–0.23% NRMSE (Figure 3), ruling out functional-form mismatch.
  • Origin of the 0.65–0.70 offset: LangFlow diffuses non-contextual embeddings (δi=0\delta_i = 0) and fits its schedule online to sequence cross-entropy: across λc∈{2,4,6}\lambda_{\mathrm{c}} \in \lbrace 2, 4, 6 \rbrace, τdenoiser(λ)\tau_{\mathrm{denoiser}}(\lambda) on LangFlow LM1B has midpoints −3.21,−3.09,−3.02-3.21, -3.09, -3.02 (−3.12-3.12 at λc=6\lambda_{\mathrm{c}} = 6 on OWT), matching LangFlow's learned medians (−3.06,−3.17-3.06, -3.17), whereas single-token Bayes KIS gives −3.71-3.71 and −3.87-3.87 due to eλe^\lambda-weighted clean-side neural error, sequence context, and online co-adaptation:
CorpusLangFlow μλ,LF,βLF\mu_{\lambda,\mathrm{LF}}, \beta_{\mathrm{LF}} [Log-NSR μ\mu]LangFlow QLF(1/2)Q_{\mathrm{LF}}(1/2)Static KIS μλ,KIS,βKIS\mu_{\lambda,\mathrm{KIS}}, \beta_{\mathrm{KIS}} [Log-NSR μ\mu]Static KIS QKIS(1/2)Q_{\mathrm{KIS}}(1/2)Fit NRMSEN=8N=8 Default →\to KISFull NN Default →\to KIS
LM1B−2.65,1.12-2.65, 1.12 [μ=+2.65\mu = +2.65]−3.06-3.06−3.38,0.89-3.38, 0.89 [μ=+3.38\mu = +3.38]−3.71-3.71 [−3.7060-3.7060]0.17%238.4 →\to 244.192.2 →\to 94.1
OWT−2.74,1.18-2.74, 1.18 [μ=+2.74\mu = +2.74]−3.17-3.17−3.53,0.92-3.53, 0.92 [μ=+3.53\mu = +3.53]−3.87-3.87 [−3.8685-3.8685]0.23%192.5 →\to 198.236.5 →\to 37.3

Replacing only the location parameter post hoc accounts for most of this GenPPL shift (92.2→93.692.2 \to 93.6 vs. 92.892.8 scale-only and 94.194.1 both on LM1B). Conversely, when training from scratch, online learners incur warmup lag while KIS centers updates at step 0 (79.379.3 vs. 80.480.4 GenPPL vs. online EMA in Table 10), motivating Section 5's step-0 KIS initialization for online tracking.

↳ Manuscript updates: Secs. 4.2, 5, App. A.2, and App. B.2 (Eqs. 22–27, Fig. 3).

References

  • [1] Y. Chen, C. Liang, H. Sui, R. Guo, C. Cheng, J. You, and G. Liu. LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling. arXiv:2604.11748, 2026.
  • [2] K. Hu, L. Qiu, Y. Lu, H. Zhao, T. Li, Y. Kim, J. Andreas, and K. He. ELF: Embedded Language Flows. arXiv:2605.10938, 2026.
  • [3] C. Lee, J. Yoo, M. Agarwal, S. Shah, J. Huang, A. Raghunathan, S. Hong, N. M. Boffi, and J. Kim. Flow Map Language Models: One-step Language Modeling via Continuous Denoising. arXiv:2602.16813, 2026.
AuthorsOpenReview ↗

Brief Followup Regarding Our Author Response

Dear Reviewer X8od,

With only a short time left in the discussion stage, we wanted to reach out and see how you feel about our responses to your review. We hope our clarifications [and new results] address your questions, and we remain at your disposal should you need any further details. Thank you again for your effort in reviewing our work!

Reviewer_vQxNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a mechanism (KIS) for adapting the diffusion denoising schedule of a continuous text diffusion model to unigram token frequencies and the structure of a fixed embedding space. It studies this mechanism under three types of models: ELF (pretrained fixed, contextualized embeddings), FLM (one-hot embeddings), and LangFlow (end-to-end learned embeddings). And it considers two settings using the KIS schedule: training (ELF) and inference (ELF, FLM, LangFlow).

Critical strengths

The performance improvements using for the contextualized embedding model (ELF) is interesting. The setup seems a bit weird, since KIS uses the frozen T5 lookup embeddings rather than top-level contextualized embeddings (this seems that is necessary to fit the KIS protocol, but also clearly not optimal). Nevertheless, it is not clear in general how to properly train contextualized text diffusion end-to-end in general, so any experimentation and new ideas in this space are welcome.

The post-hoc inference time acceleration using KIS to correct models trained with poorly calibrated noise schedules is nice, but the impact is unclear when well-calibrated noise schedules can be learned end-to-end.

Critical weaknesses

This paper fails to engage with the literature on learned noise schedules. In particular, Plaid (Gulrajani and Hashimoto, 2023) and RePlaid (Yang et al., 2026). These papers optimize the noise schedule end-to-end via the variational lower bound. They go beyond the proposals discussed in this paper, directly addressing the question discussed as future scope of adapting the noise schedule to "updating representation geometry as the training schedule evolves."

The paper cites Plaid twice in passing, but only as an example of a continuous score model. The fact that Plaid learns a noise schedule is overlooked.

The Yang RePlaid paper in particular operates on the same I-MMSE principle discussed in the present work. See Section 5.2 of Yang, which shows that

(1) The variance-optimal noise schedule makes diffusion loss constant in t (Prop 1) and empirical learned schedule tracks closely with theory (Fig 6b). (2) Under Bayes-optimal denoiser, I-MMSE identity connects constant loss to mutual information I(e; z_t) and shows that it is linear in t (Lemma 3).

KIS’s information coordinate is the CDF of Yang's variance-optimal distribution over log-SNR, computed for the "empirical token prior" (i.e., unigram model) instead of the model itself.

Aside: it's become popular recently to invent a lot of new technical language in papers. For example, an "information coordinate" is not a standard technical term, and moreover, the community already has a name for this quantity: entropic time (Stancevic and Ambrogioni, 2025). The present work cites Stancevic, but chooses to invent a new and less evocative term. Asking readers to digest new technical language when perfectly suitable terms already exist is bad writing.

The Yang paper is mentioned only once in this text, mis-cited in the related work as a text diffusion paper using "contextualized latent representations" (it uses non-contextual embeddings).

Questions for authors

Any thoughts about how to extend these ideas to contextualized embeddings? That seems to me like the most promising direction for extension of these ideas. Plaid/RePlaid doesn't really address it.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vQxN (1/4)

We thank Reviewer vQxN for your insightful feedback on Plaid and RePlaid, the I-MMSE derivation in Section 5.2 of RePlaid, entropic-time terminology, representation taxonomy, and extending KIS to contextualized embeddings. Below we address each question directly.

Q1 — Literature on Learned Noise Schedules (Plaid & RePlaid)

Reviewer vQxN (Q1): Engagement with the literature on learned noise schedules, particularly Plaid and RePlaid, and how Plaid learns its noise schedule via the variational bound and variance reduction.

Answer: We agree and expanded our discussion of Plaid and RePlaid across Sections 1–2 and Appendix C.

Online learned schedules update parameters during training by reusing per-step losses without extra denoiser passes: Plaid [1] learns endpoints via the variational bound and shapes its interior profile via loss-variance reduction, while RePlaid [2] adapts online to keep expected timestep loss constant across noise levels. Post-hoc methods (Entropic Time [5], Align Your Steps [9]) sweep a trained denoiser at inference, whereas KIS computes an offline reference from token frequencies and an embedding table before step 0.

↳ Manuscript updates: Secs. 1–2 and App. C (Table 7 below), removing the inaccurate statement that adaptive schedules require extra denoiser passes.

MethodUncertainty SignalSchedule Rule
FLM [8]Uniform prior top-1 argmax error on the one-hot simplexAnalytical decoding error schedule for training and sampling
ELF [4]Heuristic logit-normal parameters Pmean,PstdP_{\mathrm{mean}}, P_{\mathrm{std}}Fixed logit-normal density for training and quantile grid for sampling
Plaid [1]Training variational bound and per-step loss estimator varianceJointly learned endpoints via variational bound and interior shape via variance reduction
RePlaid [2]Online timestep denoising loss during trainingAdaptive schedule targeting constant expected timestep loss across noise levels
LangFlow [3]Token cross-entropy of a denoiser during trainingLearned Gumbel time distribution and its quantile schedule
InfoNoise [6]Online denoising loss or information rateTraining noise distribution adapted with the objective's loss weight
Entropic Time [5]Conditional entropy change, often estimated from a denoiserSampling schedule with equal estimated entropy increments
Align Your Steps [9]Reverse process approximation errorOffline optimization of finite solver steps
Convex Allocation [10]Cross-noise gradient curvature in the convex regimeTraining noise allocation balancing tail coverage and transition focus
KISEmpirical token prior and fixed token embedding tableShifting training noise median to Q(1/2)Q(1/2); positive affine fit to interior sampling nodes
AuthorsOpenReview ↗

Response to Reviewer vQxN (2/4)

Q2 — I-MMSE Relation to RePlaid & Why Constant-Loss / Constant-Entropy Schedules Are Suboptimal

Reviewer vQxN (Q2): Connection to Section 5.2 of RePlaid (Propositions 1–2 and I-MMSE Lemma 3) under the unigram token prior.

Answer: Under a Bayes-optimal unigram denoiser, our population curve equals the CDF of RePlaid's variance-optimal density by Gaussian I-MMSE; however, directly enforcing constant per-step loss or entropy decay underperforms constrained location-scale calibration.

On Rλ=exp⁡(λ/2)eY+εR_\lambda = \exp(\lambda/2)e_Y + \varepsilon with Y∼πY \sim \pi, let Hπ(λ)=H(Y∣Rλ)H_\pi(\lambda) = H(Y \mid R_\lambda), ΔH=Hπ(λn)−Hπ(λc)\Delta H = H_\pi(\lambda_{\mathrm{n}}) - H_\pi(\lambda_{\mathrm{c}}), and τπ(λ)=[Hπ(λn)−Hπ(λ)]/ΔH\tau_\pi(\lambda) = [H_\pi(\lambda_{\mathrm{n}}) - H_\pi(\lambda)]/\Delta H. By Gaussian I-MMSE [7], −Hπ′(λ)=ddλI(eY;Rλ)=12exp⁡(λ)MMSE⁡π(exp⁡(λ))-H_\pi'(\lambda) = \frac{\mathrm{d}}{\mathrm{d}\lambda} I(e_Y; R_\lambda) = \frac{1}{2}\exp(\lambda)\operatorname{MMSE} _ \pi(\exp(\lambda)). In RePlaid [2], Proposition 1 (Sec. 5.2) minimizes ELBO variance via pθ∗(λ)∝fθ(λ)=12exp⁡(λ)MSE⁡θ(exp⁡(λ))p_\theta^{\ast}(\lambda) \propto f_\theta(\lambda) = \frac{1}{2}\exp(\lambda)\operatorname{MSE} _ \theta(\exp(\lambda)); under MSE⁡θ=MMSE⁡π\operatorname{MSE} _ \theta = \operatorname{MMSE} _ \pi, RePlaid's Lemma 3 (App. G) gives linear mutual-information decay and Proposition 2 (Sec. 5.2) gives linear single-token cross-entropy in t=1−τπ(λ)t = 1 - \tau_\pi(\lambda), so pπ∗(λ)=τπ′(λ)p_\pi^{\ast}(\lambda) = \tau_\pi'(\lambda).

This equivalence highlights where KIS departs from prior schedule rules (Tables 3 and 10 below). RePlaid [2], Entropic Time [5], and FLM [8] directly equalize per-step loss, entropy, or decoding error, whereas our ablations show that unconstrained equalization hurts flow matching and finite-step solvers:

  1. Shifting only the training median beats constant-rate densities: Under velocity and flow-matching weights w(λ)w(\lambda), shifting only the median beats sampling directly from qKIS(λ)q_{\mathrm{KIS}}(\lambda) (Eq. 16, 6σ06\sigma_0 support) both unweighted (122.8122.8 vs. 79.379.3 GenPPL on ELF-B; 98.498.4 vs. 92.592.5 on FLM-B) and with importance weights q0/qKISq_0/q_{\mathrm{KIS}} preserving the original objective up to negligible truncation error (82.182.1 vs. 79.379.3), because retaining tail coverage [10] prevents uncalibrated ELF-B's N=32→128N=32 \to 128 degradation under SDE churn (113.4→121.2113.4 \to 121.2 GenPPL).
  2. Preserving solver step ratios beats raw quantiles at inference: Proposition 1's 1D W2W_2-optimal positive affine fit (a∗,b∗)(a^{\ast}, b^{\ast}) keeps interior solver step ratios invariant, whereas raw constant-entropy quantiles degrade frozen sampling (108.9→110.3108.9 \to 110.3 vs. 107.1107.1 GenPPL on ELF-L).
  3. Offline step-0 calibration & metric-driven extensions: Unlike Plaid/RePlaid's co-trained lookup ELBO, KIS estimates τ^(λ)\widehat{\tau}(\lambda) offline in 95 s before step 0, extends to frozen encoder prototypes eˉy\bar e_y, and lowers GenPPL over a RePlaid-style online EMA-loss baseline across all five sampling step counts NN (79.379.3 vs. 80.480.4 at N=128N=128, App. E.2) by avoiding early-training warmup lag. Treating τ^(λ)\widehat{\tau}(\lambda) as a location-scale reference also motivates optimizing schedules directly for downstream generation metrics.

↳ Manuscript updates: Secs. 2, 3.3, 5, App. A.3 (Eq. 16), App. B.1 (Eqs. 18–21), and App. E.2 (Table 10).

Model and TableTraining Schedule RuleN=4N=4N=8N=8N=16N=16N=32N=32Full NN
FLM-B, N=1024N=1024 (Table 3)Default schedule λ∼q0(λ)\lambda \sim q_0(\lambda)n/a248.4198.5151.196.9
FLM-B, N=1024N=1024 (Table 3)KIS: shift center only, λ′=λ+δ\lambda' = \lambda + \deltan/a214.6172.3134.892.5
FLM-B, N=1024N=1024 (Table 3)Shift center, narrow spread, λ′=a+bλ\lambda' = a + b\lambdan/a229.1184.7143.295.1
FLM-B, N=1024N=1024 (Table 3)Direct KIS density λ∼qKIS\lambda \sim q_{\mathrm{KIS}}, unweightedn/a261.8205.4154.698.4
ELF-B, N=128N=128 (Table 10)Default, Pmean=−1.500,Pstd=0.800P_{\mathrm{mean}}=-1.500, P_{\mathrm{std}}=0.800189.6180.9146.7113.4121.2
ELF-B, N=128N=128 (Table 10)KIS: shift center only, Pmean=−0.700,Pstd=0.800P_{\mathrm{mean}}=-0.700, P_{\mathrm{std}}=0.800181.8153.2119.393.879.3
ELF-B, N=128N=128 (Table 10)Online adaptive center via EMA loss183.1155.4121.094.680.4
ELF-B, N=128N=128 (Table 10)Shift center, narrow spread, Pmean=−0.700,Pstd=0.600P_{\mathrm{mean}}=-0.700, P_{\mathrm{std}}=0.600190.1164.6129.6100.185.4
ELF-B, N=128N=128 (Table 10)Direct KIS density λ∼qKIS\lambda \sim q_{\mathrm{KIS}}, unweighted266.8202.3148.6114.2122.8
ELF-B, N=128N=128 (Table 10)Direct KIS + IS weight, q0/qKISq_0 / q_{\mathrm{KIS}} reweight192.4161.4126.897.982.1
AuthorsOpenReview ↗

Response to Reviewer vQxN (3/4)

Q3 — Adopting "Entropic Time" Terminology

Reviewer vQxN (Q3): Avoiding the new term "information coordinate" when the community already uses "entropic time" in Stancevic, Handke, and Ambrogioni (2025).

Answer: We agree: we removed "information coordinate" throughout and adopted normalized entropy reduction / entropic time.

↳ Manuscript updates: Abstract, Secs. 1–3, and after Eq. 5 in Sec. 3.3, crediting entropic time [5, 6].

Q4 — Representation Taxonomy for RePlaid

Reviewer vQxN (Q4): Correcting the citation of Yang et al. (RePlaid) in Related Work, which uses noncontextual learned embeddings rather than contextualized latent representations.

Answer: We agree and corrected our representation taxonomy.

↳ Manuscript updates: Secs. 1–2, grouping Plaid [1], RePlaid [2], and LangFlow [3] under non-contextual learned token embeddings, distinct from one-hot FLM [8] and contextual encoder latents in ELF [4].

Q5 — Online Adaptation to Evolving Embedding Geometry

Reviewer vQxN (Q5): Acknowledging that jointly learned schedules in Plaid and RePlaid already adapt to evolving embedding geometry during training rather than framing this as an unsolved open problem.

Answer: We agree and revised Sections 1, 2, and 5 to credit Plaid, RePlaid, and LangFlow for online adaptation.

↳ Manuscript updates: Secs. 1–2 note online adaptation without extra denoiser passes as embeddings evolve, and Sec. 5 proposes initializing online tracking at step 0 with KIS.

Q6 — Value of KIS vs. End-to-End Learned Schedules & Expanded Multi-Scale Training

Reviewer vQxN (Q6): Clarifying the value and boundaries of post-hoc KIS calibration when noise schedules can be learned end-to-end, and expanding training calibration beyond ELF-B.

Answer: End-to-end schedules cannot calibrate existing checkpoints post hoc, require training-loop changes, and incur early-training warmup lag, giving offline KIS a clear pre-training and post-hoc role.

  • Post-hoc vs. step-0 trade-offs: Open-weight fixed-schedule models (FLM [8], ELF [4]) cannot use online schedule learners without retraining from scratch, while post-hoc sweeps (τdenoiser\tau_{\mathrm{denoiser}}, 4.24.2 min; Entropic Time [5]; AYS [9]) require an existing checkpoint (79.079.0 vs. 79.379.3 LM1B GenPPL; −8.0-8.0% OWT GenPPL) and cannot calibrate training before step 0.
  • Faster convergence than online EMA: Because online schedules start from noisy early denoiser losses, step-0 KIS lowers GenPPL over online EMA across all five sampling step counts NN at 20k training steps (79.379.3 vs. 80.480.4 at N=128N=128, Table 10) without modifying the training loop or running an 8-run PmeanP_{\mathrm{mean}} sweep (Table 11). On LangFlow [3], substituting KIS at inference tests post-hoc replacement on a checkpoint already co-trained with its Gumbel schedule, creating a train-test schedule shift rather than comparing from-scratch training.
  • 14-pair multi-scale training (Overall Sec. 4, Table 1): Under 1M official LM1B steps for FLM-B, 20k for ELF, and 59k on OWT, KIS lowers GenPPL on FLM-B (96.9→92.596.9 \to 92.5 LM1B; 70.3→62.970.3 \to 62.9 OWT), ELF-B with encoder on LM1B (121.2→79.3121.2 \to 79.3 at 20k; 86.2→71.486.2 \to 71.4 at 60k), and all ELF-B/M/L without-encoder and ELF-M/L with-encoder settings while matching ELF-B OWT (32.0→31.932.0 \to 31.9), taking 95 s (lookup) to 3.5–4.8 min (encoder prototypes).

↳ Manuscript updates: Abstract, Sec. 1, Secs. 4.1–4.4 (Tables 1, 2, 6), Sec. 5, and App. D.2.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Overall Rebuttal (1/4)

We thank Reviewers vQxN, X8od, and Sx98 for their constructive feedback. Below we summarize our new experiments, clarifications, and blue-highlighted manuscript revisions.

1. Offline Calibration vs. Online Learned and Post-Hoc Schedules — Reviewer vQxN Q1, Q3–Q6; Reviewer X8od Q1, Q4; Reviewer Sx98 Q2

Takeaway: KIS calibrates training schedules before step 0 without modifying the training loop and calibrates frozen checkpoints in closed form without validation rollouts.

Online methods (Plaid [1], RePlaid [2], LangFlow [3], InfoNoise [6]) update schedules from per-step losses without extra denoiser passes as embeddings evolve, while post-hoc sweeps (Entropic Time [5], Align Your Steps [9], and τdenoiser\tau_{\mathrm{denoiser}} in Table 6, 4.24.2 min) require a trained checkpoint (79.079.0 LM1B GenPPL; −8.0-8.0% OWT GenPPL at N=32N=32). End-to-end online schedules have three practical limitations:

  • Inapplicable to existing checkpoints: Online learners cannot calibrate already-trained open-weight models post hoc (FLM [8], ELF [4]), whereas KIS calibrates them in closed form without denoiser rollouts.
  • Early-training warmup lag: Online schedules start from noisy losses of an unconverged denoiser; centering training noise at step 0 with KIS lowers GenPPL over an online EMA-loss schedule across all five sampling step counts NN at 20k training steps (Table 10: 79.379.3 vs. 80.480.4 at N=128N=128; 93.893.8 vs. 94.694.6 at N=32N=32).
  • Training-loop changes and co-adapted checkpoints: Online methods require modifying the training loop; when applied post hoc to a frozen LangFlow checkpoint already co-trained with its Gumbel schedule, KIS shifts the schedule away from the training distribution (92.2→94.192.2 \to 94.1 LM1B GenPPL).

↳ Manuscript updates: Abstract, Secs. 1–3, 4.2, 5, App. C (Table 7); adopted entropic time [5, 6] at Eq. 5 and separated non-contextual embeddings (Plaid, RePlaid, LangFlow) from FLM and ELF.

2. Why Constant-Loss / Constant-Entropy Rules Fall Short, and Relation to RePlaid — Reviewer vQxN Q2

Takeaway: Under a unigram prior, our population curve equals RePlaid's variance-optimal CDF by Gaussian I-MMSE, yet directly enforcing constant per-step loss or entropy decay underperforms constrained location-scale calibration.

By Gaussian I-MMSE [7], −Hπ′(λ)=12eλMMSE⁡π(eλ)-H_\pi'(\lambda) = \frac{1}{2}e^\lambda \operatorname{MMSE} _ \pi(e^\lambda), matching fθ(λ)=12eλMSE⁡θf_\theta(\lambda) = \frac{1}{2}e^\lambda\operatorname{MSE} _ \theta in RePlaid [2] under MSE⁡θ=MMSE⁡π\operatorname{MSE} _ \theta = \operatorname{MMSE} _ \pi: flattening expected timestep loss minimizes ELBO variance (RePlaid Prop. 1, Sec. 5.2), induces linear mutual-information decay (RePlaid Lemma 3, App. G), and makes single-token cross-entropy linear in t=1−τπ(λ)t = 1 - \tau_\pi(\lambda) (RePlaid Prop. 2, Sec. 5.2). However, RePlaid [2], Entropic Time [5], and FLM [8] directly equalize per-step loss, entropy, or decoding error, which underperforms constrained calibration for flow matching and finite-step solvers:

  • Training tail coverage beats constant-rate densities: Under velocity and flow-matching weights w(λ)w(\lambda), sampling directly from qKIS(λ)q_{\mathrm{KIS}}(\lambda) on [μ0−6σ0,μ0+6σ0][\mu_0 - 6\sigma_0, \mu_0 + 6\sigma_0] (Eq. 16) underperforms shifting only the median to Q(1/2)Q(1/2), both unweighted (122.8122.8 vs. 79.379.3 GenPPL on ELF-B; 98.498.4 vs. 92.592.5 on FLM-B) and with importance weights q0/qKISq_0/q_{\mathrm{KIS}} preserving the objective up to negligible truncation error (82.182.1 vs. 79.379.3; Table 10), because concentrating samples in the transition window under-trains tail noise levels [10].
  • Preserving solver step ratios beats raw quantiles: Proposition 1's 1D W2W_2-optimal positive affine fit (a∗,b∗)(a^{\ast}, b^{\ast}) preserves interior log-SNR step ratios, whereas raw constant-entropy quantiles raise ELF-L GenPPL (108.9→110.3108.9 \to 110.3 raw vs. 107.1107.1 affine).
  • Step-0 offline calibration and metric-driven extensions: KIS estimates τ^(λ)\widehat{\tau}(\lambda) offline in 95 s before step 0, extends to contextual prototypes eˉy\bar e_y, and motivates optimizing schedules directly for generation quality.

↳ Manuscript updates: Sec. 2, Sec. 3.3, Sec. 5, App. A.3 (Eq. 16), App. B.1 (Eqs. 18–21), and App. E.2 (Table 10).

AuthorsOpenReview ↗

Overall Rebuttal (2/4)

3. Extending KIS to Contextualized Embeddings in ELF — Reviewer vQxN Q7; Reviewer X8od Q3; Reviewer Sx98 Q1

Takeaway: In ELF's frozen encoder space, averaging contextual states per token captures the signal contraction from contextual variance, aligning KIS with the denoiser's SNR-weighted MSE profile.

For continuous flow matching, the relevant difficulty curve is the denoiser's SNR-weighted latent MSE rather than discrete H(Yi∣Zλ,i,context)H(Y_i \mid Z_{\lambda,i}, \text{context}). We construct this post-hoc reference τdenoiser(λ)\tau_{\mathrm{denoiser}}(\lambda) by integrating 12eλMSE⁡θ\frac{1}{2}e^\lambda\operatorname{MSE} _ \theta on ELF-B across 49 noise levels and 512 validation sequences (Qden(1/2)=−3.02,−2.91,−2.84Q_{\mathrm{den}}(1/2) = -3.02, -2.91, -2.84 for cutoffs λc∈{2,4,6}\lambda_{\mathrm{c}} \in \lbrace 2, 4, 6 \rbrace, as eλe^\lambda weights clean-side residual MSE):

  • Static lookup vs. contextual prototypes: Static lookup KIS (Row 1, 95 s) already matches the denoiser slope b∗b^{\ast} (0.650.65 vs. 0.670.67) and cuts midpoint error by 43%. Averaging normalized encoder states over 100k training sequences into token centroids eˉy\bar e_y (Row 2, which set Pmean∗=−0.700P_{\mathrm{mean}}^{\ast} = -0.700 in Table 1) shifts Q(1/2)Q(1/2) to −2.78-2.78, matching unit-RMS lookup (−4.18-4.18) rescaled by centroid RMS rproto=0.50r_{\mathrm{proto}} = 0.50 (−4.18−2log⁡0.50=−2.79-4.18 - 2\log 0.50 = -2.79); M=4M=4 kk-means prototypes (Row 3) further reach 78.578.5 GenPPL.
  • Effective SNR mechanism: Under xi=eˉyi+δix_i = \bar e_{y_i} + \delta_i (r=0.80,sδ≈0.62r=0.80, s_\delta \approx 0.62), SNReff(λ)=eλrproto2/(1+eλsδ2)\mathrm{SNR} _ {\mathrm{eff}}(\lambda) = e^\lambda r_{\mathrm{proto}}^2 / (1 + e^\lambda s_\delta^2) has eλsδ2≈0.024≪1e^\lambda s_\delta^2 \approx 0.024 \ll 1 near λ≈−2.8\lambda \approx -2.8. KIS removes default ELF-B's N=32→128N=32 \to 128 GenPPL degradation (121.2→79.3121.2 \to 79.3 at 20k steps; 86.2→71.486.2 \to 71.4 at 60k), whereas ELF's auxiliary decoder (Rows 4–5, trained near clean states on λdec∼N(−1.62,1.62)\lambda_{\mathrm{dec}} \sim \mathcal{N}(-1.62, 1.6^2)) compresses b∗b^{\ast} to 0.41–0.440.41\text{--}0.44 and degrades N≥16N \ge 16 sampling.

↳ Manuscript updates: Sec. 4.4 (Table 6), App. E.2 (Table 10), and App. E.3 (Fig. 7).

Reference VariantLM1B Midpoint Q(1/2)Q(1/2)Gap ∣ΔQ1/2∣\lvert\Delta Q_{1/2}\rvertSlope b∗b^{\ast}Corr. ρ\rhoNRMSELM1B Train N=32N=32LM1B Train N=128N=128OWT Sampling N=4N=4OWT Sampling N=8N=8OWT Sampling N=16N=16OWT Sampling N=32N=32
Default schedule−4.39-4.391.551.00n/an/a113.4121.20.0%0.0%0.0%0.0%
Post-hoc τdenoiser\tau_{\mathrm{denoiser}} baseline−2.84-2.840.000.671.0000.00093.479.0-16.4%-14.8%-10.9%-8.0%
1. Static T5 lookup + RMS bridge−3.73-3.730.890.650.9860.038104.190.1-16.1%-14.5%-10.5%-7.6%
2. Top-level contextual centroids−2.78-2.780.060.680.9930.02493.879.3-15.4%-13.8%-9.8%-7.1%
3. Contextual kk-means, M=4M=4−2.91-2.910.070.660.9950.01992.678.5-16.8%-15.1%-11.2%-8.3%
4. Contextual decoder on ztz_t−3.47-3.470.630.410.9840.04198.484.7-12.3%-8.5%+23.8%+32.6%
5. Contextual decoder on x^0\widehat x_0−3.28-3.280.440.440.9890.03196.282.5-13.1%-3.6%+29.1%+32.9%
AuthorsOpenReview ↗

Overall Rebuttal (3/4)

4. Multi-Scale Training and (a,b)(a, b) Grid-Search Comparisons — Reviewer vQxN Q6; Reviewer X8od Q2; Reviewer Sx98 Q2

Takeaway: From-scratch KIS lowers or matches GenPPL across 14 pairs, and closed-form KIS achieves GenPPL within 0.2–0.8 points (less than 1% relative difference) of the best 25-rollout 5×5 grid results without validation rollouts.

  • Multi-scale training & sampling audit: Table 1 compares matched Default vs. KIS runs sharing initialization and minibatch order (FLM's 1M-step LM1B schedule [8], 20k LM1B steps for ELF [4], 59k OWT steps). Table 2 averages paired 512-sample generation over 3 seeds under Algorithm 2 (audited across all five models) on released FLM-B, LangFlow, and OWT ELF checkpoints and our Table 1 default LM1B ELF checkpoints, as Hu et al. [4] release only OWT checkpoints (App. D.2).
  • Affine parameterization & grid comparison: Proposition 1 identifies the 2-parameter step-ratio-preserving family (a,b)(a, b) (Eq. 9) that makes this grid search possible: closed-form KIS beats a 9-rollout 3×33 \times 3 grid, achieves GenPPL within 0.2–0.80.2\text{--}0.8 points (<1% relative difference) of the best 25-rollout 5×55 \times 5 grid (~50 min/NN) without validation rollouts, and lands inside the 8-run PmeanP_{\mathrm{mean}} sweep optimum (Table 11) before step 0.

↳ Manuscript updates: Secs. 4.1–4.3 (Tables 1, 2, 5), App. D.2–D.3 (Table 8), and App. E.2 (Table 11).

ModelSetting and ReferenceLM1B Budget NNLM1B Default GenPPLLM1B KIS GenPPLOWT Budget NNOWT Default GenPPLOWT KIS GenPPL
FLM-BOne-hot simplex, exact posterior102496.992.5102470.362.9
ELF-BWithout encoder, lookup reference128154.9142.33229.826.6
ELF-MWithout encoder, lookup reference128115.4106.83225.724.1
ELF-LWithout encoder, lookup reference128109.1104.33230.929.2
ELF-BWith encoder, contextual centroid128121.279.33232.031.9
ELF-MWith encoder, contextual centroid128112.1103.83226.423.9
ELF-LWith encoder, contextual centroid128108.9104.83231.128.8
AuthorsOpenReview ↗

Overall Rebuttal (4/4)

5. Random-Seed Protocol, Diversity, and Evaluator Checks — Reviewer Sx98 Q3–Q5

Takeaway: Training comparisons use 1 matched seed due to pretraining compute cost, and sampling evaluations average over 3 paired 512-sequence seeds; gains hold across models, diversity metrics, and Llama-3-8B evaluation.

  • Seed protocol & stability: Due to the compute cost of 20k–1M-step pretraining, each training pair uses 1 matched seed (sharing initialization and minibatch order; 13 GenPPL reductions and 1 tie across Table 1's 14 pairs, and a smooth basin across Table 11's 8-run PmeanP_{\mathrm{mean}} sweep), while sampling evaluations average over 3 paired seeds (512 sequences per seed).
  • Diversity, Llama-3-8B, & split hygiene: Frozen sampling preserves Distinct-2/3\text{Distinct-2/3} and H4-gramH_{4\text{-gram}} and confirms GenPPL rankings under Llama-3-8B (Table 9); we estimate both π\pi and contextual centroids strictly on the training split.

↳ Manuscript updates: Sec. 3.1, Secs. 4.1–4.2, Sec. 5, and App. D.1–D.3 (Tables 8–9).

References

  • [1] I. Gulrajani and T. B. Hashimoto. Likelihood-Based Diffusion Language Models. NeurIPS 2023.

  • [2] Z. Yang, W. Guo, S. Zhang, S. S. Sahoo, Y. Chen, A. Vahdat, M. Mardani, and J. Thickstun. Continuous Diffusion Scales Competitively with Discrete Diffusion for Language. arXiv:2605.18530, 2026.

  • [3] Y. Chen, C. Liang, H. Sui, R. Guo, C. Cheng, J. You, and G. Liu. LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling. arXiv:2604.11748, 2026.

  • [4] K. Hu, L. Qiu, Y. Lu, H. Zhao, T. Li, Y. Kim, J. Andreas, and K. He. ELF: Embedded Language Flows. arXiv:2605.10938, 2026.

  • [5] D. Stancevic, F. Handke, and L. Ambrogioni. Entropic Time Schedulers for Generative Diffusion Models. arXiv:2504.13612, 2025.

  • [6] G. Raya, B. Nguyen, G. Batzolis, Y. Takida, D. Stancevic, N. Murata, C.-H. Lai, Y. Mitsufuji, and L. Ambrogioni. Noise Scheduling as Information-Guided Allocation in Diffusion Training. arXiv:2602.18647, 2026.

  • [7] D. Guo, S. Shamai, and S. Verdú. Mutual Information and Minimum Mean-Square Error in Gaussian Channels. IEEE Trans. Inf. Theory, 51(4):1261-1282, 2005.

  • [8] C. Lee, J. Yoo, M. Agarwal, S. Shah, J. Huang, A. Raghunathan, S. Hong, N. M. Boffi, and J. Kim. Flow Map Language Models: One-step Language Modeling via Continuous Denoising. arXiv:2602.16813, 2026.

  • [9] A. Sabour, S. Fidler, and K. Kreis. Align Your Steps: Optimizing Sampling Schedules in Diffusion Models. ICML 2024.

  • [10] L. Ambrogioni, G. Franzese, A. Foresti, G. Raya, B. Nguyen, G. Batzolis, Y. Takida, N. Murata, C.-H. Lai, and Y. Mitsufuji. From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime. arXiv:2607.20540, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.