acceptodds
Under review as a conference paper at ICLR 2027

Understanding Encoding in Continuous Diffusion Language Models

Abstract

Continuous language models encode discrete tokens as continuous vectors, but it remains unclear whether encoding design should follow the same principles as in discrete models. In this work, we first show theoretically that encoding has an additional target-defining role in continuous models. Controlled experiments further show how this role leads encoding to affect the two model families in qualitatively different ways. Building on this distinction, we systematically study encoding design and identify geometry, dimension, and encoding–model compatibility as three key considerations. Guided by these insights, we progressively refine a continuous language model's encoding and neural parameterization, achieving a GenPPL of 15.8 on OpenWebText at approximately 130M parameters and outperforming representative state-of-the-art diffusion language models of comparable scale. Our findings establish encoding as a distinct design problem in continuous language modeling and demonstrate the value of systematic encoding analysis.

Primary area:
generative models
Keywords:
Continuous Language Models, Generative models, Diffusion Models, Encoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mgNrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how token encoding affects continuous diffusion language models. This work decomposes the training objectives and shows that encoding represents tokens and defines the continuous target distribution. Experiments on Text8 and grayscale CIFAR-10 show that encoding geometry, dimensionality, and compatibility with the model's input and output parameterization are important design considerations. The paper presents jointly learned low-dimensional sphere encodings with probability first outputs and the 130M parameter model achieves low generative PPL on OWT, outperforming diffusion baselines.

Critical strengths
  1. Clear explanation of what encoding does and why it matters.

The decomposition of the objectives makes it explicit that encoding defines the continuous target distribution. This provides a useful framework for understanding why encoding choices affect continuous and discrete models differently.

  1. Experimental results showing practical design choices.

Experiments are designed to identify which encoding properties affect performance: encoding geometry and dimensionality are important. OWT experiments demonstrate how these combined with input and output parameterizations can improve performance in large vocabulary setting.

Critical weaknesses
  1. Limited support for the performance advantage.

In the comparison, the diffusion models use different sampling protocols: generation quality depends on the number of sampling steps, the sampling method, and the decoding rule. Lower GenPPL could come from an advantageous sampling protocol, not only from model improvements.

Also, GPT-2 Large based ppl and token-level entropy provide useful evidence but may not fully capture coherence or diversity across generated texts. Additional generation quality metrics would strengthen the practical claims.

Questions for authors
  1. Does RePlaID use the same training-token budget, sampling steps, terminal decoding, and GenPPL calculation as the proposed method?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mgNr

Thank you for your thorough review and your positive assessment of our work. We appreciate the opportunity to further improve our work through this discussion. Below is our point-by-point response.

Weakness 1:

Limited support for the performance advantage. In the comparison, the diffusion models use different sampling protocols: generation quality depends on the number of sampling steps, the sampling method, and the decoding rule. Lower GenPPL could come from an advantageous sampling protocol, not only from model improvements.

We have re-evaluated our method and continuous and discrete diffusion model baselines under a unified evaluation protocol across multiple sampling budgets. All models use the same number of generated samples (N=5000), GPT-2 Large scoring, and the same definition of unigram entropy. We align sampling budgets by the actual number of network function evaluations (NFE). The RePlaID results are from our own reproduction trained for 70k steps, while the other baselines use officially released checkpoints.

Model64 NFE: GenPPL / Entropy256 NFE: GenPPL / Entropy1024 NFE: GenPPL / Entropy
Final (Ours)57.2 / 7.3828.5 / 7.3516.0 / 7.38
RePlaID53.8 / 7.4732.0 / 7.3923.1 / 7.37
LangFlow82.8 / 7.4451.7 / 7.3941.1 / 7.40
ELF-B36.6 / 6.7422.7 / 6.7118.6 / 6.68
SEDD106.4 / 7.3865.5 / 7.1642.9 / 6.83
MDLM112.2 / 7.4267.8 / 7.1943.9 / 6.86
BD3-LM B4N/AN/A94.3 / 7.47

BD3-LM B4 is marked as N/A at 64 and 256 NFE because these budgets are not supported by its sampling setup. For Final and RePlaID, the 1024-NFE GenPPL values differ slightly from those in the main text, while the entropy values are noticeably higher, primarily due to the larger sample size of N=5000N=5000. At 1024 NFE, our method achieves the lowest GenPPL while maintaining high entropy. At 256 NFE, it achieves lower GenPPL than all evaluated baselines except ELF-B, while maintaining substantially higher entropy than ELF-B. At 64 NFE, our method performs slightly worse than RePlaID but remains competitive among the evaluated baselines. Overall, these results demonstrate a favorable trade-off between generation quality and diversity under a unified evaluation protocol, particularly at 256 and 1024 NFE. We have added these results to Appendix D.6.

Weakness 2:

Also, GPT-2 Large based ppl and token-level entropy provide useful evidence but may not fully capture coherence or diversity across generated texts. Additional generation quality metrics would strengthen the practical claims.

Following your suggestion and that of Reviewer x4y1, we have added MAUVE evaluation to complement GenPPL and unigram entropy by comparing the distributions of generated and real text. We reuse the generated texts from the 1024-NFE setting above and compute MAUVE using the same held-out OpenWebText validation set as the reference corpus for all models, with GPT-2 Large as the feature extractor.

ModelMAUVE ↑
Final (Ours)0.772
RePlaID0.838
LangFlow0.900
ELF-B0.330
SEDD0.590
MDLM0.628
BD3-LM B40.005

The MAUVE results show that our model achieves a higher score than most of the evaluated baselines, although it remains below RePlaID and LangFlow. We have added these results to Appendix D.7.

Question 1:

Does RePlaID use the same training-token budget, sampling steps, terminal decoding, and GenPPL calculation as the proposed method?

Thank you for your question. Our RePlaID reproduction uses the same 70k training steps, same training-token budget, sequence length n=1024, sampling steps, terminal decoding (MAP), and GPT-2 Large GenPPL calculation as the BFN model in our main experiments. In our response to your Weakness 1, we provide additional matched comparisons across different numbers of sampling steps, which support the advantage of our model.


This concludes our point-by-point response. Thank you again for your careful review. We welcome further discussion if you have any additional questions. We hope that these clarifications and additional results address your concerns, and we kindly ask you to reassess our work and score in light of the new material.

Thank you for the additional experiments and clarification.

My question about the matched RePlaID setup is addressed: The matched NFE comparisons and MAUVE evaluation address my concerns.

I recommend specifying the sampling algorithm, time discretization, and terminal decoding rule for each baseline in the paper, as matching NFE alone does not fully resolve the sampling-protocol concern.

Also, the paper could be strengthened by discussing why the proposed model’s MAUVE is below RePlaID and LangFlow, and why its advantage over RePlaID does not hold at 64 NFE.

As the major concerns are addressed, I keep my score for weak acceptance.

AuthorsOpenReview ↗

Further Response to Reviewer mgNr

Thank you for your reply and constructive suggestions. We greatly appreciate the opportunity to further discuss our work with you. Below, we address your two remaining questions.

1. Sampling algorithms, time discretization, and terminal decoding

To further clarify the sampling protocols, we summarize the specific settings used for each model in our evaluation.

(1) Sampling algorithms. We use the official sampling algorithms provided by the respective implementations wherever possible. Specifically:

  • Ours and our reimplementation of RePlaID: reverse SDE sampling, each following its own SDE settings.
  • LangFlow and ELF-B: official SDE samplers (Euler–EDM for LangFlow; SDE for ELF-B). Together with Ours and RePlaID, all four continuous models therefore use the same broad class of SDE reverse sampling.
  • SEDD, MDLM, and BD3-LM B4: reverse samplers of masked diffusion. SEDD and MDLM both discretize this reverse on a time grid (closed-form reverse for SEDD; Euler/ancestral reverse for MDLM). BD3-LM B4 uses a first-hitting reverse of the same mask process.

Self-conditioning is used whenever the official method provides it (Ours, RePlaID, LangFlow, ELF-B). Reported NFEs already count the corresponding extra forwards.

(2) Time discretization. We align NFE budgets at 64, 256, and 1024 wherever supported. For all models that admit a fixed-step sampler, we first discretize the time axis uniformly, then map those times through each model’s official schedule to the corresponding SNR. The schedules are as follows:

  • Ours and our reimplementation of RePlaID: learned noise schedules.
  • LangFlow: official EDM schedule.
  • ELF-B: official logit-normal schedule.
  • SEDD and MDLM: official log-linear noise schedules.
  • BD3-LM B4: official log-linear noise schedule; first-hitting samples the times fed into this schedule instead of reading it on the uniform NFE grid.

(3) Terminal decoding. We use the corresponding terminal decoding rules of each model:

  • Ours, RePlaID, LangFlow, ELF-B, and MDLM use argmax decoding, while SEDD and BD3-LM B4 use Gumbel-max sampling (i.e., taking the argmax of Gumbel-perturbed scores).

Overall, we have aligned the evaluation settings as closely as possible, including the number of generated samples, GPT-2 Large scoring, unigram entropy calculation, and actual NFE budgets. For sampling-specific details, we follow the official implementations of the respective models to ensure faithful comparisons.

2. Additional ablation of posterior-mean input and its relationship to MAUVE and low-NFE sampling

We further conducted an ablation experiment to investigate the relationship between posterior-mean input, MAUVE, and sampling performance under limited NFE budgets. Specifically, we evaluated the variant of our final model without posterior-mean input, and obtained the following results:

Configuration64 NFE: GenPPL / Entropy1024 NFE: GenPPL / EntropyMAUVE
Final (Ours)57.2 / 7.3816.0 / 7.380.772
Final − posterior-mean input (Ours)52.1 / 7.4019.7 / 7.310.853
RePlaID53.8 / 7.4723.1 / 7.370.838
LangFlow82.8 / 7.4441.1 / 7.400.900

Interestingly, removing posterior-mean input substantially improves MAUVE and low-NFE sampling performance, while degrading generation performance at high NFE. In particular, its MAUVE score reaches 0.853, comparable to RePlaID and close to LangFlow, and its GenPPL / Entropy at 64 NFE improves to 52.1 / 7.40, close to RePlaID (53.8 / 7.47). However, its GenPPL at 1024 NFE is substantially worse than that of the final model.

We conjecture that posterior-mean input changes how the model utilizes the denoising state during sampling, potentially introducing a trade-off between sampling efficiency at low NFE and generation quality at high NFE. Posterior-mean input may provide a more informative estimate of the clean representation, benefiting the refinement of samples when sufficient denoising steps are available. However, under a limited sampling budget, this additional input may make the model more dependent on accurate intermediate estimates, potentially making coarse discretization less effective. This may also affect the overall distribution of generated samples, as reflected by the MAUVE results. These observations suggest that the choice of input parameterization can influence not only generation quality but also the sampling behavior across different NFE budgets.

Thank you again for your helpful comments. We hope that these additional clarifications and experiments address your remaining concerns. If you find our responses satisfactory, we would sincerely appreciate your consideration of an improved rating.

Reviewer_x4y1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies token encoding in continuous diffusion language models (CLMs). It argues that encoding provides representations and defines the target distribution. The authors establish this by breaking down objectives and making many controlled experiments. Experiments on Text8 and grayscale CIFAR-10 identify encoding geometry, dimension, and encoding–model compatibility as important design factors. The authors then train a model with spherical encodings and better parameterizations and yields a GenPPL of 15.8 on OpenWebText at 130M parameters.

Critical strengths

The paper provides a controlled study distinguishing the representation and training target of encoding. The results show that encoding geometry influences CLMs more strongly than discrete models. It also tests different encoding dimensions and model compatibility. The proposed design choices also translate into more practical settings, e.g., 130M model on OpenWebText.

Critical weaknesses

The main weakness is that generation quality is evaluated through GenPPL and unigram entropy, without metrics such as MAUVE or evaluating coherence and diversity. Furthermore, the comparisons are not conducted under fully matched sampling protocols. The theoretical decomposition largely follows standard variational identities and does not directly explain why particular encoding geometries or dimensions should be optimal. Finally, there is no seed variance for the main experiments. The method also seem overlap with prior work on learned encoding geometry, particularly RePlaID.

Questions for authors
  1. Could the authors provide matched comparisons with leading continuous diffusion language models using consistent sampling budgets and additional generation-quality metrics such as MAUVE?
  2. How do the authors explain the observation that the final posterior-mean input improves GenPPL while worsening validation perplexity, and what does this imply about actual distribution modeling quality?
  3. Could the authors clarify the precise definition of spelling accuracy and report variability across random seeds for the main geometry and OpenWebText experiments?
  4. Finally, can the authors more clearly distinguish their theoretical contributions from standard variational decompositions and prior work on learned embedding geometry, potentially through quantitative diagnostics connecting encoding properties to the proposed variational gaps?
9 comments
AuthorsOpenReview ↗

Response to Reviewer x4y1 (1/6)

Thank you for your thorough review and constructive feedback. We appreciate the opportunity to improve our work through this discussion and to clarify our core contributions. Below is our point-by-point response. To make the reasoning in our responses clearer, we have slightly reordered your questions.

Part I: Theory and Novelty

Weakness 1:

Finally, the limited uncertainty reporting for the main experiments and the overlap with prior work on learned encoding geometry, particularly RePlaID, weaken the robustness and novelty claims.

Thank you for your question. We would like to clarify that although previous studies have empirically found that CLMs require encodings that differ substantially from those used in DLMs, and have proposed specific designs such as spherical encodings and probability-first outputs, they have not provided a concrete theoretical framework to explain the underlying reasons. On the other hand, our work, as Reviewer mgNr noted, "provides a useful framework for understanding why encoding choices affect continuous and discrete models differently."

Before our work, the lack of a theoretical understanding of the reasons behind these empirical observations often led us to carry over intuitions from DLMs when discussing CLM encodings, sometimes without realizing it. This transfer often causes us to overlook the additional requirements that continuous modeling places on encoding design. For example, the following arguments may all seem plausible, even though they are not supported by the empirical evidence in RePlaID and related work:

  1. "Although one-hot encodings are high-dimensional and sparse, couldn't a learnable embedding immediately afterward make them relatively low-dimensional and dense? DLMs such as AR models do exactly this. Why couldn't CLMs do the same?"
  2. "One-hot encodings in DLMs do not need to be trained, since the subsequent embedding lookup table is learnable. In a CLM, then, can learning only the internal embedding replace designing or learning the encoding itself?"
  3. "Since the hidden dimension of the model's internal embedding is often large, is there really any benefit to using a very low-dimensional encoding at the beginning?"

Our theoretical framework explains where these intuitions come from and why they cannot justify directly transferring encoding designs from DLMs to CLMs. Specifically, we distinguish the representation role that encoding plays in both CLMs and DLMs from the target-defining role unique to CLMs, and explicitly locate the latter within the training objective. This explains the theoretical basis for both the similarities and the differences in how encoding affects DLMs and CLMs.

Therefore, the shared representation role explains the source of these intuitions, while the target-defining role unique to CLMs explains why directly transferring them is problematic. The experiments in Section 3.2 further confirm that the target-defining role causes encoding to affect CLMs very differently, supporting our theoretical distinction. This theory naturally motivates and guides the systematic experimental study of CLM encoding in Sections 4 and 5. This distinction also supports using separate terms for encoding and embedding in CLMs: in addition to providing a representation, encoding defines the continuous target, whereas the internal embedding does not serve this latter function.

Overall, our core contribution is a unified theoretical framework that explains why encoding plays different roles in continuous and discrete language models, together with controlled experiments that translate this distinction into concrete design guidance. Our decomposition can also explain the effectiveness of specific designs, such as spherical and low-dimensional encodings, providing theoretical support for previous empirical findings. Please see our response to your Weakness 2 for details.

Regarding your concern about limited uncertainty reporting, we have added relevant experiments in our response to your Question 4. Thank you for giving us the opportunity to clarify these points.

AuthorsOpenReview ↗

Response to Reviewer x4y1 (2/6)

Question 1:

How do the authors explain the observation that the final posterior-mean input improves GenPPL while worsening validation perplexity, and what does this imply about actual distribution modeling quality?

Thank you for this insightful question. This phenomenon arises because GenPPL and ValPPL evaluate different aspects of the model, and this distinction can be understood qualitatively through the gap terms in our NELBO decomposition.

Specifically, the decomposition in Theorem 1 examines how encoding enters model training through the gap terms between the NELBO and the discrete, rather than continuous, KL divergence KL!(P,∣,Qθ,η)\mathrm{KL}\\!\left(P\\,\\|\\,Q_{\theta,\eta}\right). We isolate the discrete KL because the ultimate goal of any language model is to learn a discrete distribution Qθ,η(X)Q_{\theta,\eta}(X) that approximates the discrete target distribution P(X)P(X), as illustrated in Figure 1. Therefore, the discrete KL is the objective we ultimately wish to minimize, because it most directly measures how closely the model distribution matches the discrete data distribution. As for GenPPL and entropy, although they may not align perfectly with the discrete KL, both are computed directly on generated text and thus, like the discrete KL, assess actual generation performance.

On the other hand, ValPPL is computed from the average per-token NELBO on the validation set and therefore also incorporates the two gap terms. These terms mean that the gradient of the CLM training loss does not necessarily align with the gradient of the discrete KL, so changes in ValPPL need not track changes in generation metrics. Consequently, ValPPL may not accurately reflect the model's generative ability in certain cases. This is an important insight from our theoretical analysis: encoding affects the gap terms in specific ways, introducing additional gradients that differ from those of the underlying discrete KL objective and thereby influencing learning.

Therefore, improved GenPPL and entropy can coexist with higher ValPPL. This is consistent with the insight highlighted by our theoretical framework. Thank you for your question.

AuthorsOpenReview ↗

Response to Reviewer x4y1 (3/6)

Weakness 2 & Question 2:

The theoretical decomposition is useful but largely follows standard variational identities and does not directly explain why particular encoding geometries or dimensions should be optimal.

Finally, can the authors more clearly distinguish their theoretical contributions from standard variational decompositions and prior work on learned embedding geometry, potentially through quantitative diagnostics connecting encoding properties to the proposed variational gaps?

Thank you for this thought-provoking question. It raises an important point and helps us further illustrate the concrete value of our theoretical framework. Below, we explain how our decomposition can provide an intuitive explanation for why encoding normalization benefits training, accounting for the advantage of spherical encodings in Section 4.1, and can also explain the inverted-U trend with encoding dimension in Section 4.2.

1. Geometry. Our decomposition explains why encoding normalization benefits training and, consequently, why spherical encodings are advantageous.

To illustrate this, consider the single-token setting and let a=∣z0∣a=\\|\mathbf{z}^0\\| and v=z0/∣z0∣\mathbf{v}=\mathbf{z}^0/\\|\mathbf{z}^0\\| denote the radius and direction of the continuous encoding, respectively. Applying the KL chain rule to the encoding gap GencG_{\mathrm{enc}} in Theorem 1 separates it into matching the radial distribution and matching the directional distribution conditional on the radius:

G_{\mathrm{enc}} &= \underbrace{\mathbb E_X\mathrm{KL}\\!\left(e_\eta(a\mid X)\\,\\|\\,q_{\theta,\eta}(a\mid X)\right)}_{G_{\mathrm{rad}}}\\\\ &\quad+ \underbrace{\mathbb E_{X,a}\mathrm{KL}\\!\left(e_\eta(\mathbf{v}\mid a,X)\\,\\|\\,q_{\theta,\eta}(\mathbf{v}\mid a,X)\right)}_{G_{\mathrm{ang}}}. \end{aligned}$$ In common continuous diffusion language models, given the current codebook, the encoding density is obtained by adding isotropic Gaussian noise to a deterministic encoding vector: $$e_\eta(\mathbf{z}^0\mid X) = \mathcal N\\!\left(\mathcal E_\eta(X),\sigma_0^2\mathbf{I}_d\right).$$ Because the Gaussian noise is isotropic, the distribution of the encoding norm after adding noise depends only on the norm of the original encoding vector, not its direction. Therefore, when all encoding vectors have the same norm, the conditional radial distribution $e_\eta(a\mid X)$ is identical for every token. **This makes the radial target in the encoding gap independent of token identity: the model can learn a single shared radial distribution instead of a different one for each token. Spherical encoding therefore removes the need for token-specific radial matching and simplifies the learning task specified by the encoding gap.** We also observe experimentally that removing the encoding norm constraint degrades generation performance, confirming the explanatory value of our decomposition.
Reviewer_mcNiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies encoding design in continuous diffusion LMs. A NELBO decomposition is used to separate a "representation" role of the encoding, shared with discrete LMs, from a "target-defining" role specific to CLMs. This distinction is supported with controlled experiments on Text8 and grayscale CIFAR-10 (K=27). A BFN-based CLM is generalized to learned encodings and revised input/output interfaces, reaching an OWT GenPPL of 15.8 at about 130M parameters.

Critical strengths
  • The question is important: which space to diffuse in, and how to co-design and jointly train the encoding with the architecture and the algorithm in continuous diffusion language models.
  • The controlled experiments are thoughtfully designed. The geometry variants are tested on small-scale toy models and then conducted experiments on real-scale language models.
Critical weaknesses
  • Theoretically, the decomposition does not by itself establish an encoding-dependent optimum. Theorem 1 is a valid chain-rule identity that holds for every (θ,η)(\theta,\eta), so it says nothing about whether inf⁡θLCLM\inf_\theta \mathcal L_{\mathrm{CLM}} depends on η\eta. Decomposing in the other order gives LCLM=H(P)+KL(pη(z0:T) ∥ q(z0:T))+EpηKL(pη(X∣z0) ∥ D(X∣z0))\mathcal L_{\mathrm{CLM}} = H(P) + \mathrm{KL}(p_\eta(z_{0:T})\,\|\,q(z_{0:T})) + \mathbb E_{p_\eta}\mathrm{KL}(p_\eta(X|z_0)\,\|\,D(X|z_0)). With an exact path model and an unrestricted decoder D, this attains H(P)H(P) for any encoding, just as for DLMs. I would suggest stating carefully about theoretical rigor and novelty here.
  • The contribution relative to prior work is unclear, and encoding-specific gains are not isolated.
    • Plaid already argues that embeddings play "a more fundamental role" in diffusion LMs than in AR models, and favors low-dimensional (d=16) learned embeddings. RePlaID (App. S) compares embedding dimensions and learned, frozen-random, frozen-pretrained, and one-hot embeddings. Probability-first outputs, output evidence, learned schedules and normalized embeddings appear in Plaid, RePlaID and LangFlow.
    • in Sec. 5, the largest late gain comes from the posterior-mean input (GenPPL 20.1 to 15.8). It would be clearer, if time and computation allows, to include results isolating the encoding effect: re-run the fixed encodings (one-hot, GPT-2, PCA, and a fixed random d=64 sphere) with the final interface, and add leave-one-out ablations from the final model.
  • Experimentally, lack of rigorous gen ppl vs entropy evaluation : The rigorous approach proposed by prior works is to compare the Pareto curves of all methods (with axis as entropy and gen ppl). There is usually a trade-off between gen ppl and entropy. It would be helpful to evaluate public langflow/elf/reokaud/sedd/mdlm/bd3-lm checkpoints under exactly your sampling protocol (steps, sample count, final decoding, GPT-2 Large scoring, entropy definition), and report GenPPL–entropy curves rather than single points, in order to qualify the claim in the abstract.
Questions for authors
  1. What is the rationale, theoretical or empirical, for constraining encodings to the unit sphere, which discards radial information? A comparison with unnormalized Euclidean encodings would show whether the geometry and dimension findings depend on the normalization.
  2. The posterior-mean input lowers GenPPL while raising ValPPL. How do you interpret this divergence?
  3. For training details, please specify:
    • the implemented NELBO estimator (continuous- or discrete-time), the denoising parameterization, and how the split heads are weighted in the mean-first and separate-head variants (did you use MSE and CE loss for different heads?);
    • for the controlled DDPM: the noise schedules, input scaling, and any per-dimension tuning;
    • for the 70k-step models in App. B.2: the initialization, and the RePlaID reimplementation's training budget for comparison.
  4. Qualitative samples: Appendix D.5 shows selected samples from the final configuration. Could you please show randomly selected, uncurated samples generated with the same seed?

Additional comments and suggestions are given under Weaknesses.

13 comments
AuthorsOpenReview ↗

Response to Reviewer mcNi (1/7)

Thank you for your thorough review and constructive feedback. We appreciate the opportunity to further improve our work through this discussion and to clarify our core contributions. Below is our point-by-point response. To make the reasoning in our responses clearer, we have slightly reordered your questions.

Part I: Theory and Novelty

Weakness 1:

The contribution relative to prior work is unclear. Plaid already argues that embeddings play "a more fundamental role" in diffusion LMs than in AR models, and favors low-dimensional (d=16) learned embeddings. RePlaID (App. S) compares embedding dimensions and learned, frozen-random, frozen-pretrained, and one-hot embeddings. Probability-first outputs, output evidence, learned schedules and normalized embeddings appear in Plaid, RePlaID and LangFlow.

Thank you for your question. We would like to clarify that although previous studies have empirically found that CLMs require encodings that differ substantially from those used in DLMs, and have proposed specific designs such as spherical encodings and probability-first outputs, they have not provided a concrete theoretical framework to explain the underlying reasons. On the other hand, our work, as Reviewer mgNr noted, "provides a useful framework for understanding why encoding choices affect continuous and discrete models differently."

Before our work, the lack of a theoretical understanding of the reasons behind these empirical observations often led us to carry over intuitions from DLMs when discussing CLM encodings, sometimes without realizing it. This transfer can cause us to overlook the additional requirements that CLM places on encoding design. For example, the following arguments may all seem plausible, even though they are not supported by the empirical evidence in RePlaID and related work:

  1. "Although one-hot encodings are high-dimensional and sparse, couldn't a learnable embedding immediately afterward make them relatively low-dimensional and dense? DLMs such as AR models do exactly this. Why couldn't CLMs do the same?"
  2. "One-hot encodings in DLMs do not need to be trained, since the subsequent embedding lookup table is learnable. In a CLM, then, can learning only the internal embedding replace designing or learning the encoding itself?"
  3. "Since the hidden dimension of the model's internal embedding is often large, is there really any benefit to using a very low-dimensional encoding at the beginning?"

Our theoretical framework explains where these intuitions come from and why they cannot justify directly transferring encoding designs from DLMs to CLMs. Specifically, we distinguish the representation role that encoding plays in both CLMs and DLMs from the target-defining role unique to CLMs, and explicitly locate the latter within the training objective. This explains the theoretical basis for both the similarities and the differences in how encoding affects DLMs and CLMs.

Therefore, the shared representation role explains the source of these intuitions, while the target-defining role unique to CLMs explains why directly transferring them is problematic. The experiments in Section 3.2 further confirm that the target-defining role causes encoding to affect CLMs very differently, supporting our theoretical distinction. This theory naturally motivates and guides the systematic experimental study of CLM encoding in Sections 4 and 5. This distinction also supports using separate terms for encoding and embedding in CLMs: in addition to providing a representation, encoding defines the continuous target, whereas the internal embedding does not serve this latter function.

Overall, our core contribution is a unified theoretical framework that explains why encoding plays different roles in continuous and discrete language models, together with controlled experiments that translate this distinction into concrete design guidance. Our decomposition can also explain the effectiveness of specific designs, such as spherical and low-dimensional encodings, providing theoretical support for previous empirical findings. Please see our response to your Question 2 for details. Thank you for giving us the opportunity to clarify these points.

AuthorsOpenReview ↗

Response to Reviewer mcNi (2/7)

Weakness 2:

Theoretically, the decomposition does not by itself establish an encoding-dependent optimum. Theorem 1 is a valid chain-rule identity that holds for every (θ,η)(\theta,\eta), so it says nothing about whether inf⁡θLCLM\inf_\theta \mathcal{L}_{\mathrm{CLM}} depends on η\eta. Decomposing in the other order gives

\mathcal{L}_{\mathrm{CLM}}=H(P)&+\mathrm{KL}\\!\left(p_\eta(z_{0:T}),q(z_{0:T})\right)\\\\ &+\mathbb{E}_{p_\eta}\mathrm{KL}\\!\left(p_\eta(X\mid z_0),D(X\mid z_0)\right). (*) \end{aligned}$$ > With an exact path model and an unrestricted decoder $D$, this attains $H(P)$ for any encoding, just as for DLMs. I would suggest stating carefully about theoretical rigor and novelty here. Thank you for your question. **Under unlimited model capacity and perfect optimization, both your decomposition $(*)$ and our decomposition in Theorem 1 indeed attain the encoding-independent optimum $H(P)$. However, in practice, model capacity is finite and optimization is imperfect, so this ideal situation is often difficult to attain.** In this setting, we need to consider not only the theoretical optimum but also the form of the objectives being optimized. Our decomposition is intended precisely to describe how encoding influences these objectives in concrete optimization problems. **Specifically, our decomposition in Theorem 1 examines how encoding enters model training through the gap terms between the NELBO and the discrete, rather than continuous, KL divergence $\mathrm{KL}\\!\left(P\\,\\|\\,Q_{\theta,\eta}\right)$.** We isolate the discrete KL because the ultimate goal of any language model is to learn a discrete distribution $Q_{\theta,\eta}(X)$ that approximates the discrete target distribution $P(X)$, as illustrated in Figure 1. Therefore, the discrete KL is the objective we ultimately wish to minimize. The gap terms mean that the gradient of the CLM training loss does not necessarily align with the gradient of the discrete KL, introducing additional difficulties in optimizing $Q_{\theta,\eta}(X)$. Thus, although your decomposition $(*)$ is mathematically correct, it does not isolate the discrete KL and therefore does not reveal the role of encoding as explicitly as Theorem 1 does. **Next, examining the specific forms of the gap terms reveals how encoding enters CLM training through its distinctive target-defining role, and how this differs from its representation role in discrete models.** Specifically, the key distinction is that the parameters $\theta$, which include the model's internal embedding layers, directly participate in representation but do not participate in defining the target. As model capacity increases, the parameters $\theta$ can take over the representation role of encoding quite effectively. In contrast, the effects of the target-defining role can only be reduced indirectly through training on the distribution-matching losses represented by the gap terms. Consequently, a poor encoding can substantially hinder model learning. Even in your decomposition $(*)$, this distinctive role remains visible through the presence of $p_\eta$. **The experiments in Section 3.2 strongly support this theoretical distinction.** In a setting where the number of parameters in $\theta$ is much larger than both the number of parameters in $\eta$ and the size of the training set, discrete models remain relatively stable under changes in encoding, indicating that their internal embeddings can effectively adapt to these changes. CLMs, however, remain highly sensitive to encoding, indicating that the effects introduced through the target-defining role are more difficult to eliminate by training $\theta$. **These results not only support our theoretical distinction but also show why it is important to examine the actual training objective rather than focusing only on the theoretical optimum, particularly for CLMs.** **Therefore, our framework is not intended to establish an encoding-dependent optimum in the idealized setting. Instead, it analyzes how, in actual training, encoding defines continuous matching tasks and thereby affects the difference between the training loss and the ultimate discrete KL objective, revealing the role of encoding.** Thank you for your question.
AuthorsOpenReview ↗

Response to Reviewer mcNi (3/7)

Question 1:

The posterior-mean input lowers GenPPL while raising ValPPL. How do you interpret this divergence?

This phenomenon arises because GenPPL and ValPPL evaluate different aspects of the model, and this distinction can be understood qualitatively through the gap terms in our NELBO decomposition.

Specifically, as discussed in our response to your Weakness 2, the discrete KL is the objective we ultimately wish to minimize, because it most directly measures how closely the model distribution matches the discrete data distribution. As for GenPPL and entropy, although they may not align perfectly with the discrete KL, both are computed directly on generated text and thus, like the discrete KL, assess actual generation performance.

On the other hand, ValPPL is computed from the average per-token NELBO on the validation set and therefore also incorporates the two gap terms. These terms mean that the gradient of the CLM training loss does not necessarily align with the gradient of the discrete KL, so changes in ValPPL need not track changes in generation metrics. Consequently, ValPPL may not accurately reflect the model's generative ability in certain cases. This is an important insight from our theoretical analysis: encoding affects the gap terms in specific ways, introducing additional gradients that differ from those of the underlying discrete KL objective and thereby influencing learning.

Therefore, improved GenPPL and entropy can coexist with higher ValPPL. This is consistent with the insight highlighted by our theoretical framework.

Question 2:

What is the rationale, theoretical or empirical, for constraining encodings to the unit sphere, which discards radial information? A comparison with unnormalized Euclidean encodings would show whether the geometry and dimension findings depend on the normalization.

Thank you for this thought-provoking question. It raises an important point and helps us further illustrate the concrete value of our theoretical framework. Below, we explain how our decomposition can provide an intuitive explanation for why encoding normalization benefits training, accounting for the advantage of spherical encodings in Section 4.1, and can also explain the inverted-U trend with encoding dimension in Section 4.2.

1. Geometry. Our decomposition explains why encoding normalization benefits training and, consequently, why spherical encodings are advantageous.

To illustrate this, consider the single-token setting and let a=∣z0∣a=\\|\mathbf{z}^0\\| and v=z0/∣z0∣\mathbf{v}=\mathbf{z}^0/\\|\mathbf{z}^0\\| denote the radius and direction of the continuous encoding, respectively. Applying the KL chain rule to the encoding gap GencG_{\mathrm{enc}} in Theorem 1 separates it into matching the radial distribution and matching the directional distribution conditional on the radius:

G_{\mathrm{enc}} &= \underbrace{\mathbb E_X\mathrm{KL}\\!\left(e_\eta(a\mid X)\\,\\|\\,q_{\theta,\eta}(a\mid X)\right)}_{G_{\mathrm{rad}}}\\\\ &\quad+ \underbrace{\mathbb E_{X,a}\mathrm{KL}\\!\left(e_\eta(\mathbf{v}\mid a,X)\\,\\|\\,q_{\theta,\eta}(\mathbf{v}\mid a,X)\right)}_{G_{\mathrm{ang}}}. \end{aligned}$$ In common continuous diffusion language models, given the current codebook, the encoding density is obtained by adding isotropic Gaussian noise to a deterministic encoding vector: $$e_\eta(\mathbf{z}^0\mid X) = \mathcal N\\!\left(\mathcal E_\eta(X),\sigma_0^2\mathbf{I}_d\right).$$ Because the Gaussian noise is isotropic, the distribution of the encoding norm after adding noise depends only on the norm of the original encoding vector, not its direction. Therefore, when all encoding vectors have the same norm, the conditional radial distribution $e_\eta(a\mid X)$ is identical for every token. **This makes the radial target in the encoding gap independent of token identity: the model can learn a single shared radial distribution instead of a different one for each token. Spherical encoding therefore removes the need for token-specific radial matching and simplifies the learning task specified by the encoding gap.** We also find experimentally that removing the encoding norm constraint degrades generation performance, which is consistent with this analysis. Please see our response to your Weakness 3 for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.