- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how token encoding affects continuous diffusion language models. This work decomposes the training objectives and shows that encoding represents tokens and defines the continuous target distribution. Experiments on Text8 and grayscale CIFAR-10 show that encoding geometry, dimensionality, and compatibility with the model's input and output parameterization are important design considerations. The paper presents jointly learned low-dimensional sphere encodings with probability first outputs and the 130M parameter model achieves low generative PPL on OWT, outperforming diffusion baselines.
Critical strengths
- Clear explanation of what encoding does and why it matters.
The decomposition of the objectives makes it explicit that encoding defines the continuous target distribution. This provides a useful framework for understanding why encoding choices affect continuous and discrete models differently.
- Experimental results showing practical design choices.
Experiments are designed to identify which encoding properties affect performance: encoding geometry and dimensionality are important. OWT experiments demonstrate how these combined with input and output parameterizations can improve performance in large vocabulary setting.
Critical weaknesses
- Limited support for the performance advantage.
In the comparison, the diffusion models use different sampling protocols: generation quality depends on the number of sampling steps, the sampling method, and the decoding rule. Lower GenPPL could come from an advantageous sampling protocol, not only from model improvements.
Also, GPT-2 Large based ppl and token-level entropy provide useful evidence but may not fully capture coherence or diversity across generated texts. Additional generation quality metrics would strengthen the practical claims.
Questions for authors
- Does RePlaID use the same training-token budget, sampling steps, terminal decoding, and GenPPL calculation as the proposed method?
3 comments
Thank you for the additional experiments and clarification.
My question about the matched RePlaID setup is addressed: The matched NFE comparisons and MAUVE evaluation address my concerns.
I recommend specifying the sampling algorithm, time discretization, and terminal decoding rule for each baseline in the paper, as matching NFE alone does not fully resolve the sampling-protocol concern.
Also, the paper could be strengthened by discussing why the proposed model’s MAUVE is below RePlaID and LangFlow, and why its advantage over RePlaID does not hold at 64 NFE.
As the major concerns are addressed, I keep my score for weak acceptance.
Further Response to Reviewer mgNr
Thank you for your reply and constructive suggestions. We greatly appreciate the opportunity to further discuss our work with you. Below, we address your two remaining questions.
1. Sampling algorithms, time discretization, and terminal decoding
To further clarify the sampling protocols, we summarize the specific settings used for each model in our evaluation.
(1) Sampling algorithms. We use the official sampling algorithms provided by the respective implementations wherever possible. Specifically:
- Ours and our reimplementation of RePlaID: reverse SDE sampling, each following its own SDE settings.
- LangFlow and ELF-B: official SDE samplers (Euler–EDM for LangFlow; SDE for ELF-B). Together with Ours and RePlaID, all four continuous models therefore use the same broad class of SDE reverse sampling.
- SEDD, MDLM, and BD3-LM B4: reverse samplers of masked diffusion. SEDD and MDLM both discretize this reverse on a time grid (closed-form reverse for SEDD; Euler/ancestral reverse for MDLM). BD3-LM B4 uses a first-hitting reverse of the same mask process.
Self-conditioning is used whenever the official method provides it (Ours, RePlaID, LangFlow, ELF-B). Reported NFEs already count the corresponding extra forwards.
(2) Time discretization. We align NFE budgets at 64, 256, and 1024 wherever supported. For all models that admit a fixed-step sampler, we first discretize the time axis uniformly, then map those times through each model’s official schedule to the corresponding SNR. The schedules are as follows:
- Ours and our reimplementation of RePlaID: learned noise schedules.
- LangFlow: official EDM schedule.
- ELF-B: official logit-normal schedule.
- SEDD and MDLM: official log-linear noise schedules.
- BD3-LM B4: official log-linear noise schedule; first-hitting samples the times fed into this schedule instead of reading it on the uniform NFE grid.
(3) Terminal decoding. We use the corresponding terminal decoding rules of each model:
- Ours, RePlaID, LangFlow, ELF-B, and MDLM use argmax decoding, while SEDD and BD3-LM B4 use Gumbel-max sampling (i.e., taking the argmax of Gumbel-perturbed scores).
Overall, we have aligned the evaluation settings as closely as possible, including the number of generated samples, GPT-2 Large scoring, unigram entropy calculation, and actual NFE budgets. For sampling-specific details, we follow the official implementations of the respective models to ensure faithful comparisons.
2. Additional ablation of posterior-mean input and its relationship to MAUVE and low-NFE sampling
We further conducted an ablation experiment to investigate the relationship between posterior-mean input, MAUVE, and sampling performance under limited NFE budgets. Specifically, we evaluated the variant of our final model without posterior-mean input, and obtained the following results:
| Configuration | 64 NFE: GenPPL / Entropy | 1024 NFE: GenPPL / Entropy | MAUVE |
|---|---|---|---|
| Final (Ours) | 57.2 / 7.38 | 16.0 / 7.38 | 0.772 |
| Final − posterior-mean input (Ours) | 52.1 / 7.40 | 19.7 / 7.31 | 0.853 |
| RePlaID | 53.8 / 7.47 | 23.1 / 7.37 | 0.838 |
| LangFlow | 82.8 / 7.44 | 41.1 / 7.40 | 0.900 |
Interestingly, removing posterior-mean input substantially improves MAUVE and low-NFE sampling performance, while degrading generation performance at high NFE. In particular, its MAUVE score reaches 0.853, comparable to RePlaID and close to LangFlow, and its GenPPL / Entropy at 64 NFE improves to 52.1 / 7.40, close to RePlaID (53.8 / 7.47). However, its GenPPL at 1024 NFE is substantially worse than that of the final model.
We conjecture that posterior-mean input changes how the model utilizes the denoising state during sampling, potentially introducing a trade-off between sampling efficiency at low NFE and generation quality at high NFE. Posterior-mean input may provide a more informative estimate of the clean representation, benefiting the refinement of samples when sufficient denoising steps are available. However, under a limited sampling budget, this additional input may make the model more dependent on accurate intermediate estimates, potentially making coarse discretization less effective. This may also affect the overall distribution of generated samples, as reflected by the MAUVE results. These observations suggest that the choice of input parameterization can influence not only generation quality but also the sampling behavior across different NFE budgets.
Thank you again for your helpful comments. We hope that these additional clarifications and experiments address your remaining concerns. If you find our responses satisfactory, we would sincerely appreciate your consideration of an improved rating.
Response to Reviewer mgNr
Thank you for your thorough review and your positive assessment of our work. We appreciate the opportunity to further improve our work through this discussion. Below is our point-by-point response.
Weakness 1:
We have re-evaluated our method and continuous and discrete diffusion model baselines under a unified evaluation protocol across multiple sampling budgets. All models use the same number of generated samples (N=5000), GPT-2 Large scoring, and the same definition of unigram entropy. We align sampling budgets by the actual number of network function evaluations (NFE). The RePlaID results are from our own reproduction trained for 70k steps, while the other baselines use officially released checkpoints.
BD3-LM B4 is marked as N/A at 64 and 256 NFE because these budgets are not supported by its sampling setup. For Final and RePlaID, the 1024-NFE GenPPL values differ slightly from those in the main text, while the entropy values are noticeably higher, primarily due to the larger sample size of N=5000. At 1024 NFE, our method achieves the lowest GenPPL while maintaining high entropy. At 256 NFE, it achieves lower GenPPL than all evaluated baselines except ELF-B, while maintaining substantially higher entropy than ELF-B. At 64 NFE, our method performs slightly worse than RePlaID but remains competitive among the evaluated baselines. Overall, these results demonstrate a favorable trade-off between generation quality and diversity under a unified evaluation protocol, particularly at 256 and 1024 NFE. We have added these results to Appendix D.6.
Weakness 2:
Following your suggestion and that of Reviewer x4y1, we have added MAUVE evaluation to complement GenPPL and unigram entropy by comparing the distributions of generated and real text. We reuse the generated texts from the 1024-NFE setting above and compute MAUVE using the same held-out OpenWebText validation set as the reference corpus for all models, with GPT-2 Large as the feature extractor.
The MAUVE results show that our model achieves a higher score than most of the evaluated baselines, although it remains below RePlaID and LangFlow. We have added these results to Appendix D.7.
Question 1:
Thank you for your question. Our RePlaID reproduction uses the same 70k training steps, same training-token budget, sequence length n=1024, sampling steps, terminal decoding (MAP), and GPT-2 Large GenPPL calculation as the BFN model in our main experiments. In our response to your Weakness 1, we provide additional matched comparisons across different numbers of sampling steps, which support the advantage of our model.
This concludes our point-by-point response. Thank you again for your careful review. We welcome further discussion if you have any additional questions. We hope that these clarifications and additional results address your concerns, and we kindly ask you to reassess our work and score in light of the new material.