- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BinaryDLM, a diffusion language model that uses learned binary token representations to preserve partial token identity during corruption. The method employs Binary Spherical Quantization (BSQ) to learn lexically structured binary codes, combines bit-flip corruption with token-level Diffusion Forcing for position-wise heterogeneous noise training, and introduces confidence-based token anchoring and early stopping for iterative decoding. Experiments cover models trained from scratch at scales ranging from 0.3B to 4B, as well as adaptation of a pretrained Qwen3-8B model. Results show that BinaryDLM outperforms masked diffusion and autoregressive baselines on GSM8K and achieves competitive performance across multiple downstream tasks.
Critical strengths
After reading this paper, I recognize two core strengths.
- The paper introduces a simple yet effective learned binary token representation that preserves partial token identity during diffusion. The integration of token-level Diffusion Forcing and confidence-based token anchoring further enables the model to exploit partially preserved information during training and inference. The overall design is intuitive and coherent, with ablation studies demonstrating the effectiveness of the learned binary codes and heterogeneous noise training.
- The paper provides systematic experiments across five model scales ranging from 0.3B to 4B, showing consistent improvements over masked diffusion and autoregressive baselines. Additional experiments on data-constrained training, decoding configurations, and adaptation of a pretrained Qwen3-8B model further demonstrate the method's effectiveness under different training conditions and its scalability to larger models.
Critical weaknesses
While the proposed method is simple and effective, I have two main concerns regarding whether the experimental evidence sufficiently supports its central contributions.
- The central claim of the paper is that learned binary representations preserve lexically structured partial token identity during diffusion, thereby improving denoising performance. However, the main comparisons with masked diffusion simultaneously change the corruption representation, noise sampling strategy, loss weighting, and decoding strategy. Although the proposed method achieves substantial performance improvements, the current experiments do not sufficiently disentangle the contributions of these factors. In particular, the ablation results show considerable gains from near-clean training, suggesting that the training strategy itself may account for a significant portion of the improvements. Consequently, the evidence remains insufficient to establish the extent to which the binary representation itself drives the observed performance gains.
- The paper presents the lexically structured learned binary codebook as a key innovation and demonstrates its effectiveness against random binary codes. However, prior works such as MDM-Prime-v2, BitLM, and CoBit have already explored binary representations or mechanisms for preserving partial token information. The paper lacks direct experimental comparisons with these approaches and does not include stronger fixed or structured binary codes as baselines. Therefore, while the existing results demonstrate that learned binary codes outperform random codes, they do not sufficiently establish the distinctive advantages of the proposed representation over existing binary diffusion approaches. This leaves a gap between the paper's central novelty claims and the supporting empirical evidence.
Questions for authors
I think the paper has a good idea and shows promising results, but I still have some concerns about its main contribution. If the authors can address these questions, especially the first two, I would be happy to raise my score.
- BinaryDLM differs from masked diffusion in several aspects, including the corruption representation, noise sampling strategy, loss weighting, and decoding strategy. Could the authors clarify which existing experimental results specifically support the contribution of the learned binary representation, rather than the accompanying training or decoding techniques? In particular, given the substantial improvements from near-clean training, how do the authors assess the independent contribution of the binary representation?
- Prior works such as MDM-Prime-v2, BitLM, and CoBit have already explored binary representations or partial token information preservation. Could the authors clarify the specific advantages of BinaryDLM's learned binary codebook over these approaches? Is there additional evidence, potentially from existing experiments or checkpoints, that demonstrates the benefits of learned binary codes over competitive fixed or structured binary representations?
- The paper illustrates that certain lexical groups exhibit high bit-level similarity in the learned codebook, while the appendix reports a considerable number of code collisions. Could the authors provide a more comprehensive analysis of the Hamming distance distribution across the vocabulary, the semantic characteristics of collision groups, and token identity recoverability under different noise levels? Such analyses would help substantiate whether the learned codebook captures lexical structures that facilitate progressive denoising.
- BinaryDLM uses different denoising step configurations from the masked diffusion baseline in the main experiments. Although the paper reports early stopping and tokens per forward, could the authors provide actual inference latency or throughput measurements using existing checkpoints, ideally together with accuracy–latency trade-off curves under different denoising budgets? This would help clarify the computational cost associated with the reported performance improvements.
3 comments
Response to Rebuttal
Thank the authors for more analysis and additional ablation studies, which partially address my concerns. However, the contribution of the binary representation remains insufficiently disentangled from the training strategy especially near-clean training , and a clearer evaluation of actual inference efficiency, including latency and throughput, is still missing.
Author response
we thanks the reviewer for their response, we address your raised concerns below.
1. However, the contribution of the binary representation remains insufficiently disentangled from the training strategy especially near-clean training
We respectfully disagree. Our ablations explicitly disentangle the contribution of learned binary representations from near-clean training. Holding the training strategy fixed, replacing learned codes with random codes reduces performance by 5.2–11.3 points, demonstrating the substantial contribution of the learned representation.
More importantly, as explicitly stated in the Introduction (L47–53), our central hypothesis is to preserve and exploit emerging token information throughout denoising through two complementary mechanisms:
- Learned binary representations preserve partial token information, allowing unresolved tokens to provide useful context rather than remaining completely masked.
- Near-clean training and confidence-based token anchoring exploit this emerging information, allowing high-confidence predictions to serve as stable (but revisable) context for resolving harder tokens.
Our ablations directly validate both aspects of this hypothesis:
- Without learned codes, even with near-clean training, BinaryDLM underperforms masked diffusion on GSM8K (38.2 vs. 44.4 at 0.6B; 49.0 vs. 54.3 at 1B) when using pure random binary embeddings. Introducing learned codes raises accuracy to 49.4/60.3, surpassing masked diffusion at both scales.
- Without near-clean training, even with learned codes, performance drops substantially.
Therefore, the substantial gains from near-clean training support, rather than undermine, our central hypothesis. Our contribution is precisely the combination of preserving partial information during denoising from partial noisy tokens and freezing "easy"/predictable token for additional context. The controlled ablations demonstrate that both mechanisms are essential to the effectiveness of the proposed framework, rather than attributing the improvements solely to either component.
2. a clearer evaluation of actual inference efficiency, including latency and throughput, is still missing.
We thank the reviewer for this suggestion. We conducted additional end-to-end latency and throughput benchmarks on a single H100 GPU (batch size 1), using the same 100 GSM8K 5-shot prompts, with up to 256 generated tokens.
| Model / Decoder | Latency (s/question) ↓ | Throughput (tokens/s) ↑ | Forwards/question ↓ | Correct / 100 ↑ |
|---|---|---|---|---|
| AR, greedy (vLLM) | 0.22 | 565 | 125 | 41 |
| Masked, standard | 1.12 | 103 | 135 | 48 |
| Masked, Fast-dLLM τ=0.95 | 0.98 | 122 | 90 | 48 |
| Masked, Fast-dLLM τ=0.90 | 0.95 | 127 | 83 | 46 |
| BinaryDLM | 0.91 | 126 | 81 | 55 |
Compared with optimized AR serving through vLLM, our current diffusion implementation remains slower in wall-clock latency. This is because purely optimization in the serving stack:
- Eager KV cache writes (~56% of latency): our diffusion implementation performs a separate, non-CUDA-graph forward after each block. vLLM integrates cache writes into its optimized forward pass.
- inefficient prompt prefill: our diffusion implementation copies KV cache page by page (with python loop) while vllm uses optimized GPU kernels to write KV tensors directly into the paged
- Less optimized kernels
However, BinaryDLM requires substantially fewer sequential model forwards (81 vs. 125), indicating the potential for further inference optimization.
Author Response
We thank the reviewer for their feedback.
1. Although the proposed method achieves substantial performance improvements, the current experiments do not sufficiently disentangle the contributions of these factors. In particular, the ablation results show considerable gains from near-clean training, suggesting that the training strategy itself may account for a significant portion of the improvements. Consequently, the evidence remains insufficient to establish the extent to which the binary representation itself drives the observed performance gains.
We agree that the main binary–masked comparison changes several factors simultaneously. To disentangle their contributions, we conduct ablations that remove one component at a time from the full BinaryDLM recipe.
We would like to clarify that our central motivation is to preserve and exploit as much emerging information as possible throughout denoising (L47–50). Existing approaches capture different aspects of this principle:
We aims to combine both advantages: binary representations preserve partial token information, while token-level adaptive denoising allows easier tokens to serve as early "drafts" that guide the refinement of harder tokens (L51–53). Near-clean training further supports learning from these partially resolved states.
Our ablations therefore isolate the contributions of learned binary codes, token-level noise, and near-clean sampling to this overall design.
2. Prior works such as MDM-Prime-v2, BitLM, and CoBit have already explored binary representations or partial token information preservation. Could the authors clarify the specific advantages of BinaryDLM's learned binary codebook over these approaches? Is there additional evidence, potentially from existing experiments or checkpoints, that demonstrates the benefits of learned binary codes over competitive fixed or structured binary representations?
Therefore, there are minimal/none structure in these binary. Particularly, some tokens such as 1,2,3,4,... can be close to each other because their index in tokenizer are fairly, but most of the token have random index in the tokenizer. Thus, the performance when these binary embedding would be substantially lower than learned on as in "random binary code" ablation as we report above.
3. BinaryDLM uses different denoising step configurations from the masked diffusion baseline in the main experiments. Although the paper reports early stopping and tokens per forward, could the authors provide actual inference latency or throughput measurements using existing checkpoints, ideally together with accuracy–latency trade-off curves under different denoising budgets?
Because we use early stopping, the 16-step schedule (e.g., 2*block_size) does not use higher NFE than masked diffusion. More concretely, for masked diffusion, we use the optimal/best inference budget i.e., the number of inference steps equals the block size, making the tokens-per-forward (TPF) equal to 1. In practice, BinaryDLM achieves a TPF of roughly 1.3 (for small-scale GSM8k experiments).