acceptodds
Under review as a conference paper at ICLR 2027

Binary Diffusion Language Model

Abstract

Masked diffusion replaces corrupted tokens with the same mask symbol, removing all local identity information. Instead of representing a token with one-hot vector, one can also represent them with binary vectors. A binary code makes corruption gradual: partial evidence about token identity survives at every noise level. We introduce BinaryDLM, a diffusion language model that flips bits in token codes learned through language-modeling objective, so that code similarity reflects lexical similarity. During training, we sample noise independently across positions, mixing near-clean and highly corrupted states so that informative neighbors appear alongside difficult prediction targets. During generation, high-confidence predictions are temporarily anchored as clean context for their neighbors, and blocks are refined iteratively with early stopping. Across 0.3B–4B models trained on a shared math mixture for up to 100B target tokens, BinaryDLM outperforms both masked diffusion (+0.7–10.2 points) and autoregressive baselines (+2.4–8.5 points) on GSM8K accuracy. Adapting pretrained Qwen3-8B to binary diffusion yields competitive performance with state-of-the-art masked diffusion models of the same scale. These results suggest that the corruption representation is a key design axis for diffusion language models, complementary to the denoising architecture and noise schedule.

Primary area:
generative models
Keywords:
diffusion language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aXZGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BinaryDLM, a diffusion language model that uses learned binary token representations to preserve partial token identity during corruption. The method employs Binary Spherical Quantization (BSQ) to learn lexically structured binary codes, combines bit-flip corruption with token-level Diffusion Forcing for position-wise heterogeneous noise training, and introduces confidence-based token anchoring and early stopping for iterative decoding. Experiments cover models trained from scratch at scales ranging from 0.3B to 4B, as well as adaptation of a pretrained Qwen3-8B model. Results show that BinaryDLM outperforms masked diffusion and autoregressive baselines on GSM8K and achieves competitive performance across multiple downstream tasks.

Critical strengths

After reading this paper, I recognize two core strengths.

  1. The paper introduces a simple yet effective learned binary token representation that preserves partial token identity during diffusion. The integration of token-level Diffusion Forcing and confidence-based token anchoring further enables the model to exploit partially preserved information during training and inference. The overall design is intuitive and coherent, with ablation studies demonstrating the effectiveness of the learned binary codes and heterogeneous noise training.
  2. The paper provides systematic experiments across five model scales ranging from 0.3B to 4B, showing consistent improvements over masked diffusion and autoregressive baselines. Additional experiments on data-constrained training, decoding configurations, and adaptation of a pretrained Qwen3-8B model further demonstrate the method's effectiveness under different training conditions and its scalability to larger models.
Critical weaknesses

While the proposed method is simple and effective, I have two main concerns regarding whether the experimental evidence sufficiently supports its central contributions.

  1. The central claim of the paper is that learned binary representations preserve lexically structured partial token identity during diffusion, thereby improving denoising performance. However, the main comparisons with masked diffusion simultaneously change the corruption representation, noise sampling strategy, loss weighting, and decoding strategy. Although the proposed method achieves substantial performance improvements, the current experiments do not sufficiently disentangle the contributions of these factors. In particular, the ablation results show considerable gains from near-clean training, suggesting that the training strategy itself may account for a significant portion of the improvements. Consequently, the evidence remains insufficient to establish the extent to which the binary representation itself drives the observed performance gains.
  2. The paper presents the lexically structured learned binary codebook as a key innovation and demonstrates its effectiveness against random binary codes. However, prior works such as MDM-Prime-v2, BitLM, and CoBit have already explored binary representations or mechanisms for preserving partial token information. The paper lacks direct experimental comparisons with these approaches and does not include stronger fixed or structured binary codes as baselines. Therefore, while the existing results demonstrate that learned binary codes outperform random codes, they do not sufficiently establish the distinctive advantages of the proposed representation over existing binary diffusion approaches. This leaves a gap between the paper's central novelty claims and the supporting empirical evidence.
Questions for authors

I think the paper has a good idea and shows promising results, but I still have some concerns about its main contribution. If the authors can address these questions, especially the first two, I would be happy to raise my score.

  1. BinaryDLM differs from masked diffusion in several aspects, including the corruption representation, noise sampling strategy, loss weighting, and decoding strategy. Could the authors clarify which existing experimental results specifically support the contribution of the learned binary representation, rather than the accompanying training or decoding techniques? In particular, given the substantial improvements from near-clean training, how do the authors assess the independent contribution of the binary representation?
  2. Prior works such as MDM-Prime-v2, BitLM, and CoBit have already explored binary representations or partial token information preservation. Could the authors clarify the specific advantages of BinaryDLM's learned binary codebook over these approaches? Is there additional evidence, potentially from existing experiments or checkpoints, that demonstrates the benefits of learned binary codes over competitive fixed or structured binary representations?
  3. The paper illustrates that certain lexical groups exhibit high bit-level similarity in the learned codebook, while the appendix reports a considerable number of code collisions. Could the authors provide a more comprehensive analysis of the Hamming distance distribution across the vocabulary, the semantic characteristics of collision groups, and token identity recoverability under different noise levels? Such analyses would help substantiate whether the learned codebook captures lexical structures that facilitate progressive denoising.
  4. BinaryDLM uses different denoising step configurations from the masked diffusion baseline in the main experiments. Although the paper reports early stopping and tokens per forward, could the authors provide actual inference latency or throughput measurements using existing checkpoints, ideally together with accuracy–latency trade-off curves under different denoising budgets? This would help clarify the computational cost associated with the reported performance improvements.
3 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for their feedback.

1. Although the proposed method achieves substantial performance improvements, the current experiments do not sufficiently disentangle the contributions of these factors. In particular, the ablation results show considerable gains from near-clean training, suggesting that the training strategy itself may account for a significant portion of the improvements. Consequently, the evidence remains insufficient to establish the extent to which the binary representation itself drives the observed performance gains.

We agree that the main binary–masked comparison changes several factors simultaneously. To disentangle their contributions, we conduct ablations that remove one component at a time from the full BinaryDLM recipe.

We would like to clarify that our central motivation is to preserve and exploit as much emerging information as possible throughout denoising (L47–50). Existing approaches capture different aspects of this principle:

  • Masked diffusion can adaptively reveal easier, high-confidence tokens first, allowing them to provide context for harder tokens. However, each token is either fully masked or fully revealed, preventing the model from exploiting partial information from unresolved tokens.
  • Standard continuous diffusion preserves partial information in intermediate noisy representations. However, it typically denoises all tokens under a shared noise schedule, without explicitly advancing easier tokens ahead of harder ones.

We aims to combine both advantages: binary representations preserve partial token information, while token-level adaptive denoising allows easier tokens to serve as early "drafts" that guide the refinement of harder tokens (L51–53). Near-clean training further supports learning from these partially resolved states.

Our ablations therefore isolate the contributions of learned binary codes, token-level noise, and near-clean sampling to this overall design.

MethodLearned binary codesToken-level noiseNear-clean samplingGSM8K 0.6BGSM8K 1BMATH-500 0.6BMATH-500 1B
BinaryDLM (full)✓✓✓49.460.340.347.9
Random binary codes✗✓✓38.249.031.142.7
Shared timestep✓✗✗24.0—24.0—
Without near-clean sampling✓✓✗24.430.620.333.1
Masked diffusion———44.454.341.239.4

2. Prior works such as MDM-Prime-v2, BitLM, and CoBit have already explored binary representations or partial token information preservation. Could the authors clarify the specific advantages of BinaryDLM's learned binary codebook over these approaches? Is there additional evidence, potentially from existing experiments or checkpoints, that demonstrates the benefits of learned binary codes over competitive fixed or structured binary representations?

  • MDM-Prime-v2: the binary expansion of the token index after a random index shuffle. They shuffle on purpose, to raise per-bit entropy, since BPE index order tracks frequency.
  • BitLM: an 18-bit expansion of the vocabulary index, "a fixed identifier rather than a learned semantic codebook".
  • CoBit: raw tokenizer-ID bits on LM1B and a reversible 16-bit codec on OWT, with continuous Gaussian diffusion on analog bits

Therefore, there are minimal/none structure in these binary. Particularly, some tokens such as 1,2,3,4,... can be close to each other because their index in tokenizer are fairly, but most of the token have random index in the tokenizer. Thus, the performance when these binary embedding would be substantially lower than learned on as in "random binary code" ablation as we report above.

3. BinaryDLM uses different denoising step configurations from the masked diffusion baseline in the main experiments. Although the paper reports early stopping and tokens per forward, could the authors provide actual inference latency or throughput measurements using existing checkpoints, ideally together with accuracy–latency trade-off curves under different denoising budgets?

Because we use early stopping, the 16-step schedule (e.g., 2*block_size) does not use higher NFE than masked diffusion. More concretely, for masked diffusion, we use the optimal/best inference budget i.e., the number of inference steps equals the block size, making the tokens-per-forward (TPF) equal to 1. In practice, BinaryDLM achieves a TPF of roughly 1.3 (for small-scale GSM8k experiments).

Reviewer_aXZGOpenReview ↗

Response to Rebuttal

Thank the authors for more analysis and additional ablation studies, which partially address my concerns. However, the contribution of the binary representation remains insufficiently disentangled from the training strategy especially near-clean training , and a clearer evaluation of actual inference efficiency, including latency and throughput, is still missing.

AuthorsOpenReview ↗

Author response

we thanks the reviewer for their response, we address your raised concerns below.

1. However, the contribution of the binary representation remains insufficiently disentangled from the training strategy especially near-clean training

We respectfully disagree. Our ablations explicitly disentangle the contribution of learned binary representations from near-clean training. Holding the training strategy fixed, replacing learned codes with random codes reduces performance by 5.2–11.3 points, demonstrating the substantial contribution of the learned representation.

More importantly, as explicitly stated in the Introduction (L47–53), our central hypothesis is to preserve and exploit emerging token information throughout denoising through two complementary mechanisms:

  1. Learned binary representations preserve partial token information, allowing unresolved tokens to provide useful context rather than remaining completely masked.
  2. Near-clean training and confidence-based token anchoring exploit this emerging information, allowing high-confidence predictions to serve as stable (but revisable) context for resolving harder tokens.

Our ablations directly validate both aspects of this hypothesis:

  • Without learned codes, even with near-clean training, BinaryDLM underperforms masked diffusion on GSM8K (38.2 vs. 44.4 at 0.6B; 49.0 vs. 54.3 at 1B) when using pure random binary embeddings. Introducing learned codes raises accuracy to 49.4/60.3, surpassing masked diffusion at both scales.
  • Without near-clean training, even with learned codes, performance drops substantially.

Therefore, the substantial gains from near-clean training support, rather than undermine, our central hypothesis. Our contribution is precisely the combination of preserving partial information during denoising from partial noisy tokens and freezing "easy"/predictable token for additional context. The controlled ablations demonstrate that both mechanisms are essential to the effectiveness of the proposed framework, rather than attributing the improvements solely to either component.

2. a clearer evaluation of actual inference efficiency, including latency and throughput, is still missing.

We thank the reviewer for this suggestion. We conducted additional end-to-end latency and throughput benchmarks on a single H100 GPU (batch size 1), using the same 100 GSM8K 5-shot prompts, with up to 256 generated tokens.

Model / DecoderLatency (s/question) ↓Throughput (tokens/s) ↑Forwards/question ↓Correct / 100 ↑
AR, greedy (vLLM)0.2256512541
Masked, standard1.1210313548
Masked, Fast-dLLM τ=0.950.981229048
Masked, Fast-dLLM τ=0.900.951278346
BinaryDLM0.911268155

Compared with optimized AR serving through vLLM, our current diffusion implementation remains slower in wall-clock latency. This is because purely optimization in the serving stack:

  • Eager KV cache writes (~56% of latency): our diffusion implementation performs a separate, non-CUDA-graph forward after each block. vLLM integrates cache writes into its optimized forward pass.
  • inefficient prompt prefill: our diffusion implementation copies KV cache page by page (with python loop) while vllm uses optimized GPU kernels to write KV tensors directly into the paged
  • Less optimized kernels

However, BinaryDLM requires substantially fewer sequential model forwards (81 vs. 125), indicating the potential for further inference optimization.

Reviewer_XMjKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Choosing the right diffusion process and the representation is still an open problem for achieving competitive language generation capabilities with diffusion language models. Regarding the diffusion process, initial continuous Gaussian corruption has been replaced by discrete processes, such as masked and uniform corruptions. Indeed, discrete diffusion remains the de-facto standard in large scale state-of-the-art models (e.g., DiffusionGemma, Llada-2.1, Nemotron-Labs-Diffusion) despite continuous flow models showing promise in few-step generation. Concerning the representation, modeling in the V-dimensional logit space is mostly standard. Instead, Chen et al., (2023) proposed to compress the V-dimensional logit space to a binary encoding with ceil(log V) dimensions. This binary encoding has the advantage of representing partially noised states, something that has been leveraged in continuous (Chen et al., 2022; Batzolis et al., 2026) and also discrete domains with masked diffusion by Chao et al. (2025).

This paper proposes Binary DLM that uses discrete uniform diffusion with binary encodings. Tokens are mapped to binary codes using a Binary Spherical Quantization bottleneck which maintains semantic similarity. The noising process is then defined by randomly flipping bits within each position using per-position noise levels, where some positions are kept as clean anchor tokens without noising. This token anchoring is then also used in decoding, where an increasing number of tokens is not renoised. Overall, the resulting model is a block diffusion model with fine-grained uniform diffusion.

The paper first validates the design choices by training 0.3B-4B models on math (up to 100B tokens). On GSM8K, the binary diffusion outperforms an autoregressive baseline as well as masked diffusion. Finally, the paper then adapts a pretrained Qwen3-8B to binary diffusion through a multi-stage training procedure. With a large decoding budget, the Binary DLM achieves competitive performance against similar sized masked diffusion models in math and coding.

Critical strengths

Using a binary code driven by the Binary Spherical Quantization bottleneck (Zhao et al., 2024) seems to be an effective representation for uniform diffusion, though learnable binary encodings have already been used for continuous binary diffusion by Kinakh & Voloshynovskiy (2025).

Critical weaknesses

Missing comparison to prior arts

My main concern is the missing comparison to prior arts both in terms of representation and diffusion process.

a) Representation: Given the diffusion process (i.e., uniform), the paper should compare to different representations. The standard V-dimensional logit space is the most obvious one. Given that language modeling is well-established (e.g., OWT), comparing the Binary DLM with uniform diffusion models (e.g., DUO, GIDD) would be most straight forward. For comparisons on large scale, uniform DiffusionGemma and masked Llada-2.1 (with token-to-token edits) should be taken into account too.

b) Diffusion process: Given the prior art on diffusion using binary encodings, the paper should accordingly compare to them. CoBIT (Batzolis et al., 2026) using flow matching would be a good entry point (code is available). Moreover, masked diffusion (Chao et al., 2025) for binary codes should be compared as well. Finally, the small-scale experiments on GSM8K should compare to a uniform diffusion model instead of a masked diffusion model.

Increased computational cost

My second concern is the computational cost due to the involved higher number of function evaluations (NFEs). Different methods should always equalize the number of NFEs (or report the accuracy vs. NFE frontier). However, both the small scale (2x more NFE) and in the large scale (up to 3.125x more NFE) experiments allow a considerably higher NFE budget for Binary DLM than the masked baselines. I acknowledge the 2.49 tokens per forward pass on MATH-500 (Figure 7), but this should be reported across all tasks and models. The current reported evidence points to a notably higher compute spent to achieve the results.

Questions for authors

Besides resolving my main concerns regarding comparison to prior arts for representation and diffusion process, as well as addressing the increased computational cost, I have the following comments:

  1. How do you compute the token probability in Eq. (3)? Is it the joint probability over the full code, or the product of individual binary codes? How do you cope with infeasible binary combinations at the readout (see also Chen et al., 2022).
  2. Having per token noise levels has already been explored for continuous diffusion [1].
  3. At times, the paper has many grammatical errors. E.g., see Section 4.3 and Section 4.4.
  4. Many well-established references are missing the correct venues. Instead, only the arXiv preprints are cited, e.g., (Arriola et al., 2025; Chen et al., 2022; Hersche et al., 2025; Li et al., 2026; Liu et al., 2025; etc.)

[1] Bansal, P. and Sanghavi, S., 2026. Token Time Continuous Diffusion for Language Modeling. arXiv preprint arXiv:2607.14106.

4 comments
AuthorsOpenReview ↗

Author response

We thank the reviewer for their feedback. We address their questions and concerns below.

1. Missing comparison to prior arts

We additionally evaluate uniform diffusion on GSM8K and MATH-500 under the same training setup.

ModelGSM8K 0.6BGSM8K 1BMATH-500 0.6BMATH-500 1B
AR (greedy)43.050.338.646.4
Masked diffusion44.454.341.239.4
Uniform diffusion35.950.330.938.8
BinaryDLM49.460.340.347.9

MATH-500: evaluated with math_verify over 3 decoding seeds; uniform diffusion uses 16 steps per block. We will add suggested baselines to the paper accordingly, it's worth noticing that diffusiongemma/llada 2.1 has larger scale compare to ours.

2. Increased computational cost

  • Although BinaryDLM uses a maximum of 16 steps per block (i.e., 2*block_size across all experiments), early stopping reduces the actual number of function evaluations (NFEs). For masked diffusion, we use the best-performing inference budget, where the number of steps equals the block size, resulting in 1 token per forward pass (TPF). In comparison, BinaryDLM achieves a TPF of approximately 1.3 in our small-scale GSM8K experiments.

3. How do you compute the token probability in Eq. (3)? Is it the joint probability over the full code, or the product of individual binary codes? How do you cope with infeasible binary combinations at the readout (see also Chen et al., 2022).

  • We clarify that the output head predicts standard vocabulary logits rather than individual bit logits. Thus, Eq. (3) defines a categorical distribution over vocabulary tokens, and invalid binary combinations do not arise at the output. Binary codes are used only at the input, where a linear layer maps each 64-bit vector to the model's hidden dimension.

4. Writing

  • We thank the reviewer for pointing out missing baselines, related work, and grammatical errors. We will address these issues in the revised paper.

I want to thank the authors for the response and the additional experiments. While I see value in the overall concept of performing uniform diffusion on semantic binary encodings, the empirical evidence does not yet support it.


Missing comparison to prior arts

Thank you for adding the uniform diffusion baseline. I am not sure, however, if the reproduction is representative. E.g., Sahoo et al. (2026) showed competitive performance against AR and masked diffusion using the DUO formulation. Actually, their model checkpoints are already available (including an AR one), this would allow for a fair comparison at reasonable scale (starting with their AR model and finetune it to a BinaryDLM one).

Beyond the representation, I'm still missing the comparisons against flow matching and masked diffusion for binary codes. As already stated, doing this in language modeling would be easiest, as those baselines already provide results in these domains.

Increase computational cost

Even though TPFs are now reported for GSM8K as well, I'm missing a fair comparison against the diffusion baselines. As BinaryDLM uses confidence-based anchoring, the baselines should be doing so as well (e.g., through confidence-based unmasking). There is a rich literature on unmasking schedules (starting with Wu et al., 2026), and these should be taken into account.


Finally, it would be good to update the manuscript with the additional experiments and elaborations. I cannot see any changes so far.

Sahoo et al., "Scaling Beyond Masked Diffusion Language Models," ICML 2026. Wu et al., "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding," ICLR 2026.

AuthorsOpenReview ↗

Author response

We thank the reviewer for their follow-up. We have conducted additional controlled experiments to address both concerns regarding adaptive masked-diffusion decoding and comparisons with prior binary diffusion methods.

1. Even though TPFs are now reported for GSM8K as well, I'm missing a fair comparison against the diffusion baselines. As BinaryDLM uses confidence-based anchoring, the baselines should be doing so as well (e.g., through confidence-based unmasking). There is a rich literature on unmasking schedules (starting with Wu et al., 2026), and these should be taken into account.

We implement and extensively sweep two established training-free adaptive samplers on our masked-diffusion checkpoints: 1) Fast-dLLM's confidence-threshold (Wu et al., 2026) and 2) EB-Sampler (Ben-Hamu et al., 2025)

All methods use the same training data, backbone, and training budget. We report accuracy and tokens per forward (TPF), including additional forward for KV cache (hence masked diffusion TPF is <1)

DecoderGSM8K 0.6BGSM8K 1BMATH-500 0.6BMATH-500 1B
Masked, fixed schedule40.6 (0.89)50.7 (0.89)37.2 (0.89)38.6 (0.89)
Fast-dLLM τ=0.9944.4 (1.17)54.3 (1.18)41.2 (1.30)39.4 (1.29)
Fast-dLLM τ=0.9544.7 (1.37)54.1 (1.37)41.4 (1.54)39.4 (1.54)
Fast-dLLM τ=0.9044.4 (1.49)54.0 (1.50)40.2 (1.68)39.4 (1.68)
EB-Sampler γ=0.0144.6 (1.10)53.8 (1.09)41.6 (1.25)39.8 (1.19)
BinaryDLM49.4 (1.41)60.3 (1.45)40.3 (1.58)47.9 (1.59)

Each entry reports accuracy (TPF). We additionally swept Fast-dLLM thresholds from 0.7–0.99 and EB-Sampler γ from 0.01–1.0.

Importantly, even after applying competitive adaptive decoding to masked diffusion, BinaryDLM retains a substantial advantage at comparable inference efficiency.

These results demonstrate that BinaryDLM's improvements cannot be explained simply by using confidence-based decoding or a larger inference budget. Even with strong adaptive unmasking baselines, BinaryDLM maintains a substantial advantage.

2. Beyond the representation, I'm still missing the comparisons against flow matching and masked diffusion for binary codes. As already stated, doing this in language modeling would be easiest, as those baselines already provide results in these domains.

For continuous flow matching on binary diffusion, we add our implementation for CoBit baseline below. We implemented CoBit following its reference code: raw binary token codes, Gaussian diffusion , the EDM-weighted bit loss, self-conditioning. We changed only what a controlled comparison requires. It uses our backbone, data, 100B-token budget and block-diffusion.

ModelGSM8KMATH-500
BinaryDLM49.440.3
Masked diffusion40.637.2
AR43.038.6
CoBit, 16 steps/block9.99.5
CoBit, 64 steps/block15.519.0
CoBit, 256 steps/block22.322.3

Consistent with our results in Figure 3, using uniform noise schedule + random/unstructured binary embedding leads to severe degradation.

These findings are consistent with our central hypothesis (L47–53): preserving partial token information is insufficient unless the model can effectively exploit it. BinaryDLM combines semantically learned binary representations with heterogeneous near-clean training and confidence-based token anchoring for additional "context".

I want to thank the authors for their response and for generating additional results.

Increased computational cost

Thank you for providing the additional unmasking baselines (Wu et al., 2026; Ben-Hamu et al., 2025). This puts BinaryDLM as a viable competitor for fast and accurate generation in reasoning.

One small note: optimized block-diffusion fills the KV cache in the same pass as starting denoising the next block; hence, no additional forward pass would be needed. This will not change the relative ordering of the methods; it will rather improve the TPF number for all of them.

Missing comparison to prior arts

Thank you for aiming to integrate CoBit into math reasoning. As with the previously reported uniform diffusion baseline, I cannot judge if this comparison is representative. A more viable comparison would be to consider an established setup, and compare with published (best case reproducible) results on this setup. For a comparison to CoBit, this could be for example evaluating BinaryDLM on OWT/LM1b, or the same TinyGSM setup as CoBit (same models, training data etc.; it seems the code is available in one of the GitHub branches). Similarly, the DUO checkpoints (Sahoo et al., 2026) would enable a fair uniform diffusion comparison. Finally, the comparison against masked diffusion on binary codes (Chao et al., 2025) is still open.

Overall, I still appreciate the general concept of uniform diffusion on semantic binary encodings. Unfortunately, the current evaluations, in a single domain and relying only on self-generated baselines, do not allow me to sufficiently compare the proposed method against established approaches. Thus, I keep my score. I encourage the authors to pursue working on this, and I hope that my suggestions provide some guidance.

Reviewer_ec4POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces BinaryDLM, a discrete diffusion model that works on bits. Binary code similarity is supposed to reflect lexical similarity, which is archived by applying Binary Spherical Quantization with 48 bits per token on top of pretrained Qwen3-4B.

Consistent validation is performed on GSM8K.

Critical strengths

Interesting bit-level discrete diffusion model where binary code similarity reflects lexical similarity.

Critical weaknesses

Methodology:

All controlled from-scratch experiments only evaluated on GSM8K. Specifically the comparison to AR and uniform masked in (Figure 4, 5, 6).

Figure 3 is inconsistent, uses strict extraction.

No held-out likelihood/perplexity (PPL), no general-language task, no generative PPL. We understand the criticism on generative PPL, but there should be some alternative which evaluates the quality of generating text.

Consistent comparison between archs (Fig 4): not really matched in compute: block-diffusion concatenates clean + noise, roughly double FLOPs than AR. Also pretraining leakage due to relying on Qwen3 embeddings.

Table 1 doesn't really tell anything. Those are all trained under different conditions, different model sizes. Also, two important baselines missing: plan Qwen3-8B, masked-diffusion adaptation of Qwen3-8B.

More ablations should be done.

Questions for authors

What about alternative approaches to get the binary codes, e.g. binarizing the original Qwen embeddings post hoc, Lookup-Free Quantization (LFQ)?

"uniform masked" - what does this exactly mean? What is uniform?

Appendix A - where is it?

5 comments
AuthorsOpenReview ↗

Reviewer response

1. All controlled from-scratch experiments only evaluated on GSM8K. Specifically the comparison to AR and uniform masked in (Figure 4, 5, 6).

  • We've run the 0.6B/1B 100B token training on dolma3-dolmino dataset for AR, masked diffusion and binary diffusion using similar setup in the paper. We report their performance across benchmarks as below table. Binary diffusion performs on par with AR and beats masked diffusion consistently
SizeModelGSM8KASDivMATH-500HumanEvalMBPPAvg
0.6BAR26.558.710.816.516.025.7
0.6BMasked27.154.910.815.214.024.4
0.6BBinary29.556.813.515.416.926.4
1BAR39.066.419.222.626.034.6
1BMasked37.862.214.020.120.030.8
1BBinary41.764.519.020.323.733.8

2. Consistent comparison between archs (Fig 4): not really matched in compute: block-diffusion concatenates clean + noise, roughly double FLOPs than AR. Also pretraining leakage due to relying on Qwen3 embeddings.

  • We did not claim our method beat AR with same training FLOPs. What we find is that, binary diffusion performs on par or better than AR with same training tokens budget. In addition, we use standard qwen3 architecture for simplicity but training FLOPs of diffusion language model can be reduced significantly if we use encoder-decoder architecture as in [1,2,3]. Thus, this is not inherent limitation of language diffusion nor our proposed method.
  • For the leakage part, we don't see any problem with our proposed approach. We train a binary embedding for one-time and used it across all experiments. This is conceptually equivalent to any latent diffusion with pretrained VAE (for text/video/image in general).

3. Table 1 doesn't really tell anything. Those are all trained under different conditions, different model sizes. Also, two important baselines missing: plan Qwen3-8B, masked-diffusion adaptation of Qwen3-8B.

  • That's the standard in current published diffusion language model literature [1,2,4,5]. It's virtually impossible to have a "fair"/controlled experiments across all methods as most papers used their own closed-source/in-house dataset mixture and training cost is extremely high for reproducing all these baselines. That's why we have the separate controlled experiments setup for "small" experiments (0.6b-4b on math/dolmino dataset mixture). The point of the table 1 is showing diffusion language model perform competitively with AR/other baseline

What about alternative approaches to get the binary codes, e.g. binarizing the original Qwen embeddings post hoc, Lookup-Free Quantization (LFQ)?

  • Since the vocabulary size of existing tokenizers are around 200k, using fewer than 24bit for representation leads to severely collision. LFQ's implementation usually requires high memory if we set the codebook larger than 2^20, hence, we use BSQ.

"uniform masked" - what does this exactly mean? What is uniform?

  • We apologize for the confusion, that's masked diffusion.

Appendix A - where is it?

  • We included the appendix in the supplementary documents.

[1] DiffusionGemma Technical Report

[2] Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

[3] Encoder-Decoder Diffusion Language Models for Efficient Training and Inference

[4] Fast-dLLM v2: Efficient Block-Diffusion LLM

[5] LLaDA2.1: Speeding Up Text Diffusion via Token Editing

Thank you for the answer.

Thanks for the additional evaluation.

I think the abstract should be formulated more fairly given those results.

BinaryDLM outperforms both masked diffusion (+0.7–10.2 points) and autoregressive baselines (+2.4–8.5 points) on GSM8K accuracy.

Don't just limit it to GSM8K. And overall BinaryDLM does not outperform the AR baseline.

Binary diffusion performs on par with AR.

It performs worse on the larger model. Maybe with more scaling, that gap widens further. I think the results should not be oversold.

We did not claim our method beat AR with same training FLOPs.

Yes I understand. But I think this is another relevant experiment.

The point of the table 1 is showing diffusion language model perform competitively with AR/other baseline.

That is reasonable, given that you have separate controlled experiments.

However, table 1 still is lacking: E.g. there are no real AR baselines.

What about adding Qwen3-8B non-thinking and Qwen3-8B thinking?

Is the model trained with reasoning traces?

We train a binary embedding for one-time and used it across all experiments. This is conceptually equivalent to any latent diffusion with pretrained VAE (for text/video/image in general).

No it is not quite. In that example with pretrained VAE, the same pretrained VAE would usually be used across experiments, to have a fair comparison. But here, the binary codebook is only used for your binary diffusion model, but the AR and masked baseline need to learn an embedding from scratch. So only the binary diffusion model somehow gets knowledge transfer from the large pretrained model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.