acceptodds
Under review as a conference paper at ICLR 2027

A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models

Abstract

Can a diffusion language model generate a coherent token block in one forward pass? Masked models already predict every position at once, but each prediction is the marginal distribution given the visible context, so the tokens can be mutually inconsistent and later steps revise those already committed. We introduce CONDOR (Coupled-Noise Distillation for One-Step Readout), trained from scratch to map different noise samples to different coherent blocks. Initially, random noise is not naturally paired with a target. Winner-take-all supervision lets different samples specialize, and self-distillation trains the one-pass output to match the refined coherent block. TinyStories experiments show diverse, coherent continuations over successive blocks, one forward pass each. Qualitative MNIST experiments show that the same approach can extend to multimodal generation, such as text-to-image and unconditional text-and-image generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Models, Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eLLqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CONDOR, a method for single-step discrete text diffusion. The method proposed first samples a noise to seed the generation process, and second outputs a coherent block based on this noise seed. To encourage the model to output coherent text blocks instead of uniform random sampling as would naively happen in a block-diffusion method, the authors propose two stages of training. In the first stage, the diffusion language model trains solely on the minimum loss token. In the second stage, the minimum loss tokens are sampled repeatedly using a confidence score and fed back into the model to create coherent target data under the models distribution for consistency. The authors evaluate the method at a small scale on several tasks, most importantly text continuation and label-to-MNIST generation.

Critical strengths

The task being addressed is important, and single-step diffusion language models have a clear motivation and the area is interesting to prior work.

The proposed method is moderately novel, although confidence-based diffusion language model sampling for few step generation has been done before.

Critical weaknesses

First, I will start with soundness concerns. There is no connection between the noise seed and the generation outputs. Although the noise seed is added to the masked inputs, the mask tokens provide no mutual information to the model about the output, whereas noise in traditional diffusion masks visible data. Because the visible tokens receive no noise there is no justification for this point. Because the network is small scale, large noise that are out of distribution will of course harm the generation quality as seen in Table 3. This doesn't indicate a purpose for the noise seed itself.

Second, the minimum loss measure for generation has a strong soundness concern. Unlike single step generation methods based on confidence, which is a deterministic function of the data, the minimum loss is stochastic. A model does not know which token will have the minimum loss. Therefore, the minimum token being sampled is stochastic, invalidating a key setting for most self-distillation work, even though the decoding is greedy. Furthermore, backpropagating only on the minimum loss token violates standard langugage modeling. The minimizer for this loss is not necessarily a distribution. Take the distribution on two characters either a,a or b,b. Then the optimal loss objective predicts a for one token with all probability and b for a different token.

Third, the results are difficult to interperet and generally low quality. I focus here on the MNIST experiment, where the outputs are excessively blurry. MNIST is a toy dataset and outputs should be nearly perfect, highlighting the soundness concerns.

Questions for authors

What is the purpose of the noise seeding?

Why is minimum token loss an appropriate measure here, when the model itself cannot know what token will have the minimum loss?

Why aren't the MNIST outputs sharp?

1 comment

Thank you for the review. Several soundness concerns appear to stem from reading the WTA winner in Eq. (4) as a single token. Prior work already explains how WTA trains a complete candidate and demonstrates how independent noise draws can yield different outputs for the same context.

1. How does the seed affect the output? Winner-take-all (WTA) trains the closest of k predictions toward the target. Guzmán-Rivera et al. (2012) formalized this objective in Multiple Choice Learning: Learning to Produce Multiple Structured Outputs. In Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles, Lee et al. (2016) backpropagated each example’s loss through its winning predictor. For each of k noise draws, Eq. (4) sums cross-entropies over all masked positions. Every masked position in the lowest-loss draw contributes to the update.

In Self-organized formation of topologically correct feature maps, Kohonen (1982) updated the winning unit and its neighbors. Theorem 1 of Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses (Rupprecht et al., 2017) shows that under squared error, optimal predictors represent the means of their assigned targets. In Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks, Gupta et al. (2018, Eq. 5) trained on the closest of k future trajectories generated from independent noise for the same motion history.

Suppose the only valid blocks are aa and bb. On a training example labeled aa, four noise draws might decode to ab, bb, ba, and bb. Suppose the draw that produced ab has the lowest total cross-entropy against aa this time. It wins under Eq. (4), and both positions are trained on a, pushing the second position toward a as well.

Even a winning draw that already decodes to aa can improve. If it assigns a probabilities 0.8 and 0.9, its loss is still positive. Cross-entropy pushes both probabilities toward 1 for that draw. On bb examples, other draws can specialize in bb. Repeating these updates can make mixed blocks such as ab and ba rare while preserving both valid outputs. In our earlier four-token experiment, aaaa, bbbb, cccc, and dddd each accounted for about a quarter of the outputs. We observed no other sequence.

On TinyStories, we held each prefix fixed and greedily decoded one block for each of 16 noise draws. CONDOR produced an average of 11.31 distinct valid continuations per prefix, compared with 0.25 for plain cross-entropy (Table 2). Even without distillation, WTA alone (k=4) raised this count to 1.19.

2. “The minimum loss measure for generation.” We cannot tell which step this phrase refers to. Could you clarify whether your soundness concern applies to the WTA training loss, the construction of self-distillation targets, or inference with the trained model? WTA selects the noise draw with the lowest total loss over the masked positions (Eq. 4). To construct a distillation target, the student takes the vocabulary argmax at each position and the teacher greedily refills selected positions (§3.2). At inference, we draw noise once per block and take the vocabulary argmax at each position in a single forward pass (§3.3).

MNIST sharpness. Some strokes in Figure 3 are blurry. We included MNIST as a simple example of generating an image in one forward pass given a digit word. Each image token represents a 7×7 patch and predicts all 49 pixel values together (§3.3). We did not tune the hyperparameters or refine the image decoder specifically for sharpness.

Reviewer_ax9FOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CONDOR, a noise-conditioned masked diffusion model for generating an entire token block in one forward pass. The method adds Gaussian noise to mask embeddings and combines winner-take-all (WTA) supervision with same-noise self-distillation. WTA supervises the lowest-loss prediction among multiple noise samples, while self-distillation trains one-pass predictions to match targets obtained through iterative refinement. Experiments cover single-block and multi-block continuation on TinyStories, together with qualitative text-to-image and unconditional text-and-image generation on MNIST.

Critical strengths
  1. The paper explores image generation in addition to text generation. Although the MNIST evaluation is qualitative, it provides an initial demonstration that the proposed mechanism can be applied to another modality.

  2. The evaluation considers generation diversity alongside validity. Reporting distinct valid continuations and examining the effects of noise help assess whether improved one-step generation comes at the expense of output diversity. The independent-target-noise ablation is particularly useful for understanding this trade-off.

Critical weaknesses
  1. The relationship to closely related work and the noise design need clarification. Augmenting masked representations with continuous variables is closely related to CADD [1], which is not cited or discussed. Although the training mechanisms differ, the paper should explain this connection and distinguish its contributions in noise–output coupling and one-step distillation. Additionally, Eq. (2) uses a fixed relative noise scale, σ=0.5\sigma=0.5, across different mask ratios. The rationale for keeping this scale fixed across denoising stages is insufficiently explained.

  2. The baseline evaluation is incomplete. Plain CE is evaluated only at NFE=1. This is informative for comparing direct one-step generation, but does not adequately establish the quality–efficiency trade-off against standard iterative decoding. At least a Plain CE baseline with NFE=8 should be included, especially since an eight-pass WTA baseline is already reported.

  3. WTA alone provides little improvement in one-step validity at the default competition size. At NFE=1, WTA with k=4k=4 achieves 9.2% validity, compared with 9.0% for Plain CE. Although diversity improves, WTA alone does not substantially resolve the one-step coherence problem in this setting. Increasing kk to 16 improves validity to 18.4%, but also increases training cost from 5.3 to 8.3 GPU-hours, according to Appendix A.1. These results suggest that the substantial validity gains depend primarily on adding self-distillation.

  4. A quality gap remains relative to multi-step generation. The eight-pass WTA baseline achieves the highest validity and number of distinct valid continuations in Table 2: 97.3% and 14.91, respectively. The CONDOR configuration with the highest validity reaches 93.9% and 11.62. Thus, one-step generation reduces the number of forward passes but retains a measurable quality and valid-diversity gap relative to this multi-step baseline. This trade-off should be stated more explicitly.

[1] Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling https://arxiv.org/abs/2510.01329

Questions for authors
  1. For unconditional text-and-image generation, how is the digit word represented and generated? Is each digit name a single categorical token, or is it tokenized into subwords? What defines the text block and its termination, and how many forward passes are required to generate a complete word–image pair?
  2. The paper specifies that an image consists of sixteen patch tokens with continuous pixel outputs. How are text tokens and image patches embedded and decoded within the model? How are the text cross-entropy and image MSE objectives combined and weighted during joint training?
  3. How sensitive is performance to the training noise scale (σ\sigma)? Would adjusting it according to the mask ratio improve validity or diversity? The inference-time radius sweep does not directly answer this training-design question.
  4. How does the method behave as the task and model scale increase? In particular, does the validity–diversity trade-off remain favorable with larger blocks and more complex datasets beyond TinyStories and MNIST?
1 comment

Thank you for the care you put into this review. We especially appreciate you pointing us to CADD and explaining why varying the noise at inference leaves the question about training noise open. Both comments helped us see where the paper needed a clearer answer.

We will cite Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling (CADD), which adds noise to a token’s continuous representation. CONDOR samples noise independently of the token and uses the same noise to generate the draft and its distillation target.

MNIST. Each digit word (e.g., one, two, three) is a single categorical token and completes the text block. Unconditional generation takes two forward passes, one for the word and one for all 16 image tokens in a 4×4 grid. Both formats we tried worked. The 16 positions after the word are reserved for the image in one format. The other encloses them between two <img> markers. The text head predicts a token, and the image head regresses 49 pixel values from each image token’s hidden state, forming a 7×7 patch. We simply add text cross-entropy and image MSE with equal coefficients.

Experiments. WTA alone mainly improved diversity at k=4. Self-distillation substantially improved one-pass validity, though eight-pass WTA still led on validity and diversity. We fixed σ=0.5 relative to the mask embedding across mask ratios. We have not tested a mask-dependent schedule, models beyond 0.5B, text blocks longer than eight tokens, or more complex datasets.

We had to divide our limited time and resources between submissions during this rebuttal. We genuinely regret that we could not give you results for the experiments you suggested. We will carry out the additional experiments you suggested in our follow-up work.

Reviewer_Y6S2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CONDOR, a noise-conditioned masked Transformer trained from scratch to generate a complete token block in one forward pass. A best-of-k winner-take-all loss supervises partially masked examples, while same-noise self-distillation trains fully masked one-pass predictions toward targets obtained by retaining and iteratively refilling the model’s own draft. TinyStories experiments demonstrate a substantial improvement in locally acceptable and diverse eight-token continuations, including a shorter-training comparison. A 0.5B model is evaluated on eight successive blocks, and qualitative MNIST examples illustrate text-conditioned and unconditional text–image generation.

Critical strengths

The single-block improvement is substantial and persists under a shorter training budget. In Table 2, one-pass Validity increases from 9.2% for WTA to 81.6% for default CONDOR, accompanied by a substantial increase in distinct valid continuations. The 30k-update version still reaches 60.4% using 2.9 GPU-hours, below the 5.3 GPU-hours reported for WTA (Appendix A.1). Thus, additional training time alone does not explain the improvement. The noise-coupling ablation provides useful evidence for the design. Using independent target noise retains high local validity but sharply reduces valid-output diversity, supporting the importance of preserving the noise–target correspondence. The method also works with multiple teacher choices, including the current student, without requiring an external pretrained teacher in its main setting (Table 2).

Critical weaknesses

The contribution of WTA relative to the additional self-distillation objective remains insufficiently isolated. Plain CE, IWAE, and WTA are compared without self-distillation, whereas CONDOR receives the same-noise refinement objective. The independent-target-noise ablation supports coupling, but does not establish the advantage of WTA over CE or IWAE when those objectives receive the same refinement procedure. This is a particularly relevant comparison because the paper motivates WTA as the mechanism that enables different noise samples to specialize. Its absence limits the evidence for the specific training design beyond the benefit of same-noise self-distillation (Sections 3.1–3.2; Table 2). The evidence for coherent, diverse blockwise generation is too limited. Table 4 uses only four prompts with four runs each and evaluates individual blocks rather than complete continuations. At block one, 100% validity and 25% uniqueness mean that all four noise draws produce the same block for each prompt. Although the GT-prefix arm provides a fixed-prefix control, it contains only four prompts per block position. The increasing diversity in free continuation additionally reflects diverging prefixes, so broader fixed-prefix evaluation is needed to establish how reliably noise produces distinct valid blocks. Moreover, the longer stories in Table 5 explicitly regenerate unsatisfactory blocks and therefore cannot establish unfiltered rollout quality or one forward pass per accepted block. These disclosures are appreciated, but broader evaluation of the existing checkpoint is needed to substantiate the blockwise-generation claim (Section 4.3; Tables 4–5). These concerns currently lead me to weak rejection, despite the promising single-block results. I would reconsider the recommendation if matched self-distillation baselines support the proposed training design and broader unfiltered evaluation establishes reliable blockwise behavior.

Questions for authors

Can you compare WTA with IWAE—and, if feasible, plain CE—using the same noise-coupled self-distillation procedure under matched data and training budgets? This would help determine whether the main improvement depends on WTA or is largely attributable to the refinement objective. Can the existing 0.5B checkpoint be evaluated on a substantially broader, fixed set of unseen prompts, reporting unfiltered full-continuation quality, fixed-prefix noise diversity, and uncertainty aggregated by prompt? For the selected longer stories, reporting acceptance rates or average attempts per accepted block would also clarify the effective generation cost.

1 comment

Thank you for reading our paper so carefully and for the thought you put into this review. We especially appreciated that you told us which results you found convincing and where you still saw gaps in the evidence. Knowing what led you to that judgment helped us see our own results more clearly. Your suggestions have given us a much better sense of what to prioritize as we continue this work. We are grateful for the time you spent on it. With our limited time and resources divided between submissions, we genuinely regret that we could not follow through on your suggestions during this rebuttal. We will carry out the additional experiments in our follow-up work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.