acceptodds
Withdrawn from ICLR 2027

From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Uniform-state discrete diffusion updates all positions in parallel while keeping every token revisable, but earlier predictions are normally lost when the state changes. We study *persistent context*: *Committed* keeps selected clean-token predictions visible for up to the next model inputs while leaving the sampled state mutable. We ask two questions: why a denoiser should respond to the token shown at its own position, and how such a response affects the later sampling path. For the denoiser, we show that a class of cross-time distillation schemes, including Duo's consistency distillation, need not preserve leave-one-out (LOO) invariance even under an exact LOO teacher. On the sampling side, we show how a change caused by persistent context passes through the reverse updates and affects later states. This leads to an exact LOO example in which persistent context with a finite horizon improves the terminal distribution, while keeping the same context to the end makes it worse. On Duo and Duo-distilled, reference-token reinsertion produces opposite responses at moderate-to-high signal, and changing only the retained label shifts the loss–entropy tradeoff differently across checkpoints. Persistent input context also gives different loss–entropy and loss–repetition tradeoffs from using stored predictions in the reverse updates instead of the model inputs. Persistent context therefore does more than change the current sampling distribution: its effect depends both on how the denoiser uses the retained token and on how long that token remains visible during sampling.

Primary area:
generative models
Keywords:
discrete diffusion, diffusion language models, uniform-state diffusion, parallel decoding, top-p sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fHhfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new sampling mechanism for uniform discrete diffusion where the "context" can be made persistent by giving it to the network, while using the current state in the reverse transition's conditioning. They show that a certain distillation procedure does not preserve the LOO structure of the model and that this procedure has a different expressivity than top-p sampling. They design confidence strategies to choose the selected context and investigate empirically the effect of the proposed method on natural language.

Critical strengths

The proposed persistent context mechanism is simple and training free, contrary to similar ideas like self-conditioning, which makes it very easy to test and implement in a trained model. The DCD result is interesting and can guide the choice of tokens in the context, although not really surprising.

Critical weaknesses

The theoretical results are mostly toy-ish and do not give a lot of insights into the proposed sampler. For instance the top-p one is relatively artificial, since they just show that top-p cannot reproduce some distribution but they do not show the need to target such distribution or the fact that this is specifically an issue. The sampler introduces a bias such that the model is not exact anymore (in the infinite time limit) which is not very well motivated or interpretable. One could argue that temperature or top-p sampling does the same, but in this case, it is much more interpretable.
The experiments are narrow in terms of their scope (only natural language and a single model comparison).

Questions for authors

Do you have a principled explanation for the proposed sampler, apart from the toy-example considered in the appendix ? An analysis that could show the benefit of this method or a general condition (maybe on p_0) that could show the advantage of introducing a bias ? Could you explain the experimental results that you get ? For instance, why does your method sometimes seem to help at matched entropy but not at matched repetition ? Why does it seem to not improve for GIDD ? What would help the paper be more convincing would be to have an underlying predictive interpretation for these results.

Reviewer_axKoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the question that in uniform diffusion language model generation, how does the prior model prediction affect the future decoding as extra context. The paper first analyzes how distillation can introduce dependence on a token’s own input, and then proposes a sampling algorithm to keep the prior-selected tokens during generation. Empirically, the authors measure its effect on denoiser and sampling trajectories.

Critical strengths
  1. This paper brings out an important question in diffusion language model generation. The idea of reusing previously selected predictions for later decoding as context is useful and practical. 

  2. The paper is well written, and the main idea is easy to follow. 

  3. The theory and analysis are careful and provide useful insights into the main claim of the paper.

Critical weaknesses
  1. I do not fully understand the position of Section 3. This section shows that distillation does not necessarily preserve the model’s LOO invariance if the teacher is LOO-trained and the distillation objective does not enforce LOO. I believe an LOO-trained model is not a requirement for the sampling to work; then how is this discovery related to the main sampling algorithm of the paper? Is the point to show that the algorithm still works in a distilled DUO model?

  2. The experiment results do not show a consistent improvement in general quality. The model used is relatively small. The algorithm is only tested on DUO, its distilled model, and GIDD. It shows no clear gain on GIDD. Evaluating the model on a broader class of uniform state diffusion models would support its empirical gain, for example, Sumi 7B, DiffusionGemma, etc. Evaluation mainly uses log-likelihoods; evaluating on a broader set of downstream tasks would strengthen the paper.

  3. The method needs a few hyperparameters that are fixed before generation, such as persistence horizon, selection budget, spacing radius, etc. It is not clear how to choose these configurations for a new model.

Questions for authors

Weaknesses 1.

Reviewer_92GxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce 'COMMITTED', a method to maintain persistent context. They use confidence based selection to temporarily commit to certain tokens for a few denoising steps, in order to stabilize context while maintaining the ability to revise tokens. They also study how distillation affects how a denoiser uses its inputs.

Critical strengths

The observation that distillation can affect the probabilistic interpretation of Duo's reverse sampler is worth noting.

Critical weaknesses
  • The proposed method 'COMMITTED' is very heuristic: tokens are chosen using confidence, and it uses arbitrarily chosen warmup period, duration, and count. While such heuristics can be effective in practice, the gains achieved here are not convincing: Figure 3's COMMITTED improvement over the frontier is not significant and only has one data point each, whereas a full GenPPL/entropy frontier would be more informative (and I suspect that the plotted unigram entropy range doesn't currently include the OWT reference of ~around 5.43). Further, GenPPL and entropy metrics have their limitations, while distributional metrics like MAUVE are widely regarded as a more robust indicator.
  • As a human reader, I think the writing lacks clarity and narrative structure. Experimental details are also not properly introduced in the main text, and empirical breadth is very limited.
Questions for authors

I do not have specific questions that could materially affect my assessment. I think this paper falls substantially short of ICLR standards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.