- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Loopholing All the Way (LAW), a continuous generative model based entirely on self-conditioning. Unlike existing methods, LAW propagates token embeddings across iterations and decodes discrete tokens only at the final step. The effectiveness of the proposed method is validated on Sudoku, Maze, GSM8K, and OpenWebText, and ablation studies show the effectiveness of key design choices. LAW shows strong few-step performance on several reasoning tasks, particularly when fine-tuning a pretrained Duo model on TinyGSM.
Critical strengths
The principal contribution of this paper, in my view, is that it shows continuous self-conditioning alone can support competitive generation without intermediate discrete sampling. This provides a novel perspective on the role of discrete sampling in diffusion language models (DLMs), which is also shown to be compatible with pre-trained checkpoints. The effectiveness is also shown for few-step generation experiments on multiple datasets.
Critical weaknesses
The major concern is that the explanation of the key performance gains remains limited. The chosen design is validated by experimental results and ablations, but it remains unclear whether the performance gains come from removing sampling noise or changing how inputs are combined. An explanation of why the advantage is strong after fine-tuning but not when training from scratch is also expected. The lack of such intuitive discussion limits the understanding of why pure self-conditioning is beneficial.
From my perspective, it is intuitive that LDDM might learn to rely mainly on its continuous state when discrete tokens are unhelpful. What prevents it from recovering LAW-like behavior in some cases?
Questions for authors
Please refer to the 'Critical Weaknesses' section and address the concern.
2 comments
Dear reviewer,
We wanted to follow up and ask if our response has addressed your concerns. Otherwise, we would be happy to address any remaining questions and reach a consensus before the discussion period ends.
Thank you for your time!
We thank the reviewer for their positive review and for emphasizing the novelty and effectiveness of our method across different experimental settings.
General Remark: Motivated by the suggestion of the reviewer XATP that "fewer training-sequence presentations" could contribute to the gap between LDDM [1] and LAW, we found that a small adjustment to the training procedure significantly improves LAW's performance, without changing the model architecture or the method itself. In particular, we explore a training procedure closer to LDDM. With probability p, we unroll M=2 passes and apply the loss to the last pass only, rather than averaging over all passes. Otherwise, we perform a single pass, as in uniform diffusion. Since we stop gradients through the earlier passes, this allows us to train LAW for the same number of steps as LDDM-U at matched FLOPs. Additionally, with this modification in place, we observed that using cross-entropy (CE) gave more diverse samples on OpenWebText.
These minor adjustments lead to substantial improvements. When trained from scratch on TinyGSM, LAW now solves 42.2% of GSM8K problems (T=1, 1024 NFE), compared to 34.5% for LDDM-U. On OpenWebText (64 NFE), LAW now achieves lower Gen. PPL than the diffusion and flow baselines at matched entropy (Fig. 24, Suppl. D.4).
We provide full details on this variant in Suppl. D of the updated manuscript.
We address the reviewer's concerns below.
Yes. With the Expected Embedding state, the GSM8K accuracy peaks at 38.5% when the state is added to the discrete tokens, as in LDDMs, vs. 53.4% when it replaces them, as in LAW (256 NFE, T=1; Fig. 7).
Yes. To isolate the sampling noise, we introduce two new baselines that replace the Expected Embedding in LAW by either:
As in the submission, we fine-tune the Duo checkpoint on TinyGSM for 50k steps with batch size 64. We find that
Suppl. E.3 shows the GSM8K accuracy of all variants with 256 NFE at T=1, without regularization (Fig. 26a) and with temperature regularization (ρ=4; Fig. 26b).
It is not entirely clear why. The gap is not explained by the difference in training duration: we evaluated the LAW checkpoints trained from scratch every 10k steps, and the accuracy plateaus after ~70k steps (out of 192k; Fig. 23). The Duo initialization might help LAW converge to a better solution, possibly thanks to the representations learned during pre-training. Understanding why is an interesting question for future work.
We found that LDDMs rely mainly on the discrete tokens. When we set the token embeddings of LDDM-U and LDDM-M to zero during sampling, they solve almost none of the GSM8k problems. With a zeroed SC state, the accuracy drops by half or more but remains well above 0% (Fig. 25). We see three reasons, now in Suppl. A.3:
[1] Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall, Jo et al., ICLR 2026.
[2] Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data?, Grinsztajn et al., NeurIPS 2022.
[3] The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions, D'Amario et al., Frontiers in Computational Neuroscience 2022.
[4] Large Language Models Can Be Easily Distracted by Irrelevant Context, Shi et al., ICML 2023.
[5] Feature Selection, L1 vs. L2 Regularization, and Rotational Invariance, Ng, ICML 2004.