- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
During training, discrete diffusion models randomly mask tokens in a sequence and learn to reconstruct the full sequence, whereas at generation time usually a prefix (user prompt or question) is given and the model generates the answer. To address this train-test mismatch, the paper proposes Prefix-Conditioned Diffusion (PCD), which keeps the prefix clean during training and adds an autoregressive causal attention and the loss on the prefix.
Critical strengths
- The idea is simple and practical. It does not require labeled prompt-response pairs, and the inference procedure is unchanged.
- The performance gains on benchmarks are consistent.
- The method works on large-scale language models.
Critical weaknesses
-
Insufficient justification for the prefix AR loss and causal attention. In the method, the paper introduces a prefix AR loss and causal prefix attention in addition to keeping the prefix clean, but the justification for these components is insufficient. The reason for the AR loss is simply limited to obtaining a learning signal from the prefix. In the experiments, Figure 6 does not include a setting with the AR loss weight set to 0. The authors acknowledge the absence of this experiment (line 368), but discussion and experiments supporting the use of causal attention and the AR loss on the prefix are still lacking. To support this, results without the AR loss and without causal attention are needed.
-
Limited novelty and missing comparison with block diffusion, which already uses a clean prefix. Block diffusion already conditions on clean preceding blocks, and the authors acknowledge that PCD is a restricted case of Adaptive Block Diffusion. The main contribution is therefore introducing prefix conditioning into the stable stage of the LLaDA2.0 pipeline, which can be viewed as a scheduling choice, so the novelty appears limited. Moreover, the simplest alternative for introducing prefix conditioning in the stable stage is to keep the stable stage as block diffusion through scheduling, but no comparison with such a model is provided. In addition, the gap shrinks during the final decay stage (block difusion) in Figure 3, which suggests that benefits similar to PCD may be obtainable through block diffusion.
-
Constrained and the method favored benchmarks. According to the benchmark statistics (line 412), the median prefix length is 754 tokens and the median target length is 12 tokens, meaning about 98% of the text is prefix. This setting favors a method trained to condition on a clean prefix. Assuming training with , PCD learns with 37.5% of the text given as prefix on average, so the benchmark composition structurally favors PCD. As a result, the benchmark results are biased toward the method. In real usage, many cases have a short prefix and a very long answer, and the current benchmarks seem far from reflecting such cases.
-
Unclear significance of the matched training-prefix intervention in Section 4.4. This experiment compares two models trained without the AR loss that differ only in whether the prefix is corrupted, but the setup is highly unfavorable to the prefix-corrupted model. The causal attention structure, designed under the assumption of a clean prefix, is still maintained in the prefix-corrupted model, and the evaluation is performed with a clean prefix. The corrupted model therefore inevitably underperforms due to train-test mismatch and the architectural mismatch.
Questions for authors
-
Are there results comparing against PCD with the AR loss weight set to 0? Also, without the AR loss there is no reason to keep causal prefix attention. What are the results with bidirectional prefix attention?
-
In Table 3, the AR-dominant/native mix, which is close to AR training, shows a gain similar to Mixed PCD (38.98 vs. 38.94). How can one distinguish whether the gain comes from the clean prefix or from mixing in AR training loss?
-
Does the advantage of PCD remain when compared with a baseline in which the stable stage of LLaDA2.0 is kept as block diffusion through scheduling?
-
Native closes much of the gap during decay. Does the gap persist if a longer decay is applied to native as well?
-
Are there results in settings with short prompts and long answers to generate? The benchmark setting has prefixes that consists about 98% of the text, which strongly favors PCD. Can the method be evaluated on tasks with very short prefixes and very long targets?
-
In the matched training-prefix intervention in Section 4.4, why was the control experiment constructed with causal prefix attention and no loss on the prefix, rather than as native MDM? What are the results when the corrupted-prefix model and the clean-prefix model is evaluated with a corrupted prefix or with a MDM loss?
5 comments
Response to Reviewer kBsH (2/3) — Q3–Q4
Q3. How does PCD compare with continuing block diffusion throughout the stable stage?
Block diffusion already conditions on clean preceding blocks. The design question we address is how to preserve useful context during LLaDA2.0’s high-throughput stable stage. This stage switches to full-sequence diffusion to process large amounts of pretraining text, but independent token corruption rarely leaves a long prefix intact. PCD samples a boundary in ordinary pretraining text, preserves the preceding tokens, concentrates corruption in the suffix, and applies next-token supervision to the preserved prefix. It requires no annotated prompt–response pairs.
Our main experiment used 400B stable tokens and 50B decay tokens per branch. To test continued BD directly at a matched budget, we used Qwen3-1.7B and branched from a common checkpoint after 1B tokens of BD warmup. One branch switched to Full PCD, and the other continued BD at block size 32. Both saw the same additional 1B tokens in the same order, with matched optimizer settings and the same block-32 decoder.
| End of the 1B stable stage | GSM8K | MATH500 | HumanEval |
|---|---|---|---|
| Full PCD | 35.10% | 12.80% | 15.85% |
| Continued BD | 32.75% | 5.60% | 15.24% |
Full PCD solved 31 more GSM8K problems and 36 more MATH500 problems than continued BD. The two methods differed by one problem on HumanEval. On the same eight H800 GPUs, Full PCD processed approximately 150,000 original training tokens per second, compared with 83,400 for continued BD. At this matched token budget, PCD achieved higher accuracy on both math benchmarks with 1.8× the training throughput.
This throughput difference matters for the scale at which the stable stage is intended to operate. Our main experiment used 400B stable tokens per branch. For a model with 100B parameters, sustaining enough throughput to process a corpus of that size within the available compute budget is a practical requirement. This is why we focus on adapting the high-throughput stable stage. PCD lets the model learn to use an intact prefix at this stage while retaining the efficiency needed to train on large volumes of pretraining text.
Q4. Does Native catch up with Full PCD as block-diffusion decay continues?
Native closes part of the gap in our main experiment. After the 400B-token stable stage, Full PCD leads Native by 2.48 points on the 12-task average. Giving both models another 50B tokens of identical block-diffusion training narrows that lead to 1.64 points, with Full PCD still ahead on 10 of the 12 tasks.
To examine how the gap changes during decay, we ran a smaller matched comparison on Qwen3-1.7B. We started from each method’s 1B-token stable checkpoint (the 0M row) and continued both on the same data with block size 32. The learning rate reached at 50M tokens and stayed there through 100M. Each cell reports Full PCD / Native in percent.
| Decay tokens | GSM8K | MATH500 | HumanEval |
|---|---|---|---|
| 0M | 35.10 / 33.74 | 12.80 / 9.40 | 15.85 / 17.07 |
| 25M | 31.84 / 28.89 | 13.80 / 11.00 | 14.63 / 15.24 |
| 50M | 34.57 / 33.13 | 11.20 / 10.00 | 15.85 / 11.59 |
| 100M | 36.62 / 33.74 | 11.80 / 10.60 | 18.90 / 14.63 |
The clearest catch-up is on MATH500, where Native reduces the gap from 3.40 points at 0M to 1.20 at 50M. From 50M to 100M, both models solve three more problems, leaving the gap at 1.20 points. On GSM8K, Full PCD’s lead fluctuates and ends at 2.88 points, up from 1.36 at 0M. On HumanEval, Native starts two problems ahead, while Full PCD leads by seven problems at both 50M and 100M. From 50M to 100M, Native improved on all three benchmarks, but its gap to Full PCD did not narrow on any of them.
Response to Reviewer kBsH (3/3) — Q5–Q6
Q5. Do the gains extend to shorter prompts and longer answers?
Table 13 summarizes five benchmarks. Its 754-token and 12-token figures are the separate medians of prompt length and reference-answer length. We agree that these benchmarks emphasize long conditioning contexts. To test shorter prompts, we reduced the number of worked examples from five to two on GSM8K and from four to two on MATH500. We selected questions with reference solutions of at least 128 tokens and evaluated the two 1B-stable checkpoints on identical questions, prompts, and decoding settings.
| Two-shot evaluation | Median prompt length, including examples | Full PCD | Native |
|---|---|---|---|
| GSM8K | 474 tokens | 15.03% (81/539) | 11.69% (63/539) |
| MATH500 | 313.5 tokens | 6.42% (19/296) | 5.07% (15/296) |
With two worked examples, the prompts still contained several hundred tokens. Under this setting, Full PCD solved 18 more GSM8K problems and four more MATH500 problems than Native.
Q6. Does the fixed-attention control favor clean-prefix training? What happens when the evaluation prefix is masked?
Section 4.4 isolates whether the prefix is corrupted during training. Both branches use the same causal prefix attention, suffix targets, data order, and optimization, with prefix AR supervision disabled. Substituting the full native MDM objective would also change which positions receive supervision and how they attend to one another.
To test your concern about the evaluation condition, we used the same checkpoints from Section 4.4. For each of three training seeds, we evaluated both branches on the same 976 validation blocks twice, first with a visible prefix and then with every prefix token masked. We kept the prefix boundaries, attention, suffix inputs, suffix masks, and scored targets fixed. The table reports mean masked-suffix reconstruction loss in nats. Lower is better.
| Evaluation prefix | Trained with a clean prefix | Trained with a corrupted prefix |
|---|---|---|
| Fully visible | 3.9505 | 4.0197 |
| Fully masked | 4.8170 | 4.4321 |
The ordering reversed for all three training seeds. With a visible prefix, clean-prefix training lowered loss by 0.0692 nats. With the prefix fully masked, corrupted-prefix training lowered loss by 0.3849 nats. This reversal shows where each approach helps. Preserving the training prefix improves suffix prediction when context is available, while corrupting it improves robustness when that context disappears. In prompt continuation, the model has access to the full prefix, which is the setting PCD was designed for.
Response to Reviewer kBsH (1/3) — Q1–Q2
Thank you for raising two questions that matter for interpreting our results: what the clean prefix and its AR supervision each contribute during the stable stage, and how those gains change with the amount of visible context or subsequent training. We address these questions with matched training runs and by evaluating the same checkpoints under different decoding and prefix conditions.
Q1. What happens when the prefix AR weight is zero? Can the prefix then use bidirectional attention?
We trained three Qwen3-1.7B variants from the same warmup checkpoint on the same 1B stable tokens. All three preserved a clean prefix for suffix denoising. Starting from Full PCD, we first set the prefix AR weight to zero while keeping causal attention. We then changed only the prefix attention to bidirectional, keeping the AR weight at zero. We evaluated each checkpoint with both block-diffusion decoding and standard left-to-right AR decoding. GSM8K and MATH500 report accuracy, and HumanEval reports pass@1.
With causal prefix attention held fixed, adding AR supervision modestly improved diffusion decoding on all three tasks. The gains were larger under standard left-to-right AR decoding: 11.0 percentage points on GSM8K, 11.6 on MATH500, and 13.4 on HumanEval. These results support using prefix AR supervision to help retain the model’s autoregressive generation ability during conversion.
A bidirectional prefix also supports suffix denoising when the AR weight is zero. Compared with causal no-AR, it scored higher on GSM8K and HumanEval and lower on MATH500. Full PCD uses a causal prefix so that each prefix position can predict the next token without seeing its target, while the suffix can still attend to the entire clean prefix.
Q2. Does the similar performance of the AR-dominant/native mixture and Mixed PCD in Table 3 mean that the gains come from AR supervision?
Both configurations use the same native diffusion branch for half of the examples. They differ on the other half. Mixed PCD samples the prefix boundary and denoises suffixes of varying lengths. The AR-dominant configuration places the boundary at its maximum, leaving just one suffix token, so nearly all supervision on that branch comes from prefix next-token prediction.
The two configurations do have nearly identical six-task averages: 38.94 for Mixed PCD and 38.98 for the AR-dominant/native mixture, compared with 34.28 for Native. Their task profiles differ. Mixed PCD scores higher on GSM8K (66.94% vs. 62.93%), while the AR-dominant/native mixture scores higher on HumanEval (29.88% vs. 28.05%).
Two controlled comparisons help interpret these averages. Q1 holds the clean prefix and its attention fixed and shows that AR supervision helps downstream generation, especially under AR decoding. Section 4.4 asks the complementary question: with AR supervision disabled in both branches, does preserving the training prefix still help? With the same attention and suffix targets, preserving the prefix reduced suffix reconstruction loss by 0.0692 nats when the evaluation prefix was visible. Thus, AR supervision contributes to generation quality, and the clean prefix itself provides a measurable conditioning benefit even at zero AR weight.