acceptodds
Under review as a conference paper at ICLR 2027

Prefix-Conditioned Diffusion: Reducing the Pretraining–Generation Mismatch in Diffusion Language Models

Abstract

Diffusion language models (dLLMs) generate text through iterative denoising, allowing multiple tokens to be predicted in parallel. However, pretraining may mask tokens throughout a sequence, whereas prompt continuation conditions on an intact prefix. This difference remains in conversion pipelines that denoise entire sequences during the stable stage. We propose Prefix-Conditioned Diffusion (PCD), which samples a boundary, preserves the prefix, and denoises the suffix. The training recipe also applies autoregressive supervision to the prefix. We evaluate PCD in the stable stage of a warmup, stable, and decay conversion pipeline, with inference unchanged. Matched experiments across model families show improvements in reasoning and coding over native diffusion training. A matched continuation study further shows that the advantage persists after a shared decay stage. In a separate reconstruction diagnostic, the full PCD recipe's advantage over native diffusion training reverses as more evaluation prefix tokens are masked. Controlled experiments show lower suffix reconstruction loss with a clean training prefix, an intact evaluation prefix, and no autoregressive loss.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, prefix conditioning, autoregressive-to-diffusion conversion, continued pretraining, block diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kBsHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

During training, discrete diffusion models randomly mask tokens in a sequence and learn to reconstruct the full sequence, whereas at generation time usually a prefix (user prompt or question) is given and the model generates the answer. To address this train-test mismatch, the paper proposes Prefix-Conditioned Diffusion (PCD), which keeps the prefix clean during training and adds an autoregressive causal attention and the loss on the prefix.

Critical strengths
  • The idea is simple and practical. It does not require labeled prompt-response pairs, and the inference procedure is unchanged.
  • The performance gains on benchmarks are consistent.
  • The method works on large-scale language models.
Critical weaknesses
  • Insufficient justification for the prefix AR loss and causal attention. In the method, the paper introduces a prefix AR loss and causal prefix attention in addition to keeping the prefix clean, but the justification for these components is insufficient. The reason for the AR loss is simply limited to obtaining a learning signal from the prefix. In the experiments, Figure 6 does not include a setting with the AR loss weight set to 0. The authors acknowledge the absence of this experiment (line 368), but discussion and experiments supporting the use of causal attention and the AR loss on the prefix are still lacking. To support this, results without the AR loss and without causal attention are needed.

  • Limited novelty and missing comparison with block diffusion, which already uses a clean prefix. Block diffusion already conditions on clean preceding blocks, and the authors acknowledge that PCD is a restricted case of Adaptive Block Diffusion. The main contribution is therefore introducing prefix conditioning into the stable stage of the LLaDA2.0 pipeline, which can be viewed as a scheduling choice, so the novelty appears limited. Moreover, the simplest alternative for introducing prefix conditioning in the stable stage is to keep the stable stage as block diffusion through scheduling, but no comparison with such a model is provided. In addition, the gap shrinks during the final decay stage (block difusion) in Figure 3, which suggests that benefits similar to PCD may be obtainable through block diffusion.

  • Constrained and the method favored benchmarks. According to the benchmark statistics (line 412), the median prefix length is 754 tokens and the median target length is 12 tokens, meaning about 98% of the text is prefix. This setting favors a method trained to condition on a clean prefix. Assuming training with p∈[0,0.75]p \in [0, 0.75], PCD learns with 37.5% of the text given as prefix on average, so the benchmark composition structurally favors PCD. As a result, the benchmark results are biased toward the method. In real usage, many cases have a short prefix and a very long answer, and the current benchmarks seem far from reflecting such cases.

  • Unclear significance of the matched training-prefix intervention in Section 4.4. This experiment compares two models trained without the AR loss that differ only in whether the prefix is corrupted, but the setup is highly unfavorable to the prefix-corrupted model. The causal attention structure, designed under the assumption of a clean prefix, is still maintained in the prefix-corrupted model, and the evaluation is performed with a clean prefix. The corrupted model therefore inevitably underperforms due to train-test mismatch and the architectural mismatch.

Questions for authors
  • Are there results comparing against PCD with the AR loss weight set to 0? Also, without the AR loss there is no reason to keep causal prefix attention. What are the results with bidirectional prefix attention?

  • In Table 3, the AR-dominant/native mix, which is close to AR training, shows a gain similar to Mixed PCD (38.98 vs. 38.94). How can one distinguish whether the gain comes from the clean prefix or from mixing in AR training loss?

  • Does the advantage of PCD remain when compared with a baseline in which the stable stage of LLaDA2.0 is kept as block diffusion through scheduling?

  • Native closes much of the gap during decay. Does the gap persist if a longer decay is applied to native as well?

  • Are there results in settings with short prompts and long answers to generate? The benchmark setting has prefixes that consists about 98% of the text, which strongly favors PCD. Can the method be evaluated on tasks with very short prefixes and very long targets?

  • In the matched training-prefix intervention in Section 4.4, why was the control experiment constructed with causal prefix attention and no loss on the prefix, rather than as native MDM? What are the results when the corrupted-prefix model and the clean-prefix model is evaluated with a corrupted prefix or with a MDM loss?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kBsH (1/3) — Q1–Q2

Thank you for raising two questions that matter for interpreting our results: what the clean prefix and its AR supervision each contribute during the stable stage, and how those gains change with the amount of visible context or subsequent training. We address these questions with matched training runs and by evaluating the same checkpoints under different decoding and prefix conditions.

Q1. What happens when the prefix AR weight is zero? Can the prefix then use bidirectional attention?

We trained three Qwen3-1.7B variants from the same warmup checkpoint on the same 1B stable tokens. All three preserved a clean prefix for suffix denoising. Starting from Full PCD, we first set the prefix AR weight to zero while keeping causal attention. We then changed only the prefix attention to bidirectional, keeping the AR weight at zero. We evaluated each checkpoint with both block-diffusion decoding and standard left-to-right AR decoding. GSM8K and MATH500 report accuracy, and HumanEval reports pass@1.

DecodingStable-stage configurationGSM8KMATH500HumanEval
DiffusionFull PCD, causal prefix35.1%12.8%15.9%
DiffusionCausal prefix, AR weight zero34.5%11.4%13.4%
DiffusionBidirectional prefix, AR weight zero35.4%8.0%15.9%
ARFull PCD, causal prefix55.7%29.2%39.6%
ARCausal prefix, AR weight zero44.7%17.6%26.2%
ARBidirectional prefix, AR weight zero45.5%17.6%25.6%

With causal prefix attention held fixed, adding AR supervision modestly improved diffusion decoding on all three tasks. The gains were larger under standard left-to-right AR decoding: 11.0 percentage points on GSM8K, 11.6 on MATH500, and 13.4 on HumanEval. These results support using prefix AR supervision to help retain the model’s autoregressive generation ability during conversion.

A bidirectional prefix also supports suffix denoising when the AR weight is zero. Compared with causal no-AR, it scored higher on GSM8K and HumanEval and lower on MATH500. Full PCD uses a causal prefix so that each prefix position can predict the next token without seeing its target, while the suffix can still attend to the entire clean prefix.

Q2. Does the similar performance of the AR-dominant/native mixture and Mixed PCD in Table 3 mean that the gains come from AR supervision?

Both configurations use the same native diffusion branch for half of the examples. They differ on the other half. Mixed PCD samples the prefix boundary and denoises suffixes of varying lengths. The AR-dominant configuration places the boundary at its maximum, leaving just one suffix token, so nearly all supervision on that branch comes from prefix next-token prediction.

The two configurations do have nearly identical six-task averages: 38.94 for Mixed PCD and 38.98 for the AR-dominant/native mixture, compared with 34.28 for Native. Their task profiles differ. Mixed PCD scores higher on GSM8K (66.94% vs. 62.93%), while the AR-dominant/native mixture scores higher on HumanEval (29.88% vs. 28.05%).

Two controlled comparisons help interpret these averages. Q1 holds the clean prefix and its attention fixed and shows that AR supervision helps downstream generation, especially under AR decoding. Section 4.4 asks the complementary question: with AR supervision disabled in both branches, does preserving the training prefix still help? With the same attention and suffix targets, preserving the prefix reduced suffix reconstruction loss by 0.0692 nats when the evaluation prefix was visible. Thus, AR supervision contributes to generation quality, and the clean prefix itself provides a measurable conditioning benefit even at zero AR weight.

AuthorsOpenReview ↗

Response to Reviewer kBsH (2/3) — Q3–Q4

Q3. How does PCD compare with continuing block diffusion throughout the stable stage?

Block diffusion already conditions on clean preceding blocks. The design question we address is how to preserve useful context during LLaDA2.0’s high-throughput stable stage. This stage switches to full-sequence diffusion to process large amounts of pretraining text, but independent token corruption rarely leaves a long prefix intact. PCD samples a boundary in ordinary pretraining text, preserves the preceding tokens, concentrates corruption in the suffix, and applies next-token supervision to the preserved prefix. It requires no annotated prompt–response pairs.

Our main experiment used 400B stable tokens and 50B decay tokens per branch. To test continued BD directly at a matched budget, we used Qwen3-1.7B and branched from a common checkpoint after 1B tokens of BD warmup. One branch switched to Full PCD, and the other continued BD at block size 32. Both saw the same additional 1B tokens in the same order, with matched optimizer settings and the same block-32 decoder.

End of the 1B stable stageGSM8KMATH500HumanEval
Full PCD35.10%12.80%15.85%
Continued BD32.75%5.60%15.24%

Full PCD solved 31 more GSM8K problems and 36 more MATH500 problems than continued BD. The two methods differed by one problem on HumanEval. On the same eight H800 GPUs, Full PCD processed approximately 150,000 original training tokens per second, compared with 83,400 for continued BD. At this matched token budget, PCD achieved higher accuracy on both math benchmarks with 1.8× the training throughput.

This throughput difference matters for the scale at which the stable stage is intended to operate. Our main experiment used 400B stable tokens per branch. For a model with 100B parameters, sustaining enough throughput to process a corpus of that size within the available compute budget is a practical requirement. This is why we focus on adapting the high-throughput stable stage. PCD lets the model learn to use an intact prefix at this stage while retaining the efficiency needed to train on large volumes of pretraining text.

Q4. Does Native catch up with Full PCD as block-diffusion decay continues?

Native closes part of the gap in our main experiment. After the 400B-token stable stage, Full PCD leads Native by 2.48 points on the 12-task average. Giving both models another 50B tokens of identical block-diffusion training narrows that lead to 1.64 points, with Full PCD still ahead on 10 of the 12 tasks.

To examine how the gap changes during decay, we ran a smaller matched comparison on Qwen3-1.7B. We started from each method’s 1B-token stable checkpoint (the 0M row) and continued both on the same data with block size 32. The learning rate reached 2×10−62\times10^{-6} at 50M tokens and stayed there through 100M. Each cell reports Full PCD / Native in percent.

Decay tokensGSM8KMATH500HumanEval
0M35.10 / 33.7412.80 / 9.4015.85 / 17.07
25M31.84 / 28.8913.80 / 11.0014.63 / 15.24
50M34.57 / 33.1311.20 / 10.0015.85 / 11.59
100M36.62 / 33.7411.80 / 10.6018.90 / 14.63

The clearest catch-up is on MATH500, where Native reduces the gap from 3.40 points at 0M to 1.20 at 50M. From 50M to 100M, both models solve three more problems, leaving the gap at 1.20 points. On GSM8K, Full PCD’s lead fluctuates and ends at 2.88 points, up from 1.36 at 0M. On HumanEval, Native starts two problems ahead, while Full PCD leads by seven problems at both 50M and 100M. From 50M to 100M, Native improved on all three benchmarks, but its gap to Full PCD did not narrow on any of them.

AuthorsOpenReview ↗

Response to Reviewer kBsH (3/3) — Q5–Q6

Q5. Do the gains extend to shorter prompts and longer answers?

Table 13 summarizes five benchmarks. Its 754-token and 12-token figures are the separate medians of prompt length and reference-answer length. We agree that these benchmarks emphasize long conditioning contexts. To test shorter prompts, we reduced the number of worked examples from five to two on GSM8K and from four to two on MATH500. We selected questions with reference solutions of at least 128 tokens and evaluated the two 1B-stable checkpoints on identical questions, prompts, and decoding settings.

Two-shot evaluationMedian prompt length, including examplesFull PCDNative
GSM8K474 tokens15.03% (81/539)11.69% (63/539)
MATH500313.5 tokens6.42% (19/296)5.07% (15/296)

With two worked examples, the prompts still contained several hundred tokens. Under this setting, Full PCD solved 18 more GSM8K problems and four more MATH500 problems than Native.

Q6. Does the fixed-attention control favor clean-prefix training? What happens when the evaluation prefix is masked?

Section 4.4 isolates whether the prefix is corrupted during training. Both branches use the same causal prefix attention, suffix targets, data order, and optimization, with prefix AR supervision disabled. Substituting the full native MDM objective would also change which positions receive supervision and how they attend to one another.

To test your concern about the evaluation condition, we used the same checkpoints from Section 4.4. For each of three training seeds, we evaluated both branches on the same 976 validation blocks twice, first with a visible prefix and then with every prefix token masked. We kept the prefix boundaries, attention, suffix inputs, suffix masks, and scored targets fixed. The table reports mean masked-suffix reconstruction loss in nats. Lower is better.

Evaluation prefixTrained with a clean prefixTrained with a corrupted prefix
Fully visible3.95054.0197
Fully masked4.81704.4321

The ordering reversed for all three training seeds. With a visible prefix, clean-prefix training lowered loss by 0.0692 nats. With the prefix fully masked, corrupted-prefix training lowered loss by 0.3849 nats. This reversal shows where each approach helps. Preserving the training prefix improves suffix prediction when context is available, while corrupting it improves robustness when that context disappears. In prompt continuation, the model has access to the full prefix, which is the setting PCD was designed for.

Reviewer_kumQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Prefix-Conditioned Diffusion (PCD), a training recipe that randomly partitions a sequence into a clean prefix and a corrupted suffix, combining autoregressive supervision on the prefix with masked-token reconstruction on the suffix. It aims to reduce the mismatch between full-sequence corruption during training and clean-prompt conditioning during generation. Applied during the stable stage of AR-to-diffusion conversion, PCD improves benchmark performance over matched native diffusion training across two model families. On LLaDA2-Mini, the 12-task average improves by 2.48 points after 400B stable-stage tokens, with a 1.64-point advantage remaining after a shared 50B-token decay stage. Separate small-scale controls isolate a benefit of clean training prefixes for suffix reconstruction, while evaluation-time prefix corruption reverses the full recipe's reconstruction advantage. These results support the recipe's empirical utility and sensitivity to context visibility, but do not establish how much of the downstream improvement comes from prefix conditioning versus auxiliary AR supervision or capability retention. Decoding efficiency and speculative drafting are not evaluated.

Critical strengths
  1. Controlled empirical gains. The paper evaluates PCD against matched native diffusion training across two model families, controlling initialization, data, token budget, and inference settings. On LLaDA2-Mini, PCD improves all 12 evaluated tasks at the end of the 400B-token stable stage, with a 2.48-point average gain.
  2. Persistence through shared continuation. The matched 50B-token decay experiment tests whether the advantage survives subsequent common training. PCD retains a 1.64-point average gain across 12 tasks, supporting the practical relevance of the stable-stage intervention beyond an intermediate checkpoint.
  3. A targeted prefix-visibility control. The separate small-scale intervention holds attention and suffix supervision fixed and removes auxiliary AR supervision, isolating a reconstruction benefit from clean training prefixes across three training seeds.
  4. Transparent reporting of scope and limitations. The paper reports diagnostic results that complicate its own interpretation, including the reversal under prefix corruption and the stronger reconstruction performance without AR supervision. It explicitly distinguishes these reconstruction findings from downstream generation gains and acknowledges unresolved attribution.
Critical weaknesses
  1. The downstream gains are not sufficiently attributed to the proposed mismatch correction. The full PCD recipe changes attention, corruption support, and supervision simultaneously. The separate prefix-visibility intervention isolates a reconstruction benefit in a small-scale block-causal setting, but does not establish how much this effect explains the full-scale task gains. The matched task-level ablations also leave alternative explanations open: in Table 3, AR-dominant/native mixing performs similarly to PCD, while the AR-weight sweep lacks a matched zero-AR baseline. Consequently, improved conditioning alignment, auxiliary AR supervision, and retention of pretrained AR capabilities remain insufficiently distinguished.

  2. Missing experimental details substantially limit reproducibility and interpretation. Appendix A.1 states that several main-run settings could not be recovered, including the prefix distribution, PCD/native mixing probability, training sequence length, batch size, optimizer, and learning-rate schedule. Complete decoding and task-specific scoring configurations are also unavailable. Some of these settings directly define the proposed intervention, making it difficult to reproduce the reported gains or determine their sensitivity to implementation choices. In addition, the main LLaDA2-Mini results describe a single paired training trajectory; the small-scale multi-seed controls do not establish uncertainty in the full-scale task improvements.

  3. The experiments do not establish whether extended PCD training is necessary relative to inexpensive adaptation. The shared decay experiment is useful and shows that the advantage persists under one subsequent training procedure, although the gap narrows. However, it does not test whether a mature native diffusion checkpoint can recover the advantage through a short, targeted clean-prefix adaptation stage. The Chat SFT comparison does not resolve this question either, since its exact starting checkpoints and their relation to decay are not recorded. A budget-dependent adaptation comparison would help determine whether PCD provides an advantage requiring extended training or primarily addresses a readily correctable conditioning mismatch.

  4. The contribution is insufficiently positioned relative to established prefix-conditioned objectives. Randomly splitting unlabeled text into an intact prefix and a prediction target has a direct precedent in the PrefixLM objective explored in T5, while XLNet provides relevant historical context for pretraining-input mismatch. Neither is discussed. These methods are not equivalent to PCD, but their relationship matters for distinguishing the established conditioning principle from the paper's diffusion-specific contribution. The discussion of Nemotron-Labs-Diffusion acknowledges prior joint AR/diffusion training, but the experiments do not establish the benefit of PCD's particular formulation over a closely related alternative under matched conditions.

  5. The evaluation leaves the practical decoding tradeoffs unresolved. The paper reports task improvements under matched inference settings, but provides no quality-versus-denoising-step or latency/throughput comparison. Thus, retaining the same inference procedure does not establish whether PCD preserves or improves the efficiency of parallel diffusion generation. Its utility for speculative drafting or self-speculation is also untested. These omissions do not invalidate the reported accuracy gains, but limit conclusions about the recipe's broader practical value for dLLMs.

Questions for authors
  1. What is the contribution of clean-prefix conditioning independently of auxiliary AR supervision? Could you add a matched task-level comparison between full PCD, PCD without prefix AR loss, and native diffusion with auxiliary AR supervision? This would help distinguish conditioning benefits from AR capability retention and connect the reconstruction controls to the main downstream results.

  2. Can a native diffusion checkpoint recover the gains through short clean-prefix adaptation? Starting from the matched stable checkpoints, could you report performance over several small adaptation budgets, with continued native training as a control? The shared decay experiment is informative, but a targeted adaptation comparison would clarify whether extended PCD training is necessary or whether a short final stage offers a comparable benefit.

  3. Can the main experimental configuration be recovered or independently reproduced? Appendix A.1 lists missing settings that include the prefix distribution, mixing probability, optimization configuration, and decoding parameters. Could you provide these details, or reproduce a representative matched comparison using a fully specified configuration? An additional training seed would also help assess the reliability of the reported advantage.

  4. What is the specific contribution beyond established prefix-conditioned and joint AR/diffusion training? Could you position PCD explicitly against T5's PrefixLM objective, the relevant mismatch motivation in XLNet, and Nemotron-Labs-Diffusion? In particular, which benefit do you expect from the single-sequence formulation, and what controlled comparison would distinguish that benefit from the shared conditioning principle?

  5. How does PCD affect the quality-efficiency tradeoff of diffusion decoding? Could you compare native and PCD checkpoints across denoising budgets, reporting both task quality and measured latency or throughput under matched conditions? This would establish whether the accuracy gains persist in the few-step regime. If speculative decoding is within scope, a drafting or self-speculation evaluation would further clarify practical utility; otherwise, explicitly delimiting that application would be helpful.

6 comments
AuthorsOpenReview ↗

Response to Reviewer kumQ (1/3) — Q1–Q2

Thank you for asking what the clean prefix contributes, how early PCD needs to be introduced, and what it costs at training and inference time. We added controlled experiments on Qwen3-1.7B to complement the 400B-token stable-stage comparison on LLaDA2-Mini.

Q1 / W1. What do the clean prefix and its AR supervision each contribute? How does PCD compare with Native + AR?

Table 4 isolates the effect of keeping the prefix intact. Both conditions disable the AR loss and use the same attention mask and suffix denoising targets. Only prefix corruption during training changes. When the full prefix is available at evaluation, training with an intact prefix lowers suffix reconstruction loss in all three seeds, from 4.0193 to 3.9501 nats on average. An intact training prefix therefore helps the model use the available context for denoising even without AR supervision.

For downstream generation, all four objectives start from the same Qwen3-1.7B warmup checkpoint and use the same 1B stable tokens. Native + AR applies native diffusion to the corrupted sequence and makes a second, clean causal pass. Its AR target positions, loss weight, and normalization match Full PCD. The table reports full-test-set results. Diffusion decoding uses token-causal attention within the prompt and generates answers in 32-token blocks with 16 denoising steps per block. Standard AR decoding generates one token at a time with greedy selection.

DecodingTraining objectiveGSM8K (%)MATH500 (%)HumanEval pass@1 (%)
DiffusionNative27.528.2012.20
DiffusionPCD without prefix AR loss29.047.8015.24
DiffusionFull PCD31.3112.2011.59
DiffusionNative + AR32.5210.4013.41
ARPCD without prefix AR loss44.7317.6026.22
ARFull PCD55.7229.2039.63
ARNative + AR57.9232.0037.20

Prefix AR supervision has its clearest benefit under AR decoding. Holding prefix visibility and attention fixed within PCD, adding the AR loss improves GSM8K, MATH500, and HumanEval by 10.99, 11.60, and 13.41 percentage points, respectively. Under diffusion decoding, it also improves GSM8K and MATH500, while HumanEval declines.

Native + AR is a strong baseline. Under diffusion decoding with a token-causal prompt, Native + AR solves 16 more GSM8K problems and 3 more HumanEval problems than Full PCD. Full PCD solves 9 more MATH500 problems. Under standard AR decoding, Native + AR leads by 2.20 points on GSM8K and 2.80 on MATH500, while Full PCD leads by 2.44 on HumanEval. Table 5 of the paper uses block-causal prompt attention. Under that setting, Native + AR scores slightly higher on all three diffusion-decoding benchmarks.

PCD obtains both training signals in one forward pass. The clean causal prefix supplies next-token targets and conditions suffix denoising. On the same eight H800s, Full PCD processed 150.0k original training tokens/s during parameter updates, compared with 89.7k for Native + AR. Processing the same 1B tokens took 14.82 and 24.79 GPU-hours of updates, respectively. Sharing the forward pass gave Full PCD 1.67× the training throughput and reduced update GPU-hours by about 40%.

Q2 / W3. Can a short switch to PCD late in training close the gap?

After 1B stable tokens, we continued the Native checkpoint with either Native or PCD and continued the Full PCD checkpoint with PCD. All three branches saw the same additional tokens in the same order. We used the block-causal prompt setting from Table 24. Avg. is the unweighted mean of the three unrounded benchmark scores.

Additional tokensTraining pathGSM8K (%)MATH500 (%)HumanEval pass@1 (%)Avg. (%)
+50MNative → Native32.909.8013.4118.71
+50MNative → PCD31.8412.2017.6820.58
+50MPCD → PCD34.8713.6012.8020.43
+75MNative → Native32.079.6017.6819.78
+75MNative → PCD31.4611.8017.6820.32
+75MPCD → PCD36.3213.6014.6321.52
+100MNative → Native31.6110.4017.0719.70
+100MNative → PCD30.1012.2016.4619.59
+100MPCD → PCD35.3313.4014.6321.12

The late switch essentially matches training with PCD throughout at +50M. PCD throughout leads at +75M and +100M. At +50M, the late-switch branch averages 20.58 versus 20.43 for PCD throughout. Its higher HumanEval score offsets its lower scores on both math benchmarks. At +75M and +100M, PCD throughout leads on average by 1.20 and 1.53 points. At +100M, it leads by 5.23 points on GSM8K and 1.20 on MATH500, while the late switch leads by 1.83 on HumanEval.

AuthorsOpenReview ↗

Response to Reviewer kumQ (2/3) — Q3–Q4

Q3 / W2. Which settings are reproducible, and do the gains hold across training seeds?

Appendix A.1 lists the settings we could verify for the original experiments. The Qwen-1.7B run used 50B stable tokens. In Table 3, Mixed PCD selects the prefix-conditioned branch with probability 0.5, and the within-sample PCD variant selects it with probability 1. Both use a prefix-fraction range of [0, 1]. The AR and denoising losses each have a default weight of 1 and are normalized by their respective target counts. The matched LLaDA2-Mini runs extend from 100B to 400B stable tokens and then use the same 50B-token decay. Evaluation uses no-shift SGLang with an 8K context. The original run records specify the settings above. For the public-data control, we give the full training and evaluation configuration below.

For the public-data control, we fixed the same 1B-token warmup checkpoint and trained Native and Full PCD independently under two stable-stage seeds. The data mixture by token count was 50% FineMath-4+, 30% Stack-Edu Python, and 20% FineWeb-Edu. Warmup used block sizes 4, 8, 16, and 32 in sequence. Stable training used document-isolated block attention at block size 32 for 7,630 updates of 64 × 2,048 tokens each. We used AdamW, raised the learning rate to 2 × 10⁻⁵ over 20 updates, and then applied cosine decay to 2 × 10⁻⁶. Both seeds were evaluated on 5-shot GSM8K and 4-shot MATH500 with block-causal prompt attention and 16 denoising steps per block (Table 22).

Stable seedGSM8K, Native → Full PCD (%)Gain (points)MATH500, Native → Full PCD (%)Gain (points)
4233.74 → 35.10+1.369.40 → 12.80+3.40
4332.98 → 34.50+1.5211.20 → 12.00+0.80

Full PCD solves 18 and 20 more GSM8K problems under the two seeds, and 17 and 4 more MATH500 problems. The direction of the gain reproduces on both benchmarks, with more variation in its size on MATH500.

Q4 / W4. How does PCD relate to prior objectives, and why use it at scale?

T5’s PrefixLM uses a bidirectional prefix for autoregressive prediction. XLNet learns bidirectional context through permuted autoregressive orderings. Nemotron-Labs-Diffusion uses clean and corrupted views to train AR and diffusion objectives jointly. PCD addresses the long stable stage of AR-to-diffusion conversion. Independent token corruption rarely leaves a long prefix intact. PCD samples a boundary in ordinary pretraining text so that the same clean prefix provides context for suffix denoising and next-token supervision in one forward pass.

Why not continue native block diffusion throughout the stable stage? Native block diffusion denoises each block conditioned on clean preceding blocks. To train all blocks in parallel, our continued-BD implementation processes both a corrupted view and a clean view of each sequence. Those views require additional computation for the same original training tokens. PCD predicts tokens in the prefix and denoises the suffix within one sequence, without processing a second full clean copy.

Starting from the same Qwen3-1.7B warmup checkpoint, we trained both branches on the same 1B original tokens, in the same order, on eight H800s. During parameter updates, continued BD at block size 32 processed 83.4k original tokens/s, compared with 150.0k for Full PCD. The same 1B tokens required 26.65 GPU-hours for continued BD and 14.82 for Full PCD. Thus, continued BD required 1.80× as many update GPU-hours to process the same data in our implementation. At a fixed compute budget, this difference determines how much pretraining data each objective can use during the stable stage.

On LLaDA2-Mini, we trained PCD for 400B stable tokens. The comparator at this scale was LLaDA2’s original Native stable objective. At the same token budget, PCD improved all 12 benchmarks and raised their average from 47.26 to 49.74. After the same 50B-token decay, PCD remained ahead on 10 of 12 benchmarks, with a 1.64-point average lead.

AuthorsOpenReview ↗

Response to Reviewer kumQ (3/3) — Q5

Q5 / W5. How does quality change with decoding speed, and can PCD support self-speculative decoding?

Direct diffusion decoding. We evaluated the two 1B stable checkpoints with the same cached decoder and block size 32. Using 8, 16, or 32 denoising steps per block commits 4, 2, or 1 token per step, respectively. Accuracy uses all 1,319 GSM8K and 500 MATH500 problems. For throughput, we used the first 96 problems of each benchmark across eight H800s, with one batch of 12 per GPU. We divided the total number of generated output tokens by the sum of the eight batch times, including prompt prefill. Arrows denote Native → Full PCD (Table 26).

Steps per blockGSM8K (%)GSM8K output tokens/s/GPUMATH500 (%)MATH500 output tokens/s/GPU
88.19 → 10.46162.6 → 114.01.00 → 1.20229.8 → 259.9
1633.74 → 35.1065.5 → 81.29.40 → 12.8094.8 → 91.3
3246.32 → 47.0147.4 → 60.221.20 → 26.4040.8 → 44.0

Full PCD has higher accuracy on both benchmarks at all three step budgets. At 8 steps, its lower GSM8K throughput mainly reflects shorter answers: Full PCD generates 234 tokens on average versus 327 for Native, with similar batch times of 24.63 and 24.11 seconds. At 32 steps, Full PCD improves both accuracy and throughput on both benchmarks.

Self-speculative decoding. The same Full PCD checkpoint drafts consecutive tokens in diffusion mode within a 32-token block, then verifies the draft in one causal forward pass. We compared this with greedy AR on both full test sets. Each request ran on one B300 at batch size 1. All settings used BF16, KV caching, and CUDA Graphs, and timing included prompt prefill (Table 6).

DecodingGSM8K (%)GSM8K output tokens/sMATH500 (%)MATH500 output tokens/s
Greedy AR55.42199.628.6205.4
Draft 4 tokens54.74256.028.2264.0
Draft 8 tokens54.74321.829.4347.3
Draft 16 tokens54.97332.929.4398.5

Drafting 16 tokens was the fastest setting we tested, reaching 1.67× greedy AR throughput on GSM8K and 1.94× on MATH500. It solved 725 GSM8K problems versus 731 for greedy AR, and 147 MATH500 problems versus 143. On both full benchmarks, accuracy stayed within one percentage point of greedy AR.

Reviewer_cd5hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the mismatch between full-sequence masked diffusion training, which can corrupt conditioning context, and continuation generation, which receives an intact prompt in the transition from AR models to dllmd. Prefix-Conditioned Diffusion (PCD) samples a sequence boundary, preserves the prefix, and applies same-position denoising supervision to masked suffix tokens. The full recipe also adds causal autoregressive supervision to the prefix. Applied during the stable stage of AR-to-diffusion conversion, it reports downstream gains on LLaDA2-Mini and Qwen-1.7B, with an average advantage remaining after matched decay on LLaDA2-Mini and no change to inference. Separate, smaller public experiments isolate a clean-prefix reconstruction benefit without AR supervision and examine sensitivity to evaluation-prefix corruption.

Critical strengths
  1. The paper addresses a concrete training–generation mismatch with a simple training change. Sampling a boundary produces clean conditioning context without requiring annotated prompt–response pairs. The matched stable-stage and decay comparisons assess whether this change has a lasting downstream effect, and the multi-domain evaluation gives the empirical finding practical relevance.

  2. The public controls make a useful distinction between recipe performance and prefix visibility. A paired intervention fixes attention, suffix targets, normalization, and optimization while changing training-prefix corruption, with no AR loss. Multiple stable-stage seeds and a fresh holdout support the reconstruction result. The paper also reports the reversal under evaluation-prefix corruption and the unsuccessful generation-readiness check, making the limits of these controls explicit.

Critical weaknesses
  1. Missing a direct comparison with continued block diffusion during the stable stage. LLaDA2.0 already uses denoising conditioned on clean preceding blocks during warmup and decay. The main PCD experiments compare against native diffusion with full-sequence corruption during the stable stage. Consequently, the reported results do not establish whether PCD offers additional benefits over simply continuing an existing block diffusion objective during this stage.

  2. Limited methodological novelty beyond combining existing mechanisms and adjusting their use across training stages. Training with a block-diffusion attention architecture has been a well known method in dllm training. Denoising conditioned on a clean prefix, causal attention, and joint AR–denoising supervision all have precedents. The paper also acknowledges that its prefix–suffix configurations form a restricted case of Adaptive Block Diffusion. The main distinction therefore appears to concern how these mechanisms are combined and applied during the stable stage; a substantive methodological contribution in the attention mechanism or learning objective remains unclear.

Questions for authors
  1. Why not continue standard block diffusion during the stable stage? Could the authors directly compare this alternative with PCD under matched initialization, data, and token budgets, followed by the same decay and inference procedures? Given the computational motivation for LLaDA2.0’s transition to full-sequence diffusion, please also report the training compute required by both approaches.

  2. What specific methodological contribution does PCD introduce beyond existing block diffusion, Adaptive Block Diffusion, and joint AR–diffusion training? Could the authors distinguish the empirical findings from adjusting the training recipe from the methodological novelty, and provide a matched downstream-task ablation to determine whether the prefix AR loss adds value beyond clean-prefix denoising alone? This will demonstrate a convincing novelty of authors' method

1 comment

Thank you for raising these two questions. We ran two matched comparisons on Qwen3-1.7B to address them directly.

W1 / Q1. Why use PCD during the stable stage instead of continuing standard block diffusion?

We chose native diffusion as the baseline in the paper because it is the stable-stage configuration used by LLaDA2.0. Standard block diffusion (BD) processes both a noisy sequence and a clean sequence that provides conditioning context. LLaDA2.0 switches to full-sequence diffusion during the stable stage, which removes the extra clean-sequence branch. Throughput matters at this scale. In our LLaDA2-Mini experiments alone, each branch processes 400B tokens during the stable stage. Our motivation was to retain the throughput needed for this volume of data while giving the model an intact prefix during training, as it receives at generation time.

The suggested continued-BD baseline lets us test that trade-off directly. Given the scale of the main runs, we used Qwen3-1.7B, the backbone of our public-data experiments. Both branches started from the same checkpoint after 1B tokens of AR-to-BD warmup. One switched to Full PCD and the other continued standard BD with block size 32. Each branch trained on the same 1B stable tokens in the same order, with matched optimizer settings and learning rates. We also matched the evaluation prompts, decoding settings, and scoring procedures.

Method after 1B stable tokensGSM8KMATH-500HumanEval pass@1Training tokens/sGPU-hours for optimizer updates
Full PCD35.10%12.80%15.85%150.0k14.82
Continued BD32.75%5.60%15.24%83.4k26.65

On the same eight H800 GPUs, PCD processed about 1.8× as many training tokens per second and used 44% fewer GPU-hours for optimizer updates. The clearest downstream difference was on MATH-500, where PCD solved 64 problems compared with 28 for continued BD. PCD also solved 31 more GSM8K problems (463 vs. 432), while HumanEval differed by one problem (26 vs. 25). After both branches underwent the same 50.07M-token BD decay at a fixed block size of 32 and were evaluated with the same decoder, PCD remained ahead by 3.64 percentage points on GSM8K and 4.00 points on MATH-500, with a one-problem difference on HumanEval.

W2 / Q2. What does PCD contribute, and what does prefix AR supervision add?

At generation time the model sees the whole prompt. Under full-sequence masking, the longer the prefix, the less likely it is to remain intact at a fixed masking rate. A direct way to address this mismatch is to make the boundary between context and continuation part of the stable-stage objective. In PCD, one boundary sampled from ordinary pretraining text determines both where masking is allowed and how each side is supervised. Varying the boundary gives the model intact prefixes of different lengths without annotated prompt–response pairs. The intact prefix provides context for reconstructing masked suffix tokens at their original positions and also supplies causal next-token targets. Both objectives use a single input sequence, without the separate clean copy used in standard BD training. We include the prefix loss to help preserve the pretrained model’s ability to generate autoregressively.

After 400B stable tokens per branch in our matched LLaDA2-Mini experiment, PCD scored higher than native full-sequence diffusion on all 12 benchmarks, raising the mean from 47.26 to 49.74 (+2.48 points). In a separate Qwen3-1.7B control, we isolated prefix visibility by disabling the AR loss and varying only whether the training prefix could be corrupted. With an intact prefix at evaluation, preserving it during training reduced suffix reconstruction loss in all three training seeds, by 0.0692 nats on average.

For the requested downstream ablation, both branches retained the clean prefix and shared the same warmup checkpoint, 1B stable tokens, data order, sampled boundaries, attention masks, suffix targets, and learning-rate schedule. The only change was the prefix AR loss weight, from 1 to 0.

Method after 1B stable tokensGSM8KMATH-500HumanEval pass@1
Full PCD463/1319 (35.10%)64/500 (12.80%)26/164 (15.85%)
PCD with prefix AR loss disabled455/1319 (34.50%)57/500 (11.40%)22/164 (13.41%)

With the clean prefix held fixed, adding AR supervision yielded 8 more correct answers on GSM8K, 7 on MATH-500, and 4 on HumanEval. The gains were modest and consistent in direction across the three tasks at this budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.