acceptodds
Under review as a conference paper at ICLR 2027

PBPO: Prefix-Block Policy Optimization for Diffusion Language Models

Abstract

Diffusion Large Language Models (dLLMs) commonly decode responses by proceeding autoregressively across blocks while iteratively denoising multiple tokens within each block. This structure makes block-level policy optimization a natural adaptation of autoregressive policy optimization. However, the exact likelihood of a completed block is intractable because it marginalizes over the intermediate states and reveal orders of the denoising process. To realize the block-level formulation, we construct a prefix-block state and use a single model evaluation to define a factorized one-step surrogate for the block likelihood. The resulting block surrogate ratio is a product of its token surrogate ratios, allowing variation at individual positions to propagate across the entire block before clipping. We therefore introduce Prefix-Block Policy Optimization (PBPO), which preserves block-level conditioning but applies surrogate importance weighting and clipping separately to each token. PBPO further diversifies repeated updates by resampling corrupted views of the prompt and revealed prefix, a design motivated by masked-dLLM pre-training. Under our analysis assumptions, block-product weighting amplifies the variance of gradient contributions relative to token-local weighting. Across two LLaDA-family initializations, six benchmarks, and three generation lengths, PBPO achieves competitive performance against existing dLLM RL methods.

Primary area:
reinforcement learning
Keywords:
Diffusion Model, Policy Optimization, Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_JJsAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Prefix-Block Policy Optimization (PBPO), a RLVR algorithm designed for masked DLLMs that generate text via block-wise decoding. The authors uses a prefix-block surrogate and uses decoupled clipping since the exact conditional likelihood of a block is intractable for direct policy gradient. The results show that PBPO outperforms recent DLLM RL baselines on reasoning benchmarks.

Critical strengths
  1. The algorithm is easy to follow and clean. Decoupling the block-level conditioning state from token-level importance ration clipping is intuitive and practical. PBPO evaluates all token probabilities in one block with a single forward pass and prevent outlier token ratios by multiplying together and destabilizing the entire block update. The empirical results shows improvements over multiple RL baselines.
Critical weaknesses
  1. During rollout with block length 32, the model takes 16 denoising steps per block. But the training surrogate evaluates all 32 tokens conditioned only on the entire block is fully masked. So the policy is never optimized on intermediate intra-block states, which is a mismatch between rollout generation and policy optimization
  2. Per-update masking of the revealed prefix is presented as a core contribution of PBPO over d1's prompt masking method. But the Table 2 show that adding prefix masking on MATH500 hurts the accuracy compared to prompt-only masking. So it is unclear how much improvement over d1 comes from prefix block conditioning.
Questions for authors

Please check the weakness above.

4 comments
AuthorsOpenReview ↗

Response from Authors - Part 1/2

Dear Reviewer JJsA,

We thank you for recognizing the clarity and practicality of PBPO and its empirical improvements over existing dLLM RL baselines. We address your concerns and questions point by point below.

W1. Relationship between multi-step rollout generation and the training surrogate

We would like to clarify that PBPO’s one-step predictions provide a tractable surrogate for the likelihood of multi-step block-wise generation. The exact likelihood of both a completed block and the full response is intractable because it requires marginalizing over the intermediate denoising states and token-reveal orders that can produce the same output. In our formulation, a completed block is treated as a block-level action whose internal denoising trajectory is latent. Its conditional likelihood takes the form

πθ(oi(b)∣q,oi(<b))=∑τb∈T(oi(b))pθ(oi(b),τb∣q,oi(<b))\pi_\theta(o_i^{(b)} \mid q, o_i^{(<b)}) = \sum_{\tau_b \in \mathcal{T}(o_i^{(b)})} p_\theta(o_i^{(b)}, \tau_b \mid q, o_i^{(<b)})

where the sum ranges over compatible intra-block trajectories.

Under this block-level formulation, policy optimization can be expressed through the marginal likelihood of completed-block actions; a separate objective at every intermediate state is not required. We approximate the intractable marginal using factorized one-step predictions from the prefix-block state, allowing all token probabilities in a block to be evaluated with one forward pass. This base state uses the context present at an actual rollout boundary: preceding blocks are revealed, while the current and future blocks remain masked. Thus, our surrogate retains dependencies across completed blocks while making their optimization computationally tractable.

This likelihood challenge also motivates prior dLLM RL methods, which use tractable surrogates or reformulated objectives. For example, d1 uses one-step token predictions under a mean-field approximation, wd1 optimizes reward-weighted token likelihoods, and GDPO constructs a sequence-level ratio from a diffusion lower bound. Approximating the distribution induced by multi-step denoising is therefore a central challenge of dLLM RL. Our contribution is to ground the surrogate in block-boundary conditioning and separate this conditioning structure from token-level importance weighting, with theoretical analysis and empirical evidence supporting the resulting optimization behavior.

We chose block length 32 for the main experiments to follow the multi-step, block-wise rollout procedure used by d1 and wd1 and maintain an aligned comparison, as described in Section 4.1 and Appendix A.2. Since changing the block length or other rollout-generation settings can affect rollout quality and, consequently, downstream performance, we keep these settings aligned with prior work. Therefore, the comparison more clearly distinguishes gains associated with the optimization objectives and likelihood estimators from changes in rollout generation.

AuthorsOpenReview ↗

Response from Authors - Part 2/2

W2. Contribution of prefix-block conditioning versus additional prefix masking

We would like to clarify that prefix-block conditioning and additional prefix masking serve different roles. Our prefix-block state reveals the completed prefix and masks the current and future blocks; additional prefix masking optionally corrupts part of that revealed prefix. In Table 2, we retain prefix-block conditioning in every configuration and vary only conditioning corruption. Therefore, the MATH500 comparison between prompt-only and prompt-plus-prefix masking measures the effect of additional prefix corruption.

Our existing results already show improvements over d1 with a clean revealed prefix. Specifically, the prompt-only PBPO variant in Table 2 outperforms the corresponding d1 results in Table 1 on both MATH500 and Countdown at all three generation lengths, using the same LLaDA-1.5 initialization. We also use prompt masking with a clean revealed prefix in the main MATH500 configuration, as specified in Appendix A.2.

We provide an additional controlled PBPO–d1 comparison on GSM8K and MATH500 using LLaDA-8B-Instruct (Table R1). In these runs, we use prompt masking only, no additional prefix masking, symmetric clipping coefficients of 0.5/0.5 as in d1, and prompt-masking probability 0.15. Notably, we already use the same prompt-masking probability as d1 in all experiments reported in the paper.

For both PBPO and d1, we use the unified training settings described in Section 4.1 and Appendix A.2, including the optimizer, learning rate, batching, and rollout generation, in the original and controlled experiments. Under these aligned settings, the primary methodological difference is the likelihood estimator: our estimator evaluates token probabilities from the corresponding prefix-block state, while d1 uses its one-step mean-field estimator.

Table R1. Controlled PBPO–d1 comparison on LLaDA-8B-Instruct. Each entry reports accuracy (%) / average effective length (tokens). The d1 row reproduces the main-paper results. Evaluation follows the unified protocol in Section 4.1. The higher accuracy in each column is bolded.

MethodGSM8KMATH500
Generation length128256512128256512
PBPO (controlled)76.72 / 113.7884.08 / 221.0884.38 / 281.7032.60 / 111.0037.40 / 217.2638.00 / 362.63
d172.55 / 115.8578.01 / 217.1381.20 / 275.1529.00 / 122.0534.40 / 234.4737.60 / 392.75

We observe that PBPO outperforms d1 in all six accuracy comparisons, with gains of 3.18–6.07 percentage points on GSM8K and 0.40–3.60 on MATH500. Thus, these results support the effectiveness of prefix-block conditioning.

On MATH500, we also observe shorter effective responses from PBPO than from d1 at every tested generation length, pairing higher accuracy with more compact responses.

Thank you for the detailed rebuttal and additional experiments.

I still have some concerns regarding the novelty relative to related methods and the lack of a theoretical justification of the surrogate approximation.

I will keep my original score unchanged, but my opinion is more on the borderline.

AuthorsOpenReview ↗

Follow-up response from Authors

Dear Reviewer JJsA,

We thank you for your continued consideration of our rebuttal and the additional experiments. We would like to clarify the two remaining concerns below.

Novelty relative to related methods

We position PBPO’s contribution in its block-boundary policy-optimization formulation and the analysis of its importance-weighting structure. Different response blocks are evaluated from different revealed prefixes, while all tokens within a block share the same conditioning state. We derive the corresponding block-product objective and then retain its conditioning structure while assigning surrogate importance weights and clipping operations separately to individual tokens. Thus, PBPO explicitly separates the granularity of conditioning from the granularity of importance weighting.

We have provided a detailed comparison with d1 in our response to Reviewer 2Ubt (W1), and broader comparisons with d2-StepMerge, SPG, and other related methods in our response to Reviewer ivyd (W1). These comparisons distinguish the scoring states, approximation targets, and update rules, while explicitly identifying shared components. For example, SPG estimates evidence bounds from sampled, partially masked active blocks, whereas PBPO scores every block at its fully masked boundary and uses individually clipped current-to-behavior ratios. We also discuss the relationship between d2-StepMerge’s temporal segments and PBPO’s block-boundary states. We invite you to consult these responses for the fuller methodological comparison. In the revised manuscript, we will expand the related-work discussion and clarify the attribution and scope of our contributions.

The additional experiments support the practical value of this formulation under aligned settings. As reported in our initial response to you, PBPO outperforms d1 in all six controlled accuracy comparisons on GSM8K and MATH500. Our response to Reviewer ivyd (W4) additionally reports coding results in which PBPO outperforms ESPO in all six comparisons and SPG in five of six, including all three HumanEval generation lengths. These results complement the methodological discussion with comparisons against additional related objectives.

Theoretical motivation and scope of the surrogate analysis

For the surrogate construction, we start from the completed-block marginal likelihood, which integrates over the latent intra-block denoising trajectories. We choose the block-boundary state because it is the context present immediately before that block is generated: preceding blocks are revealed, while the active and future blocks are masked. The factorized one-step predictions provide a tractable block surrogate that preserves dependencies across completed blocks and allows all positions within a block to be scored in one forward pass. Therefore, the approximation is grounded in the conditioning structure of the rollout policy.

For the importance-weighting choice, Theorem 1 holds this conditioning structure and the one-step surrogate fixed, and isolates the variance amplification caused by multiplying token ratios into a shared block weight. Under the assumptions stated in Appendix C, the block-product contribution has greater total variance than the token-level contribution. The same comparison is preserved when both contributions are multiplied by a fixed nonzero advantage.

We discuss the theoretical scope more fully in our response to Reviewer ivyd (W2). The theorem characterizes gradient variance within the specified surrogate probability space; the approximation error of the true multi-step marginal is a separate quantity. The practical utility of the surrogate and the complete clipped objective is assessed through our controlled comparisons and downstream evaluations. We invite you to consult that response for the detailed assumptions and the respective roles of the theoretical analysis and empirical evidence. We will make these distinctions more explicit in Sections 3.1–3.4 of the revised manuscript.

Reviewer_AHrLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an RL post-training method for masked diffusion language models with block decoding. Computing the likelihood of a response is intractable in these models, since it requires summing over all the possible orders in which the tokens can be unmasked. The paper approximates it with one forward pass per block, conditioned on the previously decoded blocks, and uses this estimate in a GRPO-style objective. It also randomly masks parts of the prompt and of the previous blocks at each update. The proposed method is evaluated with two models and 6 benchmarks, outperforming previous approaches in most settings.

Critical strengths
  • The research direction is interesting and important. Masked diffusion models are becoming more mature and are starting to rival AR models of similar size, so good RL pipelines for them are much needed. The main obstacle is that the likelihood is intractable and this paper proposes to approximate it.
Critical weaknesses
  • The models used (LLaDA-8B-Instruct and LLaDA-1.5) are not trained in block-AR mode, and only the generation is block by block. Since the method is built around prefix-block conditioning, it would be good to see how it depends on the block length used during training and how it behaves when the model is actually trained like this.
  • It is hard to understand which parts of the method bring the gains. Compared to d1, PBPO changes the likelihood estimator, but also uses clip-higher, prefix masking and per-task hyperparameters. The ablation in Table 2 only covers the masking, and without it PBPO drops below GDPO and DMPO in almost all cases. A controlled comparison isolating each component of the method is needed, in particular the prefix-block estimator against d1's.
Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response from Authors

Dear Reviewer AHrL,

We thank you for recognizing the importance of this research direction and the motivation for developing tractable RL post-training methods for masked dLLMs. We address your concerns point by point below.

W1. Dependence on training block length and compatibility with the backbone’s pre-training

Prefix-block conditioning is defined by the block-wise generation policy: each state represents the context visible immediately before the next block is denoised. PBPO can therefore optimize this generation policy using a masked-diffusion backbone; block-autoregressive pre-training is not a prerequisite for constructing these states. LLaDA’s masked-denoising formulation already accepts partially masked inputs, including the revealed-prefix and masked-suffix states used by PBPO.

We chose training block length 32 to match the rollout configuration used in prior work, including d1 and wd1, and maintain an aligned comparison. Since changing the block length or token-selection strategy can affect rollout quality and downstream performance, sharing the rollout-generation configuration helps distinguish improvements associated with the optimization objectives and likelihood estimators.

We provide additional MATH500 results for PBPO initialized from LLaDA-8B-Instruct and trained with block lengths 16 and 64. Table R1 includes these variants alongside the main-paper result at block length 32.

Table R1. PBPO with different RL training block lengths, initialized from LLaDA-8B-Instruct. Each entry reports accuracy (%) / average effective length (tokens). The block-length-32 row reproduces the main-paper result. All variants are evaluated with block length 32 using the protocol in Section 4.1. The highest accuracy in each column is bolded.

Training block lengthMATH500
Generation length128256512
1631.20 / 122.2035.40 / 229.2840.40 / 372.35
32 (main paper)34.80 / 121.6236.80 / 226.6141.40 / 359.10
6434.60 / 123.0638.40 / 237.6138.80 / 407.39

PBPO remains effective across training block lengths 16, 32, and 64: all three variants improve over the LLaDA-8B-Instruct initialization on MATH500 at every evaluated generation length. These results support PBPO’s applicability across the tested training block lengths.

W2. Attribution of gains and a controlled comparison with d1

We provide additional experimental results comparing PBPO with d1 on GSM8K and MATH500 using LLaDA-8B-Instruct under aligned training settings (Table R2). In these additional experiments, we configure PBPO with prompt masking only, no additional prefix masking, symmetric clipping coefficients of 0.5/0.5 as in d1, and prompt-masking probability 0.15. We use these clipping and prompt-masking settings for both tasks in this comparison. Notably, the prompt-masking probability already matches d1 in all experiments reported in the paper.

In both the original and controlled experiments, PBPO and d1 share the unified training settings described in Section 4.1 and Appendix A.2, including the optimizer, learning rate, batching, and rollout generation. Under these aligned settings, the primary methodological difference is the likelihood estimator: PBPO evaluates token probabilities from the corresponding prefix-block state, while d1 uses its one-step mean-field estimator.

Table R2. Controlled comparison between PBPO and d1 on LLaDA-8B-Instruct. Each entry reports accuracy (%) / average effective length (tokens). The d1 row reproduces the main-paper results. Evaluation uses the same protocol as Table R1. The higher accuracy in each column is bolded.

MethodGSM8KMATH500
Generation length128256512128256512
PBPO (controlled)76.72 / 113.7884.08 / 221.0884.38 / 281.7032.60 / 111.0037.40 / 217.2638.00 / 362.63
d172.55 / 115.8578.01 / 217.1381.20 / 275.1529.00 / 122.0534.40 / 234.4737.60 / 392.75

PBPO outperforms d1 in all six accuracy comparisons, with gains of 3.18–6.07 percentage points on GSM8K and 0.40–3.60 on MATH500. Thus, these results support the effectiveness of the prefix-block estimator under the controlled settings.

We would also like to clarify the comparison of the fully clean PBPO variant in Table 2 with GDPO and DMPO. Table 2 studies conditioning diversification within PBPO. Its fully clean variant removes prompt masking, a component also used by d1. Consequently, comparing this ablation with complete baseline configurations involves component differences beyond the likelihood estimator. The ablation rows quantify the effect of conditioning diversification within PBPO and should be interpreted in that context.

Thanks for answering my comments and providing additional results.

Regarding needing a model that was pre-trained in block-AR mode, I agree that this is not a requirement, specially if you post-train in this setting. Nonetheless, all modern diffusion models are trained in a block-AR fashion (newer versions of LLaDA, DiffusionGemma, etc) because you reach better performance. My suggestion was to use any of these newer methods and apply the proposed approach there, where you actually leverage the fact that the pre-training compute was spent on the block-AR setting instead of introducing this during post-train.

Overall, I am still indecisive, specially considering that other reviewers felt the same about the paper and might have a higher confidence rating.

AuthorsOpenReview ↗

Follow-up response from Authors

Dear Reviewer AHrL,

We thank you for clarifying the motivation behind your suggestion and for considering our additional results. We appreciate the relevance of evaluating PBPO on newer backbones pretrained with block-AR decoding, and would like to explain how this relates to our experimental design and formulation.

Choice of backbones and controlled evaluation

We chose LLaDA-8B-Instruct and LLaDA-1.5 to evaluate the contribution of PBPO under aligned initializations and training settings with the existing RL baselines. Changing the backbone or its pretraining procedure can affect both its initial capabilities and the quality of the subsequent rollouts. Therefore, sharing the initialization, optimizer, batching, and rollout-generation settings helps attribute performance differences to the optimization methods. Block-wise rollout generation is already used by prior methods such as d1 and wd1; we follow that procedure, including block length 32, as described in Section 4.1 and Appendix A.2.

The additional controlled comparison directly supports the benefit of the prefix-block estimator within this setting. With prompt masking and clipping aligned with d1, PBPO achieves higher accuracy in all six comparisons on GSM8K and MATH500. Thus, the reported improvement persists under the aligned configuration described in our initial response.

Applicability to backbones pretrained with block-AR decoding

For a backbone pretrained with block-AR decoding, the revealed-prefix and masked-active-block state used by PBPO corresponds directly to its block-wise conditioning structure. PBPO can use the backbone’s block-denoising predictions to construct token surrogate ratios while retaining its native block attention and decoding configuration. Consequently, the learned block-wise dependencies remain available through the pretrained initialization, and the policy objective can operate on the same conditioning structure.

We appreciate your suggestion and view these newer backbones as natural candidates for extending PBPO’s empirical evaluation. To provide context on the resources required, DiffusionGemma is a 26B-class model; LLaDA2.0-Flash and LLaDA2.1-Flash each have 100B total parameters, while LLaDA2.0-Mini and LLaDA2.1-Mini each have 16B. LLaDA2.0-Uni likewise uses a 16B language-model backbone. These parameter footprints exceed the 8B scale of our initializations and increase the memory required for training and rollout collection.

Given our available compute budget and the time available during the rebuttal period, we prioritized completing controlled comparisons and evaluation across different denoising settings at the 8B scale. Extending the matched RL comparisons to these larger backbones requires additional computational resources and experiment time. We value this extension and will clarify both PBPO’s applicability to block-AR-pretrained models and the practical rationale for our experimental scope in the revised manuscript.

Empirical evidence

Our evaluation establishes consistent gains across two LLaDA-family initializations, six benchmarks, and three generation lengths: PBPO leads or ties the strongest baseline in 32 of the 36 main comparisons. The additional controlled results further support the effectiveness of the estimator under aligned settings.

Although we post-train PBPO using a single block-AR rollout configuration, its performance gains persist across evaluation settings with different block lengths and token-selection strategies. We examine this generalization in Section 4.2 and Appendix B.3 by reusing the LLaDA-1.5 policies trained with confidence-based block-wise decoding and block length 32. Their trained parameters remain fixed throughout these evaluations. Specifically, we evaluate confidence-based decoding with low-confidence remasking at block lengths 16 and 64 (Tables 7–10), and random token selection at block lengths 16, 32, and 64 (Tables 11–16). All five evaluation configurations use a maximum generation length of 256 and 128 denoising steps across the same six benchmarks.

PBPO achieves the highest accuracy on GSM8K and Sudoku in every one of these five configurations. Across the 30 benchmark–configuration comparisons, PBPO leads in 20 and ties the leading baseline in two. Under random token selection, PBPO ranks first or second in 17 of the 18 comparisons; at block length 16, PBPO leads or ties on all six benchmarks. Therefore, the same policies trained under one block-AR rollout setting retain a performance advantage across multiple evaluation-time denoising configurations, supporting PBPO’s effectiveness beyond its training rollout configuration.

Reviewer_ivydOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

PBPO applies tokenwise clipped policy optimization to block-boundary predictions, with resampled prompt/prefix corruption. Six-benchmark results are useful, but substantial overlap with prior methods and missing comparisons leave its algorithmic novelty unestablished.

Critical strengths

The method is practically simple: one forward pass scores each active block while retaining completed prefixes. The approach is straightforward to implement on existing masked DLMs.

Critical weaknesses
  1. Lack of novelty and substaintially missing related work: d2-StepMerge (2509.21474, Eq. 9) already scores each temporal segment in one pass and clips individual token ratios. Aligning segments with block boundaries recovers clean-prefix PBPO's objective when advantage scaling, clipping, and KL settings match. SPG (2510.09541, Section 3.3 and Algorithm 1) already combines clean preceding blocks, masked future blocks, prompt/prefix perturbations, and resampling within repeated updates. PBPO restricts scoring to fully masked active blocks and changes the loss. SPG appears in the bibliography but receives no substantive comparison. The paper must identify what remains algorithmically new, which is totally unclear to me.

Other prior work missing or receiving deficient discussion: TraceRL, DiRL/DiPO, GSPO, ESPO, JustGRPO, StableDRL, Lightning RL.

  1. As acknowledge by the authors, the theorem does not establish the practical mechanism. Appendix C compares unclipped, advantage-free contributions under an iid factorized surrogate. It neither bounds bias from multi-step rollout versus boundary-state scoring nor establishes lower variance for reward-weighted, clipped updates.

  2. The block comparison is confounded. Equations 6 and 7 normalize by blocks and tokens, changing raw gradient scale. The collapse of an unnormalized product-ratio baseline does not exclude established alternatives such as GSPO/ESPO-style length normalization.

  3. The empirical case omits decisive baselines: d2, SPG, ESPO, JustGRPO, and StableDRL are absent. The actual performance do not exceed these baselines.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response from Authors - Part 1/3

Dear Reviewer ivyd,

We thank you for recognizing PBPO’s practical simplicity and its compatibility with existing masked diffusion language models. We address your concerns and questions point by point below.

W1. Relationship to prior methods and the contribution of PBPO

We would like to clarify that PBPO’s contribution concerns the complete optimization formulation and its analysis. We score every response block from its corresponding boundary state, with the active and future blocks fully masked; we then apply current-to-behavior surrogate ratios and clipping separately to individual tokens, while resampling corrupted conditioning views across repeated updates. Thus, the comparison with prior methods involves the scoring states, the likelihood surrogate, and the update rule together. In the revised manuscript, we will expand the related-work section with substantive comparisons to d2-StepMerge [1], SPG [2], and the other cited methods, explicitly attributing shared components and clarifying PBPO’s methodological contribution.

Comparison with d2-StepMerge. d2-StepMerge [1], Equation 9, approximates the likelihood of a sampled denoising trajectory by partitioning it into temporal segments and scoring tokens revealed within each segment from its starting state. PBPO constructs its scoring states directly from completed response blocks, and evaluates every token in a block without conditioning on its realized intra-block reveal order. When temporal segments coincide exactly with decoded blocks and the remaining objective settings match, the clean-prefix token-ratio terms can coincide. We will state this relationship explicitly. PBPO further combines boundary scoring with freshly corrupted prompt/prefix views during repeated updates, and develops the direct block-level objective alongside the token-level objective to analyze the effect of changing importance-weighting granularity while preserving the conditioning structure. This formulation, analysis, and systematic evaluation constitute our contribution beyond the shared tokenwise clipped expression.

Comparison with SPG. SPG [2] shares the use of structured block-wise conditioning and resampled perturbations. Its Section 3.3 randomly selects an active block and masks tokens within that block at sampled noise levels; its objective maximizes an evidence lower bound for positive-advantage responses and uses an evidence upper bound or upper/lower-bound mixture for negative-advantage responses. In PBPO, we evaluate all response blocks at their fully masked boundaries and use their predictions in individually clipped current-to-behavior ratios. Consequently, the active-block inputs and gradient weights differ even when prompt/prefix perturbations are shared. Additional prefix masking diversifies conditioning within PBPO, while the boundary-state surrogate and its token-level policy objective define the optimization formulation. We provide a direct empirical comparison with SPG under unified settings in W4.

Trajectory-based methods and rollout choices. TraceRL [3] incorporates denoising trajectories into policy optimization and introduces a diffusion-based value model for step-wise advantages. DiRL/DiPO [4] uses trajectory-conditioned token probabilities within an efficient post-training framework for block diffusion models. These methods make different choices about the trajectory information retained during scoring. JustGRPO [5] uses autoregressive generation during RL training, whereas PBPO retains multi-token denoising within rollout blocks. LightningRL [6] additionally optimizes inference parallelism through its reward design, normalization, and regularization. We will expand the related-work discussion to explain these distinctions in scoring, rollout generation, and training objectives.

Sequence-level weighting and stabilization. GSPO [7] uses length-normalized sequence-level importance weights; ESPO [8] adapts this approach to diffusion models using sequence-level ELBO estimates. StableDRL [9] addresses noisy estimated ratios through unconditional clipping and self-normalized gradient weights. These are relevant alternatives for controlling update behavior, and we will discuss them explicitly alongside PBPO’s separation of conditioning and importance-weighting granularities. W3 explains the distinction between objective averaging and ratio normalization, and W4 reports additional ESPO results.

AuthorsOpenReview ↗

Response from Authors - Part 2/3

W2. Scope and role of the theoretical analysis

We would like to clarify the mechanism isolated by Theorem 1. Both gradient contributions use the same prefix-block conditioning and the same factorized one-step surrogate. The comparison changes only whether each token’s score receives its own surrogate ratio or a shared product of all ratios in the block. Under the conditional independence, identical-moment, and regularity assumptions stated in Appendix C, the shared product produces strictly greater total gradient variance. Thus, the analysis identifies a source of variance amplification within the specified surrogate model.

For advantage weighting, Appendix C, Remark 3 already states that multiplying both contributions by the same fixed nonzero advantage scales their covariance matrices by the square of that advantage and preserves the comparison. The factorization assumptions concern the one-step surrogate probability space; they do not assert independence of tokens generated through the actual multi-step denoising process.

The relationship between rollout likelihood and boundary scoring concerns the likelihood approximation itself. PBPO treats the completed block as the action and its internal denoising trajectory as latent, then constructs a tractable surrogate for the intractable completed-block marginal. Theorem 1 characterizes importance-weighting variance within that surrogate; it does not supply a bound on the approximation error of the multi-step marginal or a variance guarantee for the complete reward-weighted, clipped update. We will foreground this scope in Section 3.4, consistent with the existing qualifications in Appendix C.

For the practical objective, our evidence comes from the training and evaluation results. Figure 1 demonstrates the behavior of the implemented block-level and token-level objectives, while the main experiments assess downstream performance across two initializations, six benchmarks, and three generation lengths. The additional SPG and ESPO comparisons in W4 further evaluate the complete training method against alternative objectives. Therefore, our justification combines an explicit analysis of weighting granularity with empirical evidence for the practical formulation.

W3. Objective averaging and length-normalized importance ratios

We would like to distinguish the outer averaging of objective terms from normalization of the importance ratio itself. Equation 6 averages block terms, whereas Equation 7 averages token terms. Theorem 1, however, compares both contributions using the same per-block sum convention, independently of these outer denominators. Under its stated assumptions, averaging both contributions over the same block length preserves the variance comparison:

Var⁡ ⁣[gblockB]−Var⁡ ⁣[gtokenB]=Var⁡[gblock]−Var⁡[gtoken]B2>0,B:=block length\operatorname{Var}\!\left[\frac{g_{\mathrm{block}}}{B}\right]-\operatorname{Var}\!\left[\frac{g_{\mathrm{token}}}{B}\right]=\frac{\operatorname{Var}[g_{\mathrm{block}}]-\operatorname{Var}[g_{\mathrm{token}}]}{B^2}>0,\qquad B := \text{block length}

Thus, the theoretical difference between the weighting structures persists when their averaging conventions are matched. This clarifies the role of normalization in interpreting the theorem separately from the raw gradient scales of the implemented objectives.

GSPO-style length normalization changes the ratio itself, and ESPO applies this principle to sequence-level ELBO differences. Applied to our factorized block surrogate, length normalization would replace the product ratio with its geometric mean:

rgeom=(∏k=1Bρk)1/B=exp⁡ ⁣(1B∑k=1Blog⁡ρk)r_{\mathrm{geom}}=\left(\prod_{k=1}^{B}\rho_k\right)^{1/B}=\exp\!\left(\frac{1}{B}\sum_{k=1}^{B}\log\rho_k\right)

where the factors are the individual token surrogate ratios. This reduces the scale of the aggregate log-ratio, but still assigns a shared weight and clipping decision to all token contributions in the block. For example, reciprocal token ratios can yield a geometric mean of one even when the individual ratios are far from one. Clipping that aggregate therefore imposes a different constraint from clipping each token ratio separately.

Our analysis motivates token-local weighting within the prefix-block surrogate. Figure 1 reports the direct block-product objective specified in the paper, rather than a comparison against every possible normalized alternative. We now also provide an empirical comparison with ESPO, which uses the length-normalized sequence-level construction identified in the review, under the unified settings described in W4.

AuthorsOpenReview ↗

Response from Authors - Part 3/3

W4. Additional baseline comparisons under unified settings

We provide additional experimental results for SPG and ESPO on HumanEval and MBPP, alongside the PBPO results from the main paper. These comparisons use LLaDA-8B-Instruct and the unified experimental settings described in Section 4.1 and Appendix A.2, including the optimizer, learning rate, batching, rollout generation, and evaluation. The SPG and ESPO rows are new results, and the PBPO row reproduces the corresponding LLaDA-8B-Instruct entries in Tables 1 and 6.

Table R1. PBPO, SPG, and ESPO on coding benchmarks. Each entry reports execution-based pass@1 (%) / average effective length (tokens). All methods are evaluated in the zero-shot setting at maximum generation lengths 128, 256, and 512. The highest pass@1 in each column is bolded.

MethodHumanEvalMBPP
Generation length128256512128256512
PBPO30.49 / 95.8137.20 / 117.8637.80 / 127.5943.97 / 84.7248.25 / 93.3547.47 / 97.53
SPG28.05 / 91.8032.93 / 112.6932.93 / 123.1647.47 / 81.9543.97 / 89.8144.75 / 95.33
ESPO21.95 / 119.1136.59 / 195.9835.98 / 259.0142.41 / 117.4744.75 / 194.1442.80 / 254.18

We observe that PBPO achieves higher pass@1 than ESPO in all six comparisons and higher pass@1 than SPG in five of six comparisons. In particular, PBPO outperforms both methods on HumanEval at every generation length, exceeding SPG by 2.44–4.87 percentage points and ESPO by 0.61–8.54 points. On MBPP, PBPO exceeds ESPO by 1.56–4.67 points across all three lengths and exceeds SPG by 4.28 and 2.72 points at lengths 256 and 512, respectively.

PBPO also produces shorter effective responses than ESPO in all six configurations while achieving higher pass@1. Therefore, these comparisons demonstrate stronger coding performance against ESPO with more compact responses, together with a performance advantage over SPG in five of the six evaluated configurations.

We would also like to clarify how to interpret comparisons with published results. Headline scores reflect the complete training and evaluation pipeline, including the initialization, training data, batching and rollout budgets, generation configuration, and evaluation protocol. Accounting for these settings is necessary to attribute differences to the optimization method. Our additional comparisons evaluate PBPO, SPG, and ESPO within a unified setting and provide direct evidence that PBPO’s performance gains extend to the two additional baselines evaluated here.

We will incorporate these comparisons and the expanded discussion of the cited methods into the revised manuscript.

AuthorsOpenReview ↗

References for additional related works

References

[1] Wang et al. d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation.

[2] Wang et al. SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models.

[3] Wang et al. Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models.

[4] Zhu et al. DiRL: An Efficient Post-Training Framework for Diffusion Language Models.

[5] Ni et al. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models.

[6] Hu et al. LightningRL: Breaking the Accuracy–Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learning.

[7] Zheng et al. Group Sequence Policy Optimization.

[8] Ou et al. Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective.

[9] Zhong et al. Stabilizing Reinforcement Learning for Diffusion Language Models.

Reviewer_2UbtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of computing the importance sampling ratio that arises when performing PPO-style reinforcement learning on diffusion LLMs (dLLMs). Considering that recent dLLM works adopt block-wise decoding, it proposes the idea of performing policy optimization at the level of prefix-block states. However, since block-level importance weights have high variance, it proposes the PBPO objective, which decouples the two: importance weights are computed at the token level, while conditioning remains at the block level. During training, it uses a condition masking technique that enables updates with diverse conditions from a single rollout. Experiments with LLaDA-8B-Instruct and LLaDA-1.5 backbones on six benchmarks show that PBPO achieves comparable or better performance than other baselines while producing shorter responses.

Critical strengths

S1. Reasonable methodology and very strong experimental results.

The proposed design, with block-wise conditioning and token-wise importance weights, is a reasonable choice that is computationally feasible and seems likely to work well empirically.

The experimental results are also very strong. Although the abstract modestly describes the performance as competitive, PBPO achieves the best performance on every benchmark and sequence length except Countdown (Table 1). The effective sequence length results (Table 5) are also impressive. The experiments also deserve credit for their diversity: the method is tested on two backbone models, compared against four prior methods and evaluated extensively across three sequence lengths and six benchmarks. That said, some aspects of these results may not be fully reliable (W2).

Critical weaknesses

W1. Limited novelty compared to d1.

The advantage computation and normalization, the token-level clipped ratio, and prompt masking are all identical to d1. The differences appear to be limited to (1) computing the likelihood by conditioning on the blocks up to the previous one, and (2) prefix masking, the latter of which did not yield consistent performance improvements.

W2. Unclear reliability of the experimental results.

d1 uses a KL penalty, whereas this paper does not. It is unclear exactly which settings were used to run d1, and the source of the performance gap needs to be disentangled. GDPO uses a sequence-level likelihood ratio, whose variance is expected to be large, following this paper’s argument (Section 3.4). Consistent with this, the GDPO authors note that GDPO requires a smaller learning rate than d1 and may otherwise diverge (GDPO, Appendix D.1). However, it appears that this paper uses a learning rate of 3e-6 across all experiments (Appendix A.2). I strongly suspect this is why GDPO exhibited divergent behavior during training.

Overall

The objective does not appear substantially different from d1, and its design seems to stem from empirical observations rather than theoretical derivation. The experimental results therefore seem crucial to justify the method, but their reliability is unclear. Hence, I recommend weak reject.

Questions for authors

Q1. (Clarification) The PBPO objective does not include a KL penalty, while some baselines such as d1 originally use one. How were these baselines configured in your experiments? Were they trained with or without the KL term? If they were trained with it, additional results for these baselines trained without the KL term would help isolate the effect of the proposed objective.

Q2. (Clarification) Was GDPO also trained with a learning rate of 3e-6, or with the lower learning rates used in the original GDPO paper? If the former, this may be the cause of the observed divergence, and I would like to see results under the original settings.

Q3. (Clarification) What does |o_i| denote in Eq. 3 and Eq. 7? In my reading, the generation length seems natural for Eq. 3, whereas the effective sequence length seems natural for Eq. 7. I would appreciate a clarification on which length is used in each.

5 comments
AuthorsOpenReview ↗

Response from Authors - Part 1/3

Dear Reviewer 2Ubt,

We thank you for recognizing the methodological rationale of PBPO and the strength and breadth of our experimental evaluation. We address your concerns and questions point by point below.

W1. Limited novelty compared to d1

PBPO develops a policy-optimization formulation that explicitly accounts for the conditioning structure of block-wise diffusion generation. Its contributions concern the likelihood surrogate, the granularity of importance weighting, and the resulting optimization behavior.

PBPO’s estimator differs from d1’s in its conditioning structure. Each block is evaluated from a state that reveals the preceding completed blocks while masking the current and future blocks. Different blocks therefore use different conditioning states, preserving dependencies across completed blocks, while tokens within the same block share a state and are predicted in parallel. This changes the conditional distributions entering the objective, the probabilities used to construct the surrogate ratios, and the resulting gradients.

Sections 3.1–3.3 first construct a tractable block-likelihood surrogate and its corresponding block-level objective. The resulting block ratio is a product of token ratios, so a change at one position propagates into the weight applied to every token in the block. PBPO retains the same prefix-block conditioning while assigning importance weights and clipping operations separately to individual tokens. Thus, the formulation preserves dependencies across completed blocks while avoiding a shared multiplicative weight for all token updates.

Theorem 1 and Appendix C characterize the variance amplification caused by block-product weighting for the surrogate gradient contributions studied in the analysis. Because the two formulations share prefix-block conditioning, the comparison isolates importance-weighting granularity. Figure 1 complements this analysis with a direct comparison under identical training configurations and demonstrates substantially more stable optimization with PBPO.

We would like to clarify that prompt masking follows d1, as explicitly credited in Section 3.3.1, and is not claimed as a novelty of PBPO.

Additional prefix masking diversifies the revealed response context and is task-dependent. Table 2 shows improvements over prompt-only masking on Countdown of 9.76, 11.33, and 6.64 percentage points at generation lengths 128, 256, and 512, respectively. On MATH500, prompt-only masking is preferable, and the main configuration accordingly uses a clean prefix. Therefore, additional prefix masking is optional; prefix-block conditioning and the separation of conditioning from importance weighting define the core formulation.

The empirical evaluation spans two initializations, six benchmarks, three generation lengths, and additional decoding configurations. PBPO leads or ties the strongest baseline in 32 of the 36 main comparisons.

W2. Unclear reliability of the experimental results

For d1, we directly used the implementation released in the authors’ GitHub repository and adopted the parameters reported in their paper, including its KL regularization.

The original GDPO experiments used the shared learning rate of 3e-6. We additionally trained GDPO using the lower learning rate specified in its original paper and report the complete comparison in our response to Q2.

Notably, as stated in Section 4.1, the original GDPO results were evaluated using the last saved checkpoint immediately before training collapse. Thus, the reported GDPO results reflect performance before the observed collapse, while PBPO was evaluated at its final prescribed checkpoint.

AuthorsOpenReview ↗

Response from Authors - Part 2/3

Q1. How was d1 configured, and does the improvement persist without its KL penalty?

Our original d1 baseline uses the authors’ released implementation and the parameters reported in their paper, including the KL penalty.

We provide additional experimental results for d1 without KL regularization, denoted d1 w/o KL in Tables R1a–R1b. This variant uses the same released d1 implementation with its KL term disabled. The tables compare this variant with PBPO and the original d1 baseline.

All additional runs use LLaDA-8B-Instruct, with the coding model evaluated on both HumanEval and MBPP. The optimizer, batching, rollout generation, and evaluation follow the unified settings described in Section 4.1 and Appendix A.2. The PBPO and original d1 entries reproduce Tables 1, 5, and 6.

Tables R1a–R1b. PBPO, original d1, and d1 without KL. Each entry reports accuracy (%) / average effective length (tokens). For HumanEval and MBPP, accuracy denotes execution-based pass@1. Column headers specify the benchmark and maximum generation length; the highest accuracy in each column is bolded.

Table R1a. Mathematical reasoning and coding (HumanEval).

MethodGSM8KMATH500HumanEval
Generation length128256512128256512128256512
PBPO76.72 / 113.7884.08 / 221.0884.38 / 281.7034.80 / 121.6236.80 / 226.6141.40 / 359.1030.49 / 95.8137.20 / 117.8637.80 / 127.59
d172.55 / 115.8578.01 / 217.1381.20 / 275.1529.00 / 122.0534.40 / 234.4737.60 / 392.7529.88 / 119.1635.37 / 194.9337.80 / 246.77
d1 w/o KL73.84 / 105.7879.98 / 123.6079.08 / 123.3526.20 / 85.4230.40 / 197.8235.60 / 353.5623.17 / 87.5928.66 / 104.5723.78 / 113.90

Table R1b. Coding (MBPP) and planning.

MethodMBPPCountdownSudoku
Generation length128256512128256512128256512
PBPO43.97 / 84.7248.25 / 93.3547.47 / 97.5352.73 / 103.3764.06 / 134.6164.45 / 159.1634.93 / 102.9438.13 / 164.9838.13 / 244.54
d141.25 / 114.7242.41 / 189.0141.63 / 241.5330.08 / 107.4923.44 / 209.7530.08 / 397.2021.29 / 115.5414.98 / 225.429.76 / 435.74
d1 w/o KL38.13 / 80.1139.69 / 89.9536.96 / 94.6346.48 / 80.6853.52 / 79.3255.08 / 80.8320.81 / 31.3920.14 / 136.0018.56 / 229.16

PBPO outperforms d1 w/o KL in all 18 accuracy comparisons across six benchmarks and three generation lengths. Specifically, PBPO exceeds d1 w/o KL by 5.80–8.60 percentage points on MATH500, 6.25–10.54 on Countdown, and 14.12–19.57 on Sudoku. PBPO also achieves higher accuracy than d1 w/o KL on GSM8K, HumanEval, and MBPP at every tested generation length.

These results show that PBPO’s performance advantage persists against d1 with its KL term disabled, strengthening the empirical support across mathematical reasoning, planning, and coding.

AuthorsOpenReview ↗

Response from Authors - Part 3/3

Q2. Was GDPO trained with the shared learning rate, and what happens with its published lower learning rate?

The original GDPO experiments used the shared learning rate of 3e-6.

Notably, as stated in Section 4.1, GDPO was evaluated using the last saved checkpoint immediately before training collapse. The GDPO entries in the main table therefore represent performance before the observed collapse, while PBPO was evaluated at its final prescribed checkpoint.

We provide additional experimental results for GDPO trained with the lower learning rate specified in the original GDPO paper, denoted GDPO lower LR in Tables R2a–R2b. The tables compare this variant with PBPO and the original GDPO baseline.

The additional GDPO comparison uses LLaDA-8B-Instruct. The remaining optimizer, batching, rollout-generation, and evaluation settings follow Section 4.1 and Appendix A.2. The PBPO and original GDPO entries reproduce Tables 1, 5, and 6.

Tables R2a–R2b. PBPO, original GDPO, and GDPO with its published lower learning rate. Each entry reports accuracy (%) / average effective length (tokens). The highest accuracy in each column is bolded.

Table R2a. Mathematical reasoning and coding (HumanEval).

MethodGSM8KMATH500HumanEval
Generation length128256512128256512128256512
PBPO76.72 / 113.7884.08 / 221.0884.38 / 281.7034.80 / 121.6236.80 / 226.6141.40 / 359.1030.49 / 95.8137.20 / 117.8637.80 / 127.59
GDPO75.66 / 109.4679.91 / 192.2080.59 / 204.8531.40 / 120.4534.80 / 222.8137.40 / 367.3623.17 / 116.8737.20 / 223.5436.59 / 393.85
GDPO lower LR71.19 / 113.8979.98 / 210.6581.96 / 250.3330.00 / 121.5835.00 / 234.1937.40 / 391.1826.83 / 120.1033.54 / 213.2537.80 / 281.17

Table R2b. Coding (MBPP) and planning.

MethodMBPPCountdownSudoku
Generation length128256512128256512128256512
PBPO43.97 / 84.7248.25 / 93.3547.47 / 97.5352.73 / 103.3764.06 / 134.6164.45 / 159.1634.93 / 102.9438.13 / 164.9838.13 / 244.54
GDPO42.41 / 115.9341.25 / 209.9442.80 / 330.0862.50 / 114.2573.05 / 213.2773.05 / 351.7928.09 / 114.1728.09 / 213.9226.12 / 355.32
GDPO lower LR36.58 / 118.5045.53 / 205.5743.19 / 264.3933.20 / 109.5530.47 / 213.8750.78 / 390.5523.40 / 116.1016.12 / 222.3713.49 / 448.90

PBPO achieves higher accuracy than GDPO lower LR in 17 of the 18 comparisons and ties with GDPO lower LR on HumanEval at generation length 512. Thus, PBPO’s performance advantage persists when GDPO uses its published lower learning rate.

PBPO also produces shorter effective responses than GDPO lower LR on Countdown, Sudoku, HumanEval, and MBPP at all three generation lengths, while achieving higher or equal accuracy than GDPO lower LR. On these four benchmarks, the comparison therefore demonstrates stronger or equal accuracy with more compact responses.

Q3. What does the response-length notation denote in Equations 3 and 7?

In Equation 3, the length denotes the full generated response-tensor length. This length determines the masked suffix of each prefix-block state.

In Equation 7, the token summation and normalization use the effective response length. All response positions up to and including the first EOS token contribute to the loss, and every position after the first EOS is excluded. We will distinguish the two lengths explicitly:

\qquad L_i := \text{number of response tokens up to and including the first EOS}.$$ If no EOS token is generated, all generated response positions contribute to the loss, so the effective response length equals the full generated response length. We will specify this convention alongside the distinct length notation in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.