- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Prefix-Block Policy Optimization (PBPO), a RLVR algorithm designed for masked DLLMs that generate text via block-wise decoding. The authors uses a prefix-block surrogate and uses decoupled clipping since the exact conditional likelihood of a block is intractable for direct policy gradient. The results show that PBPO outperforms recent DLLM RL baselines on reasoning benchmarks.
Critical strengths
- The algorithm is easy to follow and clean. Decoupling the block-level conditioning state from token-level importance ration clipping is intuitive and practical. PBPO evaluates all token probabilities in one block with a single forward pass and prevent outlier token ratios by multiplying together and destabilizing the entire block update. The empirical results shows improvements over multiple RL baselines.
Critical weaknesses
- During rollout with block length 32, the model takes 16 denoising steps per block. But the training surrogate evaluates all 32 tokens conditioned only on the entire block is fully masked. So the policy is never optimized on intermediate intra-block states, which is a mismatch between rollout generation and policy optimization
- Per-update masking of the revealed prefix is presented as a core contribution of PBPO over d1's prompt masking method. But the Table 2 show that adding prefix masking on MATH500 hurts the accuracy compared to prompt-only masking. So it is unclear how much improvement over d1 comes from prefix block conditioning.
Questions for authors
Please check the weakness above.
4 comments
Response from Authors - Part 2/2
W2. Contribution of prefix-block conditioning versus additional prefix masking
We would like to clarify that prefix-block conditioning and additional prefix masking serve different roles. Our prefix-block state reveals the completed prefix and masks the current and future blocks; additional prefix masking optionally corrupts part of that revealed prefix. In Table 2, we retain prefix-block conditioning in every configuration and vary only conditioning corruption. Therefore, the MATH500 comparison between prompt-only and prompt-plus-prefix masking measures the effect of additional prefix corruption.
Our existing results already show improvements over d1 with a clean revealed prefix. Specifically, the prompt-only PBPO variant in Table 2 outperforms the corresponding d1 results in Table 1 on both MATH500 and Countdown at all three generation lengths, using the same LLaDA-1.5 initialization. We also use prompt masking with a clean revealed prefix in the main MATH500 configuration, as specified in Appendix A.2.
We provide an additional controlled PBPO–d1 comparison on GSM8K and MATH500 using LLaDA-8B-Instruct (Table R1). In these runs, we use prompt masking only, no additional prefix masking, symmetric clipping coefficients of 0.5/0.5 as in d1, and prompt-masking probability 0.15. Notably, we already use the same prompt-masking probability as d1 in all experiments reported in the paper.
For both PBPO and d1, we use the unified training settings described in Section 4.1 and Appendix A.2, including the optimizer, learning rate, batching, and rollout generation, in the original and controlled experiments. Under these aligned settings, the primary methodological difference is the likelihood estimator: our estimator evaluates token probabilities from the corresponding prefix-block state, while d1 uses its one-step mean-field estimator.
Table R1. Controlled PBPO–d1 comparison on LLaDA-8B-Instruct. Each entry reports accuracy (%) / average effective length (tokens). The d1 row reproduces the main-paper results. Evaluation follows the unified protocol in Section 4.1. The higher accuracy in each column is bolded.
| Method | GSM8K | MATH500 | ||||
|---|---|---|---|---|---|---|
| Generation length | 128 | 256 | 512 | 128 | 256 | 512 |
| PBPO (controlled) | 76.72 / 113.78 | 84.08 / 221.08 | 84.38 / 281.70 | 32.60 / 111.00 | 37.40 / 217.26 | 38.00 / 362.63 |
| d1 | 72.55 / 115.85 | 78.01 / 217.13 | 81.20 / 275.15 | 29.00 / 122.05 | 34.40 / 234.47 | 37.60 / 392.75 |
We observe that PBPO outperforms d1 in all six accuracy comparisons, with gains of 3.18–6.07 percentage points on GSM8K and 0.40–3.60 on MATH500. Thus, these results support the effectiveness of prefix-block conditioning.
On MATH500, we also observe shorter effective responses from PBPO than from d1 at every tested generation length, pairing higher accuracy with more compact responses.
Thank you for the detailed rebuttal and additional experiments.
I still have some concerns regarding the novelty relative to related methods and the lack of a theoretical justification of the surrogate approximation.
I will keep my original score unchanged, but my opinion is more on the borderline.
Follow-up response from Authors
Dear Reviewer JJsA,
We thank you for your continued consideration of our rebuttal and the additional experiments. We would like to clarify the two remaining concerns below.
Novelty relative to related methods
We position PBPO’s contribution in its block-boundary policy-optimization formulation and the analysis of its importance-weighting structure. Different response blocks are evaluated from different revealed prefixes, while all tokens within a block share the same conditioning state. We derive the corresponding block-product objective and then retain its conditioning structure while assigning surrogate importance weights and clipping operations separately to individual tokens. Thus, PBPO explicitly separates the granularity of conditioning from the granularity of importance weighting.
We have provided a detailed comparison with d1 in our response to Reviewer 2Ubt (W1), and broader comparisons with d2-StepMerge, SPG, and other related methods in our response to Reviewer ivyd (W1). These comparisons distinguish the scoring states, approximation targets, and update rules, while explicitly identifying shared components. For example, SPG estimates evidence bounds from sampled, partially masked active blocks, whereas PBPO scores every block at its fully masked boundary and uses individually clipped current-to-behavior ratios. We also discuss the relationship between d2-StepMerge’s temporal segments and PBPO’s block-boundary states. We invite you to consult these responses for the fuller methodological comparison. In the revised manuscript, we will expand the related-work discussion and clarify the attribution and scope of our contributions.
The additional experiments support the practical value of this formulation under aligned settings. As reported in our initial response to you, PBPO outperforms d1 in all six controlled accuracy comparisons on GSM8K and MATH500. Our response to Reviewer ivyd (W4) additionally reports coding results in which PBPO outperforms ESPO in all six comparisons and SPG in five of six, including all three HumanEval generation lengths. These results complement the methodological discussion with comparisons against additional related objectives.
Theoretical motivation and scope of the surrogate analysis
For the surrogate construction, we start from the completed-block marginal likelihood, which integrates over the latent intra-block denoising trajectories. We choose the block-boundary state because it is the context present immediately before that block is generated: preceding blocks are revealed, while the active and future blocks are masked. The factorized one-step predictions provide a tractable block surrogate that preserves dependencies across completed blocks and allows all positions within a block to be scored in one forward pass. Therefore, the approximation is grounded in the conditioning structure of the rollout policy.
For the importance-weighting choice, Theorem 1 holds this conditioning structure and the one-step surrogate fixed, and isolates the variance amplification caused by multiplying token ratios into a shared block weight. Under the assumptions stated in Appendix C, the block-product contribution has greater total variance than the token-level contribution. The same comparison is preserved when both contributions are multiplied by a fixed nonzero advantage.
We discuss the theoretical scope more fully in our response to Reviewer ivyd (W2). The theorem characterizes gradient variance within the specified surrogate probability space; the approximation error of the true multi-step marginal is a separate quantity. The practical utility of the surrogate and the complete clipped objective is assessed through our controlled comparisons and downstream evaluations. We invite you to consult that response for the detailed assumptions and the respective roles of the theoretical analysis and empirical evidence. We will make these distinctions more explicit in Sections 3.1–3.4 of the revised manuscript.
Response from Authors - Part 1/2
Dear Reviewer JJsA,
We thank you for recognizing the clarity and practicality of PBPO and its empirical improvements over existing dLLM RL baselines. We address your concerns and questions point by point below.
W1. Relationship between multi-step rollout generation and the training surrogate
We would like to clarify that PBPO’s one-step predictions provide a tractable surrogate for the likelihood of multi-step block-wise generation. The exact likelihood of both a completed block and the full response is intractable because it requires marginalizing over the intermediate denoising states and token-reveal orders that can produce the same output. In our formulation, a completed block is treated as a block-level action whose internal denoising trajectory is latent. Its conditional likelihood takes the form
πθ(oi(b)∣q,oi(<b))=∑τb∈T(oi(b))pθ(oi(b),τb∣q,oi(<b))
where the sum ranges over compatible intra-block trajectories.
Under this block-level formulation, policy optimization can be expressed through the marginal likelihood of completed-block actions; a separate objective at every intermediate state is not required. We approximate the intractable marginal using factorized one-step predictions from the prefix-block state, allowing all token probabilities in a block to be evaluated with one forward pass. This base state uses the context present at an actual rollout boundary: preceding blocks are revealed, while the current and future blocks remain masked. Thus, our surrogate retains dependencies across completed blocks while making their optimization computationally tractable.
This likelihood challenge also motivates prior dLLM RL methods, which use tractable surrogates or reformulated objectives. For example, d1 uses one-step token predictions under a mean-field approximation, wd1 optimizes reward-weighted token likelihoods, and GDPO constructs a sequence-level ratio from a diffusion lower bound. Approximating the distribution induced by multi-step denoising is therefore a central challenge of dLLM RL. Our contribution is to ground the surrogate in block-boundary conditioning and separate this conditioning structure from token-level importance weighting, with theoretical analysis and empirical evidence supporting the resulting optimization behavior.
We chose block length 32 for the main experiments to follow the multi-step, block-wise rollout procedure used by d1 and wd1 and maintain an aligned comparison, as described in Section 4.1 and Appendix A.2. Since changing the block length or other rollout-generation settings can affect rollout quality and, consequently, downstream performance, we keep these settings aligned with prior work. Therefore, the comparison more clearly distinguishes gains associated with the optimization objectives and likelihood estimators from changes in rollout generation.