acceptodds
Under review as a conference paper at ICLR 2027

Every Step Counts in RL for Diffusion Language Models

Abstract

Diffusion language models are a promising and more efficient alternative to au- toregressive models. However, most RL post-training techniques are not easily adapted to these models. One issue is that diffusion language models generate answers through many intermediate states, deciding at each step which masked positions to reveal and which tokens to place in them. Many existing RL methods used in this setting approximate intractable sequence likelihoods or weight every denoising step by the same advantage, without explicitly distinguishing how these intermediate states of the denoising trajectory affect the expected reward. To ad- dress this issue of trajectory blindness, we introduce PACT (Potential-Anchored Consistency of Transitions). Motivated by a reward-tilted reference distribution, PACT jointly learns the policy and a scalar potential that scores each intermedi- ate state, anchored by terminal reward. For each sampled denoising transition, a consistency loss matches the policy-to-reference log-probability ratio of revealed tokens to the change in this potential. This formulation requires neither sequence- likelihood estimation nor importance-ratio clipping. On four tasks, PACT improves LLaDA-8B-Instruct by +9.9 points on GSM8K, +12.0 on MATH500, +55.7 on Countdown, and +72.6 on Sudoku, and reaches the best average accuracy across the four tasks reported for this model, +3.7 points over the next best method. Further compared to AGRPO, it reaches that baseline’s best accuracy in about half the training compute on GSM8K.

Primary area:
reinforcement learning
Keywords:
diffusion language models, reinforcement learning, reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DVnfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Potential-Anchored Consistency of Transitions (PACT), to address the issue of trajectory blindness. The method don't need any sequence-likelihood estimate and importance-ratio clipping. PACT jointly learns the policy and a scalar potential that scores each intermedia state, anchored by terminal reward. The results show large improvement over math and reasoning datasets.

Critical strengths
  1. Method is simple and straightforward. PACT is likelihood-free and only uses the exact token conditionals of the transitions the sampler actually took. So it avoids the ELBO and masked-reconstruction surrogates and ratio clipping. And results show improvement.
Critical weaknesses
  1. For the experiments, there is only best of n test set with single seed. Also AGRPO trains at length 384 with blocks of 12, but PACT trains at 256/32 and also test is under 256/32. So the baseline has train-eval mismatch which may lead to the lower accuracy for baseline and also gap between the original paper.
  2. The paper claims that the per-transition credit assignment drives the accuracy gains, but there is only ablation for sweeping hyper parameters. And it didn't quite show too much information. I think some experiment like dropping the consistency term or do analysis on potential term is useful.
Questions for authors

Please check the weakness above.

4 comments

Thank you for the positive assessment and for the concrete suggestions. We followed them, and the additions are marked in blue in the revision: the experiment you suggest, dropping the consistency term, together with two related controls (Section 5.2), and final-checkpoint results (Appendix E). We respond to each point below.

1. W1a. Providing final checkpoint selection and stable performance on multiple seeds.

You are right that best-of-n selection on the test set needed a check, so we added results that involve no selection.

  • Selection rule. Like prior work (AGRPO, d1, SPG, RSPO), we report each run's best checkpoint, evaluated every 25 rounds, for PACT and every baseline we reproduce (Section 4.3). Because this still inflates every method slightly, we now also report the final checkpoint (round 900), which involves no selection, for PACT and for the three baselines we trained ourselves:
GSM8K, length 256wd1*AGRPO*DMPO*PACT
Best checkpoint (Table 1)84.485.185.289.1
Final checkpoint (round 900)82.583.583.288.7

*Our reproduction with the authors' code, 900 rollout rounds, decoded with the paper's evaluation protocol (as in Table 1 of the paper).

PACT loses 0.4 points between its best and its final checkpoint and remains 5.2 points above the strongest reproduced baseline at the final checkpoint (AGRPO, 83.5). The comparison therefore does not depend on checkpoint selection. These results are in the new Appendix E (in blue).

  • Seeds. We ran PACT on GSM8K for two additional seeds (334, and 335), original being 333. We can see that the performance is largely stable and around the same ballpark of 89.1 as reported in the paper.
seedbestfinal
333 (paper)89.188.7
33489.989.2
33588.687.4

2. W1b. Clarifying AGRPO's train/eval mismatch

You are right that AGRPO is trained and evaluated under different settings. We explain below how we ran it and where the mismatch comes from.

  • Code, data and settings. We ran the authors' released repository at its latest commit (458f139), with its released hyper-parameters, prompts, data and splits, rewards and answer parser:
    • training: the released train.yaml (384 tokens in blocks of 12, random unmasking, LoRA 64, lr 5e-6, G = 8, 24 Monte-Carlo samples);
    • evaluation: the released eval.yaml (256 tokens in blocks of 32, confidence order).
  • The only change is the step budget. The released config sets num_train_epochs: 5 and no step count. We set max_steps = 900 so that every method trains on the same rollout budget (900 rounds of 128 completions), which keeps the accuracy and compute comparisons on equal terms.
  • The train/eval mismatch is part of AGRPO's own design. The AGRPO paper trains at response length 384 with random unmasking and reports its results at lengths 256 and 512 with confidence-based unmasking; its authors note that the unmasking difference "introduces a train–test mismatch". Its published numbers therefore carry the same mismatch. We also evaluated our AGRPO runs at length 384, the length they are trained at: the accuracy is 83.3 on GSM8K and 47.3 on Countdown, still far below PACT's 89.1 and 79.9, so the length mismatch does not explain the gap.
  • The Countdown gap. Our 42.2 matches the 40.0 reported in the first version of the AGRPO paper. Our understanding is that the 79.7 in the current version likely needs a longer run or task-specific tuning (for example of the schedule and step count); neither is specified in the released code or the paper, and the released code has no Countdown configuration. With the official code and its released configuration at this budget, we could not reproduce 79.7.

3. W2. Ablations to show that per-transition credit drives the gains

We agree that hyper-parameter sweeps alone do not show this, so we ran the experiment you suggest. We changed one term of PACT's loss (Eq. 5) at a time, on GSM8K, with the same sampler, seed, prompts, rollout budget and learning rate:

variantconsistency residualintermediate anchorsaccuracyseconds / round
PACTper transition (K = 16)✓88.741
T1: path-level balanceonce per path (exact TB, all 128 steps)✓87.8138
T2: no anchorsper transition✗ (boundary r/β only)81.343
T3: no residual✗✓3.9 (collapsed)42
TraFL (Appendix B)once per completion (ELBO surrogate)✗ (free log Z)83.439

What each variant does:

  • T1: path-level balance (is locality needed?). One residual per path, summed over all 128 steps (exact relative trajectory balance), instead of PACT's 16 sampled per-transition residuals. It learns far more slowly (77.8 vs 85.9 at round 200) and takes 3.4× the time per round (138 s vs 41 s on the same 16 GPUs), because it needs a gradient pass through every denoising step.
  • T2: no anchors (are the anchors needed?). The potential is pinned to the reward only at the finished answer. Final accuracy drops 7.4 points (81.3 vs 88.7).
  • T3: no residual (is the residual needed?). Anchors only. The policy receives no credit signal and thus collapses.
  • TraFL (is per-completion balance enough?). One residual per completed answer, through an ELBO surrogate, with a learned normalizer. It peaks at 83.4 and then declines (Appendix B).

Together: the per-transition residual is what trains the policy (T3), the anchors make its credit accurate (T2), and imposing the condition per transition rather than per path or per completion is what makes it fast and stable (T1, TraFL). The per-transition residual on an anchored potential is therefore what makes the method work, and its locality is what makes it efficient. Each variant's loss and a longer discussion are in Section 5.2.

We hope these additions resolve your concerns, and we would be glad to clarify anything further.

I appreciate the author's response. I am keeping my score.

Thank you for taking the time to review our responses and updated experiments. We are glad that our clarifications and additional results have addressed your concerns, and we truly appreciate your positive feedback and for keeping the positive score.

Reviewer_tZWQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PACT trains diffusion language models by matching the policy-to-reference log-probability ratio of revealed tokens to changes in a learned state potential. Terminal rewards anchor the potential and provide regression targets for intermediate states. On LLaDA-8B-Instruct, it reports the best average accuracy among the listed methods across GSM8K, MATH500, Countdown, and Sudoku at two generation lengths, along with a GSM8K training-efficiency improvement over reproduced AGRPO.

Critical strengths
  • PACT offers a practical way to train on intermediate denoising decisions without estimating response-level marginal likelihoods. The potential head reuses backbone representations and is needed only during training, so it adds no inference overhead.

  • The reported gains are substantial: at length 256, average accuracy is 78.2%, versus 74.5% for the strongest other average in Table 1. Figure 4 measures actual training cost, including PACT's higher per-batch cost: it reaches reproduced AGRPO's best GSM8K accuracy in 28.6 versus 53.4 GPU-hours and also performs better at matched compute.

Critical weaknesses
  • The experiments do not isolate the claimed benefit of transition-level credit assignment. Table 2 tests parameter choices but not local versus aggregated consistency or the contribution of reward anchors. TraFL also differs in its likelihood estimator, normalizer inputs and gradient path, and optimization settings; Appendix B acknowledges that this compares the objectives as a whole. The results therefore do not establish whether local consistency, auxiliary reward prediction, or another change explains the gains.

  • The main comparisons need stronger validation. Section 4.3 selects the best test accuracy from roughly 36 checkpoints per run. Applying this rule to both PACT and reproduced baselines does not remove selection bias; binomial errors and rolling training deviations do not capture checkpoint-selection or training-seed uncertainty. AGRPO also uses different LoRA rank, rollout geometry, order sampling, and learning rate (Appendix A), and its Countdown reproduction reaches 42.2% versus the cited 79.7%, with the gap unresolved. These disclosed limitations make the ranking and efficiency claims harder to assess; the published AGRPO scores cannot simply be substituted across evaluation protocols.

Questions for authors
  • How do local and aggregated consistency compare with the sampler, likelihood calculation, trainable components, gradient paths, and budget held comparable? What is the contribution of the intermediate reward anchors? Please specify the aggregation estimator and scaling.

  • Do the gains remain with a fixed final checkpoint or independent validation selection? Please report any available across-seed uncertainty and paired per-example results.

  • What explains the AGRPO reproduction gap? Please provide the code revision and data, prompt/parser, and training settings, and clarify how these differences affect the accuracy and compute comparisons.

  • Does Eq. (5) score raw softmax probabilities or the temperature/top-p/top-k rollout distributions in Table 3? Also, how does the token-only interpretation in Section 5.1 account for the order-conditioned normalizer in the ideal tilted token conditional?

5 comments

Thank you for a precise and detailed review. We have incorporated your comments into the paper, all marked in blue in the revision: the controlled ablations you asked for (Section 5.2), final-checkpoint results (Appendix E), a statement of which probabilities Eq. 5 scores (Section 3), and a revised account of the order normalizer (Section 5.1, Appendix C). We respond to each point below.

1. W1 / Q1 Ablations for isolating transition-level credit assignment

You are right that the submitted experiments did not isolate transition-level credit assignment, so we ran the controls you describe. We changed one term of PACT's loss (Eq. 5) at a time and kept everything else identical to PACT: the sampler and its order (256 tokens, blocks of 32, 128 steps, confidence order at τ = 0.2), the likelihood calculation (exact token conditionals of the steps the sampler took, Eq. 1, with no ELBO surrogate), the trainable components (the same LoRA adapters and potential head), the gradient path, the budget (900 rounds of 128 completions), the seed and therefore the prompts, and the learning rate. The new Section 5.2 (in blue) gives each variant's loss next to its result.

variantconsistency residualintermediate anchorsaccuracyseconds / round
PACTper transition (K = 16)✓88.741
T1: path-level balanceonce per path (exact TB, all 128 steps)✓87.8138
T2: no anchorsper transition✗ (boundary r/β only)81.343
T3: no residual✗✓3.9 (collapsed)42
TraFL (Appendix B)once per completion (ELBO surrogate)✗ (free log Z)83.439
  • Local versus aggregated (T1). The aggregated residual is the sum of PACT's per-transition residuals over all T=128T = 128 steps of the path, δpath=∑tδt\delta_{\mathrm{path}} = \sum_t \delta_t, computed exactly with no sub-sampling. This is relative trajectory balance with the potential of the masked start as the log-normalizer (the token-only form of Eq. 6). The loss is δpath2/T\delta_{\mathrm{path}}^2/T, which matches PACT's gradient scale at initialization, plus the same anchors averaged over the steps. T1 approaches PACT's accuracy, but learns far more slowly (74.2 vs 80.9 at round 100, 77.8 vs 85.9 at round 200) and costs 3.4× more per round (138 s vs 41 s on the same 16 GPUs), because it back-propagates through all 128 steps of every path while PACT scores 16 sampled steps. To reach about 86 points, PACT needs roughly 200 rounds and T1 roughly 400, i.e. about a sixth of the compute.
  • The intermediate anchors (T2). Without anchors, the reward still enters through the fixed potential of the finished answer, fϕ(x0)=r/βf_\phi(x_0) = r/\beta, but only in the residual of the final step. The potentials of intermediate states then learn the reward only by chaining residuals back from that step, and the final accuracy drops by 7.4 points.
  • Auxiliary reward prediction (T3). With the anchors alone, the potential head still learns to predict the reward, but the policy's step probabilities no longer appear in the loss, so the policy receives no credit signal. Accuracy falls from 79 to 3.9 by round 725 (we stopped the run at round 750). Thus reward prediction alone does not explain the gains.

Why per-transition works better than per-path. Both losses have the same minimizer: if δpath=0\delta_{\mathrm{path}} = 0 on every path, every transition is consistent. On a finite batch, however, they send the error to different places. T1 sees only the sum, for example, a step that overshoots by +2+2 and another that undershoots by −2-2 give δpath=0\delta_{\mathrm{path}} = 0 and neither error is corrected. When the sum is non-zero, every step receives the same correction, including the steps that were already right. PACT penalizes each δt\delta_t separately, so the correction goes only to the steps that are wrong. This is the paper's first contribution (listed in Section 1): the reward of a completed answer can hide inconsistencies between the denoising steps that produced it, because steps that raise and steps that lower the expected reward cancel, so each transition should be trained on its own change in expected reward. T1 tests exactly this claim, since it differs from PACT only in scoring the path as a whole.

TraFL. As you correctly noted, TraFL differs from PACT in several components at once (likelihood surrogate, normalizer inputs, optimization), so Appendix B compares the objectives as a whole. T1 is the controlled version of that comparison: it keeps everything of PACT and changes only per-transition to per-path consistency.

2. W2 / Q2. Providing final checkpoint selection and stable performance on multiple seeds.

We agree that applying the same selection rule to every method does not remove selection bias, so we added results that involve no selection.

  • Selection rule. Like prior work (AGRPO, d1, SPG, RSPO), we report each run's best checkpoint, evaluated every 25 rounds, for PACT and every baseline we reproduce (Section 4.3). Because this still inflates every method slightly, we now also report the final checkpoint (round 900), which involves no selection, for PACT and for the three baselines we trained ourselves:
GSM8K, length 256wd1*AGRPO*DMPO*PACT
Best checkpoint (Table 1)84.485.185.289.1
Final checkpoint (round 900)82.583.583.288.7

*Our reproduction with the authors' code, 900 rollout rounds, decoded with the paper's evaluation protocol (as in Table 1 of the paper).

PACT loses 0.4 points between its best and its final checkpoint and remains 5.2 points above the strongest reproduced baseline at the final checkpoint (AGRPO, 83.5). The comparison therefore does not depend on checkpoint selection. These results are in the new Appendix E (in blue).

  • Seeds. We ran PACT on GSM8K for two additional seeds (334, and 335), original being 333. We can see that the performance is largely stable and around the same ballpark of 89.1 as reported in the paper.
seedbestfinal
333 (paper)89.188.7
33489.989.2
33588.687.4

3. W2 (second half) / Q3. Clarifying the AGRPO reproduction

Thank you for asking for these details; you are right that the paper should make the reproduction easy to check. We give them below.

  • Code, data and settings. We ran the authors' released repository at its latest commit (458f139), with its released hyper-parameters, prompts, data and splits, rewards and answer parser:
    • training: the released train.yaml (384 tokens in blocks of 12, random unmasking, LoRA 64, lr 5e-6, G = 8, 24 Monte-Carlo samples);
    • evaluation: the released eval.yaml (256 tokens in blocks of 32, confidence order).
  • The only change is the step budget. The released config sets num_train_epochs: 5 and no step count. We set max_steps = 900 so that every method trains on the same rollout budget (900 rounds of 128 completions), which keeps the accuracy and compute comparisons on equal terms.
  • The train/eval mismatch is part of AGRPO's own design. The AGRPO paper trains at response length 384 with random unmasking and reports its results at lengths 256 and 512 with confidence-based unmasking; its authors note that the unmasking difference "introduces a train–test mismatch". Its published numbers therefore carry the same mismatch. We also evaluated our AGRPO runs at length 384, the length they are trained at: the accuracy is 83.3 on GSM8K and 47.3 on Countdown, still far below PACT's 89.1 and 79.9, so the length mismatch does not explain the gap.
  • The Countdown gap. Our 42.2 matches the 40.0 reported in the first version of the AGRPO paper. Our understanding is that the 79.7 in the current version likely needs a longer run or task-specific tuning (for example of the schedule and step count); neither is specified in the released code or the paper, and the released code has no Countdown configuration. With the official code and its released configuration at this budget, we could not reproduce 79.7.

As you point out, published AGRPO scores cannot be substituted across protocols, and Table 1 does not do so: its AGRPO row is our run under our protocol.

4. Q4a. Clarifying about which probabilities Eq. 5 scores

Thank you for this question; the submission did not say this explicitly. Eq. 5 scores the raw softmax at temperature 1, without truncation, for both the policy and the reference. The temperature, top-p and top-k settings (Table 3 of the submission, Table 4 of the revision) only shape the rollout, i.e. which transitions are visited. Because the residual is a per-transition condition, it can be imposed on transitions from any behaviour distribution, and tokens outside the truncated support are simply never scored. Section 3 now states this.

5. Q4b. Clarifying about the order-conditioned normalizer

Thank you for this observation. The order-conditioned normalizer does matter here, and following your comment we revised Section 5.1 and added Appendix C (in blue) to account for it.

A step of the sampler first draws an order (which slots to reveal) and then the tokens for those slots. Under the tilted target, the token distribution for a given order is the reference's, reweighted by the potential of the next state and renormalized by that order's own normalizer Zω(xt)Z_\omega(x_t). With the order term switched off (γord=0\gamma_{\mathrm{ord}} = 0), the residual compares the token log-ratio with the change in potential, and the difference between Zω(xt)Z_\omega(x_t) and the shared potential h(xt)h(x_t) is left over as an offset of log⁡h(xt)−log⁡Zω(xt)\log h(x_t) - \log Z_\omega(x_t). This offset is exactly the order log-ratio that γord=0\gamma_{\mathrm{ord}} = 0 omits, with the sign reversed.

The offset is the same for every token choice under a given order, but it can differ from one order to another, and a single potential fϕ(xt)f_\phi(x_t) cannot cancel it for every order at once. Hence:

  • when the target does not prefer one order over another, all Zω(xt)Z_\omega(x_t) are equal to h(xt)h(x_t), the offsetvanishes, and the token-only residual is minimized exactly by the tilted token distribution;
  • when it does prefer some orders, the token-only loss reaches the tilted token distribution only approximately, and the potential absorbs part of the offset.

Section 5.1 now states exactly this ("the token-only residual then targets the token factor of the tilted kernel exactly when the target is indifferent to the order, and approximately otherwise"). Empirically the approximation costs little: adding the order term (γord=1\gamma_{\mathrm{ord}} = 1, two implementations) lowers accuracy (Table 2), and a near-random training order matches confidence order (88.06 vs 88.05).

We hope the new experiments and clarifications resolve your concerns, and we would be glad to clarify anything further.

Reviewer_bDXkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Potential-Anchored Consistency of Transitions (PACT), a post-training method for diffusion language models. PACT jointly trains a policy and a scalar potential head on intermediate denoising states. Experiments on LLaDA-8B-Instruct cover GSM8K, MATH500, Countdown, and 4×4 Sudoku. PACT achieves the highest average accuracy among the methods included in the comparison and reaches the reproduced AGRPO baseline's best GSM8K accuracy with approximately half the training compute.

Critical strengths
  • Promising empirical results, especially on GSM8K. At generation length 256, PACT improves GSM8K accuracy from 79.2% to 89.1%, compared with 86.1% for the strongest other entry in Table 1. It also improves substantially over the base model on the other three tasks and achieves the best four-task average at both evaluated lengths. The paper appropriately acknowledges that the margins over the strongest baselines on MATH500 and Sudoku are small.
Critical weaknesses
  • The methodological novelty relative to existing consistency and balance objectives needs sharper delineation. Section 2 identifies close connections to reference-relative detailed balance, one-step path consistency in KL-regularized RL, and methods that learn intermediate-state quantities. The main consistency equation is therefore not itself a new principle. The contribution appears to lie in applying this framework to a DLLM's sampled denoising transitions, together with the particular reward anchors and token-only implementation.
  • Evaluation choices limit confidence in the comparative claims. Section 4.3 reports the best test accuracy over checkpoints evaluated every 25 rollout batches. This introduces test-set selection bias. Applying the same procedure to reproduced baselines improves procedural consistency, but does not remove the bias, particularly when checkpoint variability differs across methods. Validation-based checkpoint selection followed by a single test evaluation would provide a cleaner comparison.
Questions for authors
  • Evaluation robustness: Can you provide validation-selected or final-checkpoint results, at least on GSM8K?
  • Important DLLM RL baselines are missing. The related-work discussion and Table 1 omit JustGRPO from Ni et al. (2026) [1] and Ou et al. (2026) [2], which are representative works. A fair comparison and discussion should be included.

[1] Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeguo Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, and Gao Huang. 2026. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models. In Proceedings of the 43rd International Conference on Machine Learning (ICML).

[2] Jingyang Ou, Jiaqi Han, Minkai Xu, Shaoxuan Xu, Jianwen Xie, Stefano Ermon, Yi Wu, and Chongxuan Li. 2026. Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective. International Conference on Learning Representations (ICLR).

4 comments

Thank you for your careful and constructive review, and for noting that we state the MATH500 and Sudoku margins honestly. We have incorporated your comments into the paper, all marked in blue in the revision: a sharper delineation from prior consistency and balance objectives (Appendix D), new ablations (Section 5.2), final-checkpoint results (Appendix E), and a comparison with the two works you pointed to (Section 2, Table 1). We respond to each point below.

1. W1 Addressing novelty relative to consistency and balance objectives.

Thank you for raising this; we have sharpened the delineation in the revision. Consistency and balance conditions are well established in generative modeling and reinforcement learning: detailed and trajectory balance in GFlowNets (Bengio et al., 2021; Malkin et al., 2022), path consistency in KL-regularized RL (Nachum et al., 2017; 2018), and trajectory balance for language-model post-training (TraFL, FlowRL and GFlowRL: Ahmadi et al., 2026; Zhu et al., 2025; Liu et al., 2026).

Our primary contribution is showing how these consistency conditions arise naturally and allow us to address a significant issue in diffusion language models (DLLMs) that leads to practical performance improvements. What PACT provides is a new way to use these conditions on the intermediate states of a diffusion language model. It imposes the condition on the transitions the sampler itself takes, where the revealed-token probabilities are exact, and anchors a learned potential to the reward at the states the rollout visits. This fits DLLMs naturally: it needs no sequence-likelihood estimate, no constructed intermediate states and no separate normalizer, and because the condition is local it can be imposed on a few sampled transitions per path. To our knowledge, no prior work applies a consistency condition to a DLLM in this way: TraFL and FlowRL impose balance once per completed response through an ELBO surrogate, and DTM and DDPP work on constructed states.

To support this with experiments, we ran new ablations that change one component of PACT at a time. Everything else is identical: sampler, 128 rollouts per round, seed and prompts, LoRA, learning rate, and GSM8K @256 evaluation. They show that the gains come from how PACT applies the condition, not from the condition itself.

variantconsistency residualintermediate anchorsaccuracyseconds / round
PACTper transition (K = 16)✓88.741
T1: path-level balanceonce per path (exact TB, all 128 steps)✓87.8138
T2: no anchorsper transition✗ (boundary r/β only)81.343
T3: no residual✗✓3.9 (collapsed)42
TraFL (Appendix B)once per completion (ELBO surrogate)✗ (free log Z)83.439

What each variant does:

  • T1: path-level balance (is locality needed?). One residual per path, summed over all 128 steps (exact relative trajectory balance), instead of PACT's 16 sampled per-transition residuals. It learns far more slowly (77.8 vs 85.9 at round 200) and takes 3.4× the time per round (138 s vs 41 s on the same 16 GPUs), because it needs a gradient pass through every denoising step.
  • T2: no anchors (are the anchors needed?). The potential is pinned to the reward only at the finished answer. Final accuracy drops 7.4 points (81.3 vs 88.7).
  • T3: no residual (is the residual needed?). Anchors only. The policy receives no credit signal and thus collapses.
  • TraFL (is per-completion balance enough?). One residual per completed answer, through an ELBO surrogate, with a learned normalizer. It peaks at 83.4 and then declines (Appendix B).

Together: the per-transition residual is what trains the policy (T3), the anchors make its credit accurate (T2), and imposing the condition per transition rather than per path or per completion is what makes it fast and stable (T1, TraFL).

Following your feedback, the revision (in blue) adds Appendix D, which states that the condition itself is established and delineates PACT from detailed balance, Trust-PCL, ∇-GFN, DDPP, DTM and TraFL in one table, and Section 5.2, which reports the ablations above.

2. W2/Q1. Providing final checkpoint results

This is a fair concern, and we have added the final-checkpoint results you ask for. For context, our selection rule follows prior work; we report each method's best checkpoint: we evaluate a checkpoint every 25 rounds and report the highest test accuracy among them, for PACT and for every baseline we reproduce (Section 4.3). This is the standard rule in this literature:

  • AGRPO reports "the best accuracy among" the checkpoints of its last 150 steps;
  • d1 evaluates "every 100 steps starting from step 600 and report[s] the best results";
  • SPG and RSPO report "the checkpoint that achieves the highest average test accuracy" across generation lengths;
  • wd1 lists "the training step of the best checkpoint" among its hyper-parameters, and MAPG evaluates every 100 steps.

We applied this rule to every method to keep the comparison fair. As you point out, however, choosing the best checkpoint on the test set still inflates every method's number slightly. We therefore also report final-checkpoint (round 900) results on GSM8K, which involve no selection at all, for PACT and for the three baselines we trained ourselves.

GSM8K, length 256wd1*AGRPO*DMPO*PACT
Best checkpoint (Table 1)84.485.185.289.1
Final checkpoint (round 900)82.583.583.288.7

*Our reproduction with the authors' code, 900 rollout rounds, decoded with the paper's evaluation protocol (as in Table 1 of the paper).

PACT loses 0.4 points between its best and its final checkpoint and remains 5.2 points above the strongest reproduced baseline at the final checkpoint (AGRPO, 83.5). The comparison therefore does not depend on checkpoint selection. These results are in the new Appendix E (in blue).

3. Q2. Comparison with JustGRPO and ESPO

Thank you for pointing to these two works. We added both to Section 2 (in blue). ESPO, which reports all four of our tasks, is now in Table 1 with the likelihood-based methods. JustGRPO follows a different training paradigm and does not report Countdown or Sudoku, so we discuss it in Section 2.

ESPO optimizes a sequence-level ELBO ratio with a PPO-style objective. Like the other likelihood-based methods, it weights the whole denoising trajectory by one completion-level advantage.

Results (LLaDA-8B-Instruct; generation length 256 / 512; ESPO's numbers are from its paper):

methodGSM8KMATH500CountdownSudoku4-task avg
ESPO82.3 / 83.739.0 / 43.482.0 / 79.384.7 / 80.572.0 / 71.7
PACT89.1 / 89.044.4 / 45.879.9 / 80.799.5 / 99.478.2 / 78.7

PACT improves over ESPO on the four-task average by 6.2 points at length 256 and 7.0 at length 512, and on every task and length except Countdown at length 256 (−2.1). ESPO trains its planning tasks for 10,000 steps (3,000 on the math tasks), against our 900 rounds.

JustGRPO is a different algorithm from PACT and from the methods in Table 1, in three respects:

  • Training paradigm. It abandons the diffusion sampler during RL: rollouts and training are done in a fixed left-to-right order, one token per step, so that the likelihood is exact. PACT trains the model under the sampler it is decoded with.
  • Decoding and cost. Its reported results decode one token per step (256 steps for 256 tokens), where PACT decodes two tokens per step, and they fine-tune all parameters of the 8B model, where PACT trains LoRA adapters. Its exact likelihood also needs a forward and backward pass for every token position, which its authors note adds per-iteration overhead.
  • Task coverage. It reports mathematics and code benchmarks, but not Countdown or Sudoku, so it cannot be placed in our four-task comparison.

Training JustGRPO ourselves on Countdown and Sudoku under our protocol is not feasible within the rebuttal period, for the cost reasons above. We therefore discuss it qualitatively above and in Section 2. With ESPO added, Table 1 compares PACT with eleven post-training methods (d1, wd1, SPG, UniGRPO, RSPO, MAPG, ESPO, AGRPO, DMPO, DAM and DTM) on four tasks at two generation lengths, and Appendix B adds a matched comparison with TraFL. To our knowledge, this is a broader comparison than in the papers of these methods themselves.

We hope these additions and clarifications resolve your concerns, and we would be glad to clarify anything further.

AuthorsOpenReview ↗

Rebuttal deadline approaching soon.

Dear Reviewer bDXk,

Thank you again for your careful review. As the discussion period closes in two days, we wanted to check whether our response and the revised paper address your concerns: the sharper delineation from prior consistency and balance objectives (Appendix D) with the new ablations (Section 5.2), the final-checkpoint results (Appendix E), and the comparison with ESPO and discussion of JustGRPO (Section 2, Table 1).

If anything remains unclear or unaddressed, we would be glad to clarify it or run a further analysis in the time that is left. If our response has resolved your concerns, we would be grateful if you would consider updating your assessment.

Thank you for your time.

Thank you for the clarification and additional experiments, which address most of my concerns. I raise my rating to 3

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the three reviewers their careful feedback and the AC for their valuable time. We are glad that the reviewers find PACT practical and simple: it trains on intermediate denoising decisions with the exact token conditionals of the sampled transitions, needs no sequence-likelihood estimate or ratio clipping, and adds no inference overhead (tZWQ, DVnf). They also recognise the empirical results, including the best four-task average at both lengths and the compute comparison (bDXk, tZWQ).

The discussion centered on three aspects: whether per-transition credit assignment is what drives the gains and how PACT relates to existing balance objectives; the best-of-checkpoint evaluation and the AGRPO reproduction; and two missing baselines. We addressed all three, and the changes are marked in blue in the uploaded revised manuscript which thus helped in increasing our score from (3,2,2) to (3,3,2)

Concern (reviewers)What we addedMain results
Does per-transition credit drive the gains? (tZWQ, DVnf, bDXk)Section 5.2: ablations on GSM8K changing one loss term at a time, everything else fixedPer-path instead of per-transition consistency learns far more slowly (77.8 vs 85.9 at round 200) and costs 3.4× more per round. Removing the anchors costs 7.4 points. Removing the consistency residual collapses training, thus showing that the gains arise from the per transition credit and the anchor scores.
Novelty vs consistency and balance objectives (bDXk)Appendix D: table placing PACT next to detailed balance, Trust-PCL, ∇-GFN, DDPP, DTM, TraFLPACT's contribution is how per transition consistency is imposed on a DLLM's denosing where its a novel contribution; the ablations show this is where the gains come from.
Best-of-checkpoint selection (bDXk, tZWQ, DVnf)Appendix E: final-checkpoint results for PACT and the baselines we trainedOur rule follows prior work. At the final checkpoint, with no selection, PACT still stays 5.2 points above the strongest reproduced baseline.
AGRPO reproduction and train/eval mismatch (tZWQ, DVnf)Code revision and settings; evaluation at AGRPO's training length (384)The mismatch is part of AGRPO's released design. At length 384 AGRPO reaches 83.3 on GSM8K and 47.3 on Countdown, against PACT's 89.1 and 79.9.
Missing baselines ESPO and JustGRPO (bDXk)ESPO in Table 1; JustGRPO discussed in Section 2PACT improves over ESPO by 6.2 and 7.0 points on the four-task average at lengths 256 and 512. Table 1 now has eleven baselines.
Eq. 5 probabilities and the order normalizer (tZWQ)Section 3; Section 5.1 and Appendix CEq. 5 scores the untruncated temperature-1 softmax. The token-only residual is exact when the target is indifferent to the order, and approximate otherwise.

In summary, the new ablations isolate the per-transition residual on an anchored potential as the source of the gains, and the final-checkpoint results show that the comparison does not depend on checkpoint selection. We thank the reviewers again for their constructive feedback and hope these additions help them to reconsider their scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.