acceptodds
Under review as a conference paper at ICLR 2027

DiPOD: Diffusion Policy Optimization without Drifting Apart

Abstract

RL post-training has become increasingly pivotal for improving diffusion policies, but existing diffusion policy gradient methods are often unstable and cannot achieve reliable policy improvement. We identify the cause as the double drift phenomenon: optimizing a variational surrogate can let the ELBO separate from true log-likelihood, which then makes the resulting proxy policy gradient misaligned with the true policy gradient of expected return. We propose **DiPOD**, a diffusion policy optimization framework that maintains tight-bound behavior throughout training by interleaving self-distillation with policy-improving gradient updates. This leads to a simple and practical algorithm: augmenting each diffusion policy-gradient update with an on-policy ELBO regularizer. Across diffusion language model post-training and continuous-control diffusion policies, DiPOD substantially stabilizes training and reaches higher rewards against previous methods.

Primary area:
reinforcement learning
Keywords:
Diffusion Model, Reinforcement Learning, Diffusion Language Model, Policy Gradient

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vmGMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reliability of variational likelihood proxies in diffusion policy optimization. It argues that a growing gap between the ELBO and the true log-likelihood can distort policy-gradient updates, and proposes DiPOD to counter this drift through on-policy self-distillation. The practical algorithm adds an ELBO maximization term to existing RL updates. Experiments apply this regularizer to FPO and SPG on four reasoning tasks using LLaDA-8B-Instruct. Additional analyses examine bound tightness, while continuous-control experiments evaluate an initialization-only self-distillation variant.

Critical strengths
  1. The paper provides a useful account of how optimizing a variational proxy can change the approximation gap rather than the intended output likelihood. Proposition C.6 connects this discrepancy to policy-gradient bias under explicit assumptions. This gives a meaningful motivation for maintaining bound tightness during post-training and leads to a simple modification of existing algorithms.

  2. The practical results are promising across two underlying optimization methods. The substantial gains on logical reasoning tasks support the usefulness of the regularizer beyond the idealized analysis.

Critical weaknesses
  1. The theoretical guarantee has a substantially narrower scope than the practical algorithm. Theorem C.8 assumes realizability and an approximately optimal self-distillation oracle. Its improvement bound is relative to the post-distillation policy; improvement over the entire cycle additionally requires exact or policy-preserving distillation. Approximate optimality on the reference distribution does not automatically control the discrepancy under the returned policy, as Remark C.5 acknowledges. Algorithm 2 instead combines RL and ELBO gradients in each update without establishing these conditions. The theory therefore motivates the implementation but does not establish its policy-improvement guarantee. The continuous-control results also use only initial self-distillation, so they provide limited evidence for the proposed mechanism of continually correcting drift during optimization.

  2. The experiments do not directly establish that the practical regularizer improves alignment with the true policy gradient. In the Sudoku case study, Appendix E.3 measures the ELBO gap under uniform sequential token revelation, whereas rollouts use confidence-based decoding and training uses method-specific scoring objectives. The authors acknowledge that these measurements concern a different evaluation distribution and do not measure training-gradient error. The observed association between a smaller gap and higher reward is therefore supportive but insufficient to validate the full double-drift explanation. Appendix D offers a particularly suitable setting for a stronger test because exact likelihoods and gradients are tractable, yet the reported analysis focuses on the gap for one output rather than gradient alignment and resulting policy improvement.

Questions for authors
  1. Which conclusions of Theorem C.8 are intended to apply to Algorithm 2? Can the authors characterize the effect of finite joint updates on both policy displacement and the on-policy variational discrepancy?

  2. In the two-token experiment, does DiPOD improve the cosine similarity or relative error between the variational policy-gradient estimator and the exact reward gradient throughout training, and do these changes predict improvements in expected reward?

  3. In that tractable setting, can the authors compare against regularization toward the rollout policy at a comparable policy displacement, to distinguish bound tightening from the benefit of simply limiting policy changes?

4 comments

We appreciate the reviewer’s careful technical assessment and recognition of the link between variational discrepancy and gradient bias, as well as the promising results across two underlying methods. Thank you for the detailed feedback on the scope of the guarantees and the diagnostics needed to examine the proposed mechanism.

Responses to weaknesses

Its improvement bound is relative to the post-distillation policy; improvement over the entire cycle additionally requires exact or policy-preserving distillation. Approximate optimality on the reference distribution does not automatically control the discrepancy under the returned policy, as Remark C.5 acknowledges. Algorithm 2 instead combines RL and ELBO gradients in each update without establishing these conditions. […] The continuous-control results also use only initial self-distillation, so they provide limited evidence for the proposed mechanism of continually correcting drift during optimization.

Theorem C.8's general policy-update bound already allows self-distillation to change the policy. Equation (28) (p. 23) quantifies the reference-weighted policy KL and reference-distribution gap. Remark C.5 (p. 21) explicitly notes that the returned occupancy distribution may differ from the reference. The proof therefore uses the returned policy's own gap, DρθˉkL(θˉk)D^L_{\rho_{\bar\theta_k}}(\bar\theta_k), in Equation (29), which compares J(θk+1)J(\theta_{k+1}) with J(θˉk)J(\bar\theta_k). Policy change is thus accounted for in the general analysis.

The informal statement compresses these distinctions for readability. Revised Section 3.2 now identifies the post-distillation comparison policy and its own on-policy gap explicitly; the formal theorem and proof are unchanged.

Exact or policy-preserving distillation is a separate sufficient case for the full-cycle comparison, not a necessary condition for net improvement. It makes the pre- and post-distillation returns equal and transfers the reference gap bound to the returned distribution. Without preservation, a full cycle can still improve when the subsequent gain exceeds any distillation-induced return loss.

Algorithm 2 combines both directions in one update and does not automatically inherit the oracle guarantee. We state this scope in Appendix G. A stationary joint direction need not have stationary components; Appendix G.3 reports actual component norms and opposition without asserting separate convergence.

The initialization-only control protocol is also an explicit limitation of the submission. Appendix F.1 (p. 28) states: “The only added component is an initial self-distillation stage before normal FPO++ policy-gradient training,” and leaves a fully interleaved schedule to future work. Those experiments test the benefit of a tighter-bound initialization, not repeated correction throughout motion training. We have reinforced that distinction in F.1 and qualified the finite-step policy-preservation wording. The broader contribution remains useful: the conditional analysis identifies why bound tightness matters, and the language experiments test the practical continual regularizer across two underlying methods. The control experiment has a more limited role.

In the Sudoku case study, Appendix E.3 measures the ELBO gap under uniform sequential token revelation, whereas rollouts use confidence-based decoding and training uses method-specific scoring objectives. […] Appendix D offers a particularly suitable setting for a stronger test because exact likelihoods and gradients are tractable, yet the reported analysis focuses on the gap for one output rather than gradient alignment and resulting policy improvement.

Additional real-world verification: three-shot Sudoku. We go beyond the submitted scalar-gap diagnostic by directly comparing estimated log-likelihood and matched ELBO score gradients at checkpoints from actual LLaDA-8B-Instruct RL training. At the latest measured checkpoint, version 1,200, mean score cosine is 0.516 versus 0.981 for FPO/FPO+DiPOD and 0.863 versus 0.962 for SPG/SPG+DiPOD, using 64 paths per block and 24 completions per method. This gives direct real-world evidence of score-gradient drift and better directional fidelity with DiPOD, complementing the enumerable-model analysis. We have made these results prominent in revised Section 4.1 and Appendix G.2.

The evaluation-model distinction is deliberate and explicitly documented. Appendix E.3 (p. 27) states that uniform reveals “can change the output distribution” and that the diagnostic does not directly measure “the error in the training gradient.” Accordingly, the submitted Figure 2 supports bound-tightness behavior within that evaluation model, rather than proving fidelity of the confidence-based decoder's training gradient.

The new cosine measurements use that same explicitly defined evaluation model, over the trainable LoRA coordinates. They test its score-gradient fidelity, rather than the advantage-weighted reward gradient of the confidence-based decoder. Numerical likelihood-gradient convergence remains unresolved. The updated snapshot contains 1,324/1,344 completed tasks and approximately 1,879 GPU-hours of recorded task runtime. Appendix G.2 plots all 48 complete method/checkpoint summaries, with checkpoint 200 still incomplete; we will include the complete plot in the final version. Directional improvement does not automatically improve gradient magnitude; the relative-error results are reported separately in G.2.

Complementary exact verification: two-token model.

We have added the stronger tractable test in Appendix G.1. It enumerates all four outputs and both decoding orders, measures the on-policy mean gap, and compares the reward-only proxy with the exact reward gradient at every update. At update 1,500 without KL, cosine increases from 0.8269 to 0.9575 for FPO and 0.9824 to 0.9950 for SPG; relative error decreases from 0.6155 to 0.3085 and 0.1868 to 0.1010, respectively. The complete trajectories include expected reward. Final rewards are nearly equal, so this verifies improved gradient fidelity in the tested model without claiming that it causes a strict reward advantage at every step.

Responses to questions

Which conclusions of Theorem C.8 are intended to apply to Algorithm 2? Can the authors characterize the effect of finite joint updates on both policy displacement and the on-policy variational discrepancy?

Algorithm 2 is motivated by the theorem's bound-tightening principle; it is not claimed to satisfy its oracle assumptions. As explained in the first weakness response, we do not establish a general finite-step guarantee for Algorithm 2's policy displacement or on-policy gap. This scope is now explicit in Appendix G; Appendix G.3 supplies empirical training-component diagnostics.

In the two-token experiment, does DiPOD improve the cosine similarity or relative error between the variational policy-gradient estimator and the exact reward gradient throughout training, and do these changes predict improvements in expected reward?

The gradient-evidence response above reports the new exact measurements. Appendix G.1 plots both metrics and reward throughout training. DiPOD improves alignment over almost the whole trajectory in this example, but does not produce a uniformly larger reward: final unregularized rewards are FPO 0.968514 versus FPO+DiPOD 0.969165, and SPG 0.968798 versus SPG+DiPOD 0.968528. We therefore distinguish improved gradient fidelity from guaranteed reward improvement.

In that tractable setting, can the authors compare against regularization toward the rollout policy at a comparable policy displacement, to distinguish bound tightening from the benefit of simply limiting policy changes?

We added an exact-output-KL ablation and coefficient sweep in Appendix G.1. Its reference is frozen at initialization, rather than refreshed to the latest rollout policy, and equal coefficients do not enforce equal realized displacement. We therefore do not present it as the exact moving-reference, matched-displacement control requested here.

It nevertheless separates additional gap control from ordinary anchoring: at KL coefficient 0.2, adding DiPOD reduces the on-policy mean gap from 0.037701 to 0.002172 for FPO, and 0.005597 to 0.000713 for SPG. Without KL, SPG and SPG+DiPOD end at nearly identical realized KL (0.523454 and 0.523136) and reward, while their mean gaps differ substantially (0.012307 and 0.002959). These are descriptive comparisons, not enforced displacement matching.

Independently, the exact fixed-output family in G.1 has uniform output policies for every δ\delta, hence zero marginal KL even to the current rollout policy, while its posterior gap is −12log⁡(1−4δ2)-\tfrac12\log(1-4\delta^2) and can diverge. This establishes why marginal-policy proximity alone is insufficient. It illustrates a broader structural distinction; it does not replace the missing controlled moving-reference experiment or establish that KL never helps.

Thanks for your response. I find most of my concerns have been addressed. Therefore, I will maintain weak acceptance.

Reviewer_RxEaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the instability of RL (reinforcement learning) for diffusion policies from a variational inference perspective. The authors identify a double-drift phenomenon: as RL optimization proceeds, the ELBO may deviate from the true log-likelihood, which consequently causes the proxy policy gradient to deviate from the true policy gradient. To address this issue, the authors propose an approach called DiPOD (diffusion policy optimization without drifting apart), which interleaves policy-gradient updates with self-distillation to maintain a tight variational bound. Experiments on diffusion language models and continuous-control tasks demonstrate improved training stability and reward performance.

Critical strengths

The double-drift phenomenon provides an insightful perspective on the instability of variational inference-based diffusion RL methods, connecting variational bound tightness with policy-gradient estimation accuracy.

The paper provides a theoretical analysis that connects variational discrepancy, gradient estimation error, and policy improvement. The DiPOD approach has led to substantial improvements on several reasoning benchmarks, with additional encouraging results on continuous-control tasks.

Critical weaknesses

• Theorem 3.1 establishes policy-improvement guarantees for the idealized alternating procedure in Algorithm 1, where self-distillation is approximately optimized before subsequent policy-gradient updates. However, Algorithm 2 replaces this procedure with a joint gradient update. While the intuition is that both the ELBO discrepancy and the policy-gradient error should remain small throughout training, convergence of the combined update does not necessarily imply that either component is individually small. In particular, the ELBO gradient and the RL gradient may cancel each other. It is unclear to what extent the guarantees established for Algorithm 1 carry over to Algorithm 2.

• Appendix D presents a toy experiment comparing the discrepancy between the true log-likelihood and the ELBO. However, the central motivation of the paper is not merely that the ELBO can drift from the log-likelihood, but that the discrepancy can lead to misalignment between the proxy policy gradient and the true policy gradient. The current experiment does not directly quantify this second component of the double-drift phenomenon. In particular, it would be interesting to understand how the discrepancy between the true and proxy policy gradients evolves during training, and whether DiPOD effectively reduces this discrepancy.

• The proposed method introduces an additional ELBO regularization term to stabilize policy optimization. However, it remains unclear whether the observed improvements arise specifically from tightening the variational bound or more generally from the stabilizing effect of an auxiliary regularization objective. Standard KL regularization is widely used to stabilize RL updates by constraining policy changes. A comparison with an appropriate KL-based regularization baseline would help clarify whether DiPOD offers any benefit beyond conventional policy regularization.

Questions for authors

• Please consider separately tracking the norms of the RL gradient and the ELBO regularization gradient throughout training. This will help determine whether convergence is due to both gradients decreasing or the two just canceling out each other. • Appendix D: in the small discrete state space where the true policy gradient can be computed exactly, it should be feasible to measure the cosine similarity and relative error between the true policy gradient and the proxy gradient. • A suggestion: compare DiPOD against standard policy regularization methods, such as KL regularization toward a frozen reference policy.

2 comments

Thank you for the thoughtful review and for highlighting the double-drift perspective, the connection between variational discrepancy and policy improvement, and the empirical gains. Your suggestions for direct gradient measurements, component tracking, and KL comparisons have helped us sharpen the empirical analysis.

Responses to weaknesses

Algorithm 2 replaces this procedure with a joint gradient update. […] convergence of the combined update does not necessarily imply that either component is individually small. In particular, the ELBO gradient and the RL gradient may cancel each other. It is unclear to what extent the guarantees established for Algorithm 1 carry over to Algorithm 2.

Correct: the oracle guarantee does not automatically transfer to Algorithm 2. Section 3.3 (p. 7) describes Equation (7) as a “first-order approximation”; we now state its mathematical limitation explicitly in Appendix G and clarify the informal Theorem 3.1's comparison policy and discrepancy notation. The latter guarantees improvement after distillation in terms of the returned policy's own gap; exact or policy-preserving distillation is a sufficient case that also yields the full-cycle comparison. It does not establish convergence of Algorithm 2.

The central motivation of the paper is not merely that the ELBO can drift from the log-likelihood, but that the discrepancy can lead to misalignment between the proxy policy gradient and the true policy gradient. The current experiment does not directly quantify this second component of the double-drift phenomenon.

Additional real-world verification: three-shot Sudoku. Beyond the enumerable example, we directly examine score-gradient drift on checkpoints from actual LLaDA-8B-Instruct RL training. We compare Monte Carlo estimates of the log-likelihood gradient and its matched ELBO gradient over all trainable LoRA parameters. At the latest measured checkpoint at iteration 1,200, mean cosine across 24 completions increases from 0.516 to 0.981 for FPO and from 0.863 to 0.962 for SPG with DiPOD.

Cosine near one means that the two gradient directions agree. These measurements expose substantial score-direction misalignment in the real-world FPO-trained model and higher alignment with DiPOD for both training methods. They therefore provide direct real-world empirical evidence for gradient drift and its mitigation, beyond the association between scalar ELBO gap and reward in the submitted Figure 2. Revised Section 4.1 reports the endpoint comparison, and Appendix G.2 now plots 48 complete method/checkpoint summaries; only checkpoint 200 remains incomplete.

Complementary exact verification: two-token model.

We have added direct gradient measurements in the tractable two-token setting, where all outputs and both decoding orders are exactly enumerated. At each update we compare the reward-only FPO/SPG proxy gg with the exact expected-reward gradient g⋆g_\star, excluding the KL and DiPOD terms from this comparison. At update 1,500 without KL:

Methodcos⁡(g,g⋆)\cos(g,g_\star)∥g−g⋆∥/∥g⋆∥\|g-g_\star\|/\|g_\star\|
FPO0.82690.6155
FPO+DiPOD0.95750.3085
SPG0.98240.1868
SPG+DiPOD0.99500.1010

It remains unclear whether the observed improvements arise specifically from tightening the variational bound [...] A comparison with an appropriate KL-based regularization baseline [...]

The two regularizers control different objects. A standard RL KL penalty such as KL(πθ∥πref)\mathrm{KL}(\pi_\theta\|\pi_{\rm ref}) constrains how far the marginal output policy moves from a reference policy. Double drift is governed by the within-model variational discrepancy

=\mathrm{KL}\bigl(q(z\mid x)\|p_\theta(z\mid x)\bigr),$$ which compares latent posteriors for the current model. A small policy KL does not imply a small $\mathcal D^L$: models can have similar output distributions but different latent posterior gaps. Section 3.3 (p. 7, “Relation to KL Regularization”) already states that “Policy proximity alone does not ensure a tight bound.” The mechanisms are distinct and compatible. To test this distinction, we added an eight-arm, exactly enumerated two-token ablation: FPO/SPG × DiPOD off/on × exact output KL off/on, with KL coefficient 0.2 and a frozen initial-policy reference. At update 1,500: | Method with KL | Mean ELBO gap | Exact KL to start | | --- | ---: | ---: | | FPO + KL | 0.037701 | 0.158497 | | FPO+DiPOD + KL | 0.002172 | 0.140441 | | SPG + KL | 0.005597 | 0.141339 | | SPG+DiPOD + KL | 0.000713 | 0.138272 | Adding DiPOD reduces the mean gap beyond KL alone by **94.2% for FPO** and **87.3% for SPG**. Without KL, gaps also decrease substantially while final rewards and realized KL remain close. KL itself can reduce drift, but stronger anchoring also limits reward improvement, and the coefficient sweep is not uniformly monotone in gap. Equal coefficients are not equal realized displacement; we do not present these as exactly matched-KL comparisons. Appendix G.1 separately gives a family with **identical uniform output policies and zero output KL**, yet gap $-\tfrac12\log(1-4\delta^2)$ for every output, which can grow arbitrarily large. This example illustrates the broader structural reason marginal-policy KL alone cannot guarantee removal of double drift. The new ablation and derivation are linked from the main-body KL discussion. ### Responses to questions > Please consider separately tracking the norms of the RL gradient and the ELBO regularization gradient [...] The two terms have complementary roles: the RL term pursues reward improvement, while the weighted ELBO term promotes a tighter likelihood bound on the same rollout data. Their directions need not always align. A negative cosine can reflect the regularizer counteracting changes favored by the reward surrogate; it does not by itself imply stalled learning through cancellation. We tracked both components throughout all 1,250 updates of separately instrumented three-shot Sudoku runs. Appendix G.3 plots the RL, weighted ELBO-loss, and sum norms, component cosine, and relative gradient norm before clipping and Adam. The relative gradient norm is the norm of the sum divided by the sum of component norms: 1 indicates aligned contributions or a single nonzero component; 0 indicates exact cancellation. - **SPG+DiPOD:** Both component norms decrease. Comparing medians over the first and last 100 updates, the RL norm falls from **0.0311 to 0.000739**, and the weighted ELBO norm from **0.0332 to 0.00447**. The combined norm also decreases, accompanying actual component shrinkage. - **FPO+DiPOD:** Both contributions remain active, with RL spikes and frequent opposition. The median combined norm is **1.45 initially and 2.14 late** over the same windows: the opposing components do not simply cancel the joint gradient to zero. Read alongside the higher reward and smaller measured gaps in submitted Figure 2, and the additional score-alignment results in Appendix G.2, these observations support a useful balance between reward optimization and bound tightening. This is empirical support for the practical update, without asserting that both components converge to zero. > Appendix D: in the small discrete state space where the true policy gradient can be computed exactly, it should be feasible to measure the cosine similarity and relative error between the true policy gradient and the proxy gradient. Both are now measured at every two-token update and plotted in Appendix G.1. The second weakness response reports the endpoint values and distinguishes these exact reward-gradient measurements from the estimated Sudoku score gradients. > A suggestion: compare DiPOD against standard policy regularization methods, such as KL regularization toward a frozen reference policy. Appendix G.1 now includes precisely the frozen-reference exact-output-KL ablation described in the third weakness response, including all eight arms and the coefficient sweep's interpretation.
Reviewer_77jfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies policy optimization for diffusion and flow models when the intractable likelihood is replaced by a variational surrogate. It describes a “double-drift” mechanism: policy updates can increase the discrepancy between the ELBO and the true log-likelihood, potentially causing the surrogate gradient to deviate from the true policy gradient. The proposed framework, DiPOD, interleaves policy-gradient updates with self-distillation on samples from the current reference policy. Its practical implementation adds an ELBO-maximization term to the policy update using the same rollout data. The paper provides a conditional analysis connecting variational-gap control to gradient accuracy and policy improvement. Empirical evaluation includes a three-shot Sudoku diagnostic, and LLaDA-8B-Instruct post-training on four reasoning benchmarks. The largest reported language-model improvements occur on Countdown and Sudoku.

Critical strengths
  1. A simple intervention produces substantial reported improvements on several reasoning tasks. The practical method requires adding an ELBO term on existing rollout data and can be incorporated into both FPO-style and SPG-style updates. In Table 1, SPG+DiPOD improves the reported Countdown accuracy from 51.95 to 80.08 and Sudoku accuracy from 25.12 to 97.56. These are substantial gains, and the paired comparisons with and without the regularizer provide useful evidence that this small modification can materially affect optimization.

  2. The paper offers a useful perspective on the reliability of variational policy gradients. Distinguishing objective-level bounds from gradient fidelity is valuable, and the self-distillation decomposition in Lemma C.7 helps explain why fitting current-policy samples can control both policy mismatch and variational discrepancy. The combination of a tractable example, language-model diagnostics, and additional control experiments makes the proposed principle worth investigating. I also appreciate that Appendix E.3 explicitly documents the details of its likelihood-gap measurements.

Critical weaknesses
  1. Clarification of the theoretical guarantee. I have some questions about Theorem 3.1. It seems to suggest that an approximately optimal self-distillation step is sufficient to guarantee policy improvement. However, in Theorem C.8, the guarantee for the full iteration appears to require an additional condition that self-distillation preserves the original policy. Could the authors clarify whether this condition is necessary for the claim in Theorem 3.1? Also, the practical implementation in Algorithm 2 combines the ELBO regularizer with the policy-gradient update, rather than performing separate self-distillation steps. I wonder whether the theoretical guarantee still holds for this practical algorithm.

  2. Unclear Checkpoint Selection and Insufficient Evidence of Training Stability. Figure 3 shows that FPO+DiPOD experiences a significant performance collapse in the later stages of training on Countdown, while Table 1 reports a relatively high accuracy of 75.39%. I wonder whether the results in Table 1 are obtained from the final checkpoint or a selected checkpoint. In addition, the main experiments do not report results across multiple random seeds, although Appendix E.2 mentions phase-transition-like behavior on Sudoku, and Table 3 shows that the performance is sensitive to the choice of β\beta. I think additional clarification and experiments with multiple random seeds would help establish the reliability and stability of the reported improvements.

Questions for authors
  1. Scope of the theoretical guarantee. Could the authors clarify the relationship between Theorem 3.1 and Theorem C.8, particularly the distinction between the ELBO discrepancy under the reference distribution and that under the returned policy's own distribution? Which conclusions require exact or policy-preserving self-distillation? Moreover, to what extent do these theoretical guarantees apply to the practical implementation in Algorithm 2, which combines self-distillation regularization with policy-gradient updates?

  2. Checkpoint selection and training stability. Which checkpoints are used for the evaluations in Tables 1 and 2, and are they selected based on a held-out validation set? How should the late Countdown collapse of FPO+DiPOD in Figure 3 be reconciled with its reported performance in Table 1? In addition, given the phase-transition-like behavior on Sudoku acknowledged in Appendix E.2 and the sensitivity to β\beta shown in Table 3, could the authors provide results across multiple independent training seeds? Such results would help assess the reliability and stability of the reported improvements.

  3. Comparability of Sudoku baselines. Table 2 reports 76.4% accuracy for wd1 on zero-shot Sudoku at sequence length 256, which is much higher than the 25.12% reported for SPG. However, wd1 is not included in Table 1. I wonder whether these results are directly comparable. If the results are comparable, I think wd1 should be considered as an additional strong baseline when evaluating the improvements on Sudoku.

3 comments

We thank the reviewer for the careful review and for recognizing the substantial gains from a simple modification of FPO and SPG. We appreciate your positive assessment of the variational-gradient perspective and the documented likelihood-gap measurements.

Responses to weaknesses

It seems to suggest that an approximately optimal self-distillation step is sufficient to guarantee policy improvement. However, in Theorem C.8, the guarantee for the full iteration appears to require an additional condition that self-distillation preserves the original policy. Could the authors clarify whether this condition is necessary for the claim in Theorem 3.1? Also, the practical implementation in Algorithm 2 combines the ELBO regularizer with the policy-gradient update, rather than performing separate self-distillation steps.

The general improvement bound in Theorem C.8 does not assume policy-preserving self-distillation. Approximate distillation may change the policy, and the proof explicitly accounts for that change.

Let θk\theta_k be the parameter before distillation and θˉk\bar\theta_k its output. Lemma C.7 (pp. 22–23, Equation (26)) controls both the policy mismatch and the reference-distribution gap:

+D^L_{\rho_{\theta_k}}(\bar\theta_k)\le\varepsilon.$$ The KL term explicitly permits a change in the action distribution. Remark C.5 (p. 21) states: “An approximate oracle can still return a policy whose own occupancy distribution differs from the reference occupancy distribution.” Accordingly, the subsequent update uses the returned policy's **own on-policy gap**, $\bar\varepsilon_k=D^L_{\rho_{\bar\theta_k}}(\bar\theta_k)$. Theorem C.8 (p. 23, Equation (29)) gives $$J(\theta_{k+1})\ge J(\bar\theta_k) +\frac\eta2\bigl(\|\nabla J(\bar\theta_k)\|^2-c_g^2\bar\varepsilon_k\bigr).$$ This is a policy-update guarantee relative to the post-distillation policy $\bar\theta_k$, with strict improvement when the displayed gain is positive. **The informal statement suppresses the pre- and post-distillation distinction and the two gap quantities for presentation simplicity.** Revised Theorem 3.1 now makes the comparison policy and returned-policy gap explicit; the formal C.8 statement and proof are unchanged. **Exact or policy-preserving distillation is a separate sufficient case for full-cycle improvement, not a requirement of the general bound.** It gives $J(\bar\theta_k)=J(\theta_k)$ and $\bar\varepsilon_k\le\varepsilon$, so the same bound also yields improvement relative to the pre-distillation policy under the stated gradient-size condition. Policy-changing distillation can also yield net improvement when the subsequent gain exceeds any distillation-induced return loss. Algorithm 2 is introduced in Section 3.3 (p. 7) as a “first-order approximation.” It does not enforce the oracle assumptions. We have made this scope explicit in Appendix G. The guarantee links tight on-policy bounds to adequate-gradient accuracy and subsequent policy improvement; a corresponding theoretical analysis of Algorithm 2 and other DiPOD variants remains future work.

Figure 3 shows that FPO+DiPOD experiences a significant performance collapse in the later stages of training on Countdown, while Table 1 reports a relatively high accuracy of 75.39%. I wonder whether the results in Table 1 are obtained from the final checkpoint or a selected checkpoint. In addition, the main experiments do not report results across multiple random seeds, although Appendix E.2 mentions phase-transition-like behavior on Sudoku, and Table 3 shows that the performance is sensitive to the choice of β.

Following the SPG codebase, our reproduced results select the checkpoint with the highest benchmark evaluation score among optimizer iterations 100, 200, 300, and so on. Selection uses the reported benchmark evaluation, without a separate held-out validation set. Tables 1 and 2 therefore report best observed benchmark scores, rather than performance at a prespecified final iteration. In particular, 75.39% is the best selected FPO+DiPOD Countdown accuracy. Figure 3 shows on-policy training reward over the whole run, including the late decline; the selected score is compatible with that later instability. We have specified the protocol in the Table 1 caption and Appendix E.1/Table 2, corrected Table 3's “final accuracy” wording, and retained the late-collapse qualification in the Figure 3 caption.

Independent 8B-model replications exceeded our compute budget, and d1, wd1, and SPG likewise do not report multi-seed aggregates for their main results. To make the comparison as controlled as possible, we explicitly use the same seed for all of our baseline and DiPOD runs, fixing the seeds for initialization, data sampling, and optimizer randomness. Within each matched baseline/DiPOD comparison, all other settings are held fixed and the objective is the only intentional change. We have made this protocol explicit in revised Section 4.2 and Appendix E.1. This does not quantify cross-seed uncertainty. Appendix E.2 (p. 25) already acknowledges Sudoku's “discrete phase-transition-like behavior,” and Table 3 (p. 26) documents coefficient sensitivity.

Responses to questions

Could the authors clarify the relationship between Theorem 3.1 and Theorem C.8, particularly the distinction between the ELBO discrepancy under the reference distribution and that under the returned policy's own distribution? Which conclusions require exact or policy-preserving self-distillation?

The first weakness response above gives the two distributions and the exact improvement comparison. Policy preservation is not assumed for the general policy-update bound; it is a sufficient case for the full-cycle comparison. Revised Theorem 3.1 makes this distinction and the endpoint condition explicit. Algorithm 2 is not claimed to inherit the oracle guarantee.

Which checkpoints are used for the evaluations in Tables 1 and 2, and are they selected based on a held-out validation set? How should the late Countdown collapse of FPO+DiPOD in Figure 3 be reconciled with its reported performance in Table 1? […] could the authors provide results across multiple independent training seeds?

No separate held-out validation set is used. We select the highest benchmark evaluation score among checkpoints at integer multiples of 100 optimizer iterations. The checkpoint-selection discussion above explains the 75.39% result and the single-seed limitation; best-checkpoint accuracy does not establish endpoint stability.

Table 2 reports 76.4% accuracy for wd1 on zero-shot Sudoku at sequence length 256, which is much higher than the 25.12% reported for SPG. However, wd1 is not included in Table 1. I wonder whether these results are directly comparable.

wd1 is an important stronger Sudoku baseline and is already included in the submitted Table 2 (p. 26). At zero-shot length 256, the reported comparison is wd1 76.4%, SPG 25.12%, and SPG+DiPOD 97.56%. Thus DiPOD's result also exceeds the stronger published wd1 baseline by 21.16 percentage points.

These share the zero-shot and length-256 setting, but wd1 is a published result, whereas SPG versus SPG+DiPOD is our paired comparison. We do not claim that the published comparison controls every implementation or evaluation detail. Section E.1 (p. 25) and the Table 2 caption identify these sources and distinguish zero-shot Sudoku from SPG's published three-shot setting. We have copied the published wd1 row into revised Table 1 and labeled its provenance explicitly; this adds visibility without presenting a new matched rerun.

Thank you for your detailed responses and revisions. I appreciate the authors' efforts in addressing my concerns. I have decided to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.