- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the reliability of variational likelihood proxies in diffusion policy optimization. It argues that a growing gap between the ELBO and the true log-likelihood can distort policy-gradient updates, and proposes DiPOD to counter this drift through on-policy self-distillation. The practical algorithm adds an ELBO maximization term to existing RL updates. Experiments apply this regularizer to FPO and SPG on four reasoning tasks using LLaDA-8B-Instruct. Additional analyses examine bound tightness, while continuous-control experiments evaluate an initialization-only self-distillation variant.
Critical strengths
-
The paper provides a useful account of how optimizing a variational proxy can change the approximation gap rather than the intended output likelihood. Proposition C.6 connects this discrepancy to policy-gradient bias under explicit assumptions. This gives a meaningful motivation for maintaining bound tightness during post-training and leads to a simple modification of existing algorithms.
-
The practical results are promising across two underlying optimization methods. The substantial gains on logical reasoning tasks support the usefulness of the regularizer beyond the idealized analysis.
Critical weaknesses
-
The theoretical guarantee has a substantially narrower scope than the practical algorithm. Theorem C.8 assumes realizability and an approximately optimal self-distillation oracle. Its improvement bound is relative to the post-distillation policy; improvement over the entire cycle additionally requires exact or policy-preserving distillation. Approximate optimality on the reference distribution does not automatically control the discrepancy under the returned policy, as Remark C.5 acknowledges. Algorithm 2 instead combines RL and ELBO gradients in each update without establishing these conditions. The theory therefore motivates the implementation but does not establish its policy-improvement guarantee. The continuous-control results also use only initial self-distillation, so they provide limited evidence for the proposed mechanism of continually correcting drift during optimization.
-
The experiments do not directly establish that the practical regularizer improves alignment with the true policy gradient. In the Sudoku case study, Appendix E.3 measures the ELBO gap under uniform sequential token revelation, whereas rollouts use confidence-based decoding and training uses method-specific scoring objectives. The authors acknowledge that these measurements concern a different evaluation distribution and do not measure training-gradient error. The observed association between a smaller gap and higher reward is therefore supportive but insufficient to validate the full double-drift explanation. Appendix D offers a particularly suitable setting for a stronger test because exact likelihoods and gradients are tractable, yet the reported analysis focuses on the gap for one output rather than gradient alignment and resulting policy improvement.
Questions for authors
-
Which conclusions of Theorem C.8 are intended to apply to Algorithm 2? Can the authors characterize the effect of finite joint updates on both policy displacement and the on-policy variational discrepancy?
-
In the two-token experiment, does DiPOD improve the cosine similarity or relative error between the variational policy-gradient estimator and the exact reward gradient throughout training, and do these changes predict improvements in expected reward?
-
In that tractable setting, can the authors compare against regularization toward the rollout policy at a comparable policy displacement, to distinguish bound tightening from the benefit of simply limiting policy changes?
4 comments
In the Sudoku case study, Appendix E.3 measures the ELBO gap under uniform sequential token revelation, whereas rollouts use confidence-based decoding and training uses method-specific scoring objectives. […] Appendix D offers a particularly suitable setting for a stronger test because exact likelihoods and gradients are tractable, yet the reported analysis focuses on the gap for one output rather than gradient alignment and resulting policy improvement.
Additional real-world verification: three-shot Sudoku. We go beyond the submitted scalar-gap diagnostic by directly comparing estimated log-likelihood and matched ELBO score gradients at checkpoints from actual LLaDA-8B-Instruct RL training. At the latest measured checkpoint, version 1,200, mean score cosine is 0.516 versus 0.981 for FPO/FPO+DiPOD and 0.863 versus 0.962 for SPG/SPG+DiPOD, using 64 paths per block and 24 completions per method. This gives direct real-world evidence of score-gradient drift and better directional fidelity with DiPOD, complementing the enumerable-model analysis. We have made these results prominent in revised Section 4.1 and Appendix G.2.
The evaluation-model distinction is deliberate and explicitly documented. Appendix E.3 (p. 27) states that uniform reveals “can change the output distribution” and that the diagnostic does not directly measure “the error in the training gradient.” Accordingly, the submitted Figure 2 supports bound-tightness behavior within that evaluation model, rather than proving fidelity of the confidence-based decoder's training gradient.
The new cosine measurements use that same explicitly defined evaluation model, over the trainable LoRA coordinates. They test its score-gradient fidelity, rather than the advantage-weighted reward gradient of the confidence-based decoder. Numerical likelihood-gradient convergence remains unresolved. The updated snapshot contains 1,324/1,344 completed tasks and approximately 1,879 GPU-hours of recorded task runtime. Appendix G.2 plots all 48 complete method/checkpoint summaries, with checkpoint 200 still incomplete; we will include the complete plot in the final version. Directional improvement does not automatically improve gradient magnitude; the relative-error results are reported separately in G.2.
Complementary exact verification: two-token model.
We have added the stronger tractable test in Appendix G.1. It enumerates all four outputs and both decoding orders, measures the on-policy mean gap, and compares the reward-only proxy with the exact reward gradient at every update. At update 1,500 without KL, cosine increases from 0.8269 to 0.9575 for FPO and 0.9824 to 0.9950 for SPG; relative error decreases from 0.6155 to 0.3085 and 0.1868 to 0.1010, respectively. The complete trajectories include expected reward. Final rewards are nearly equal, so this verifies improved gradient fidelity in the tested model without claiming that it causes a strict reward advantage at every step.
Responses to questions
Which conclusions of Theorem C.8 are intended to apply to Algorithm 2? Can the authors characterize the effect of finite joint updates on both policy displacement and the on-policy variational discrepancy?
Algorithm 2 is motivated by the theorem's bound-tightening principle; it is not claimed to satisfy its oracle assumptions. As explained in the first weakness response, we do not establish a general finite-step guarantee for Algorithm 2's policy displacement or on-policy gap. This scope is now explicit in Appendix G; Appendix G.3 supplies empirical training-component diagnostics.
In the two-token experiment, does DiPOD improve the cosine similarity or relative error between the variational policy-gradient estimator and the exact reward gradient throughout training, and do these changes predict improvements in expected reward?
The gradient-evidence response above reports the new exact measurements. Appendix G.1 plots both metrics and reward throughout training. DiPOD improves alignment over almost the whole trajectory in this example, but does not produce a uniformly larger reward: final unregularized rewards are FPO 0.968514 versus FPO+DiPOD 0.969165, and SPG 0.968798 versus SPG+DiPOD 0.968528. We therefore distinguish improved gradient fidelity from guaranteed reward improvement.
In that tractable setting, can the authors compare against regularization toward the rollout policy at a comparable policy displacement, to distinguish bound tightening from the benefit of simply limiting policy changes?
We added an exact-output-KL ablation and coefficient sweep in Appendix G.1. Its reference is frozen at initialization, rather than refreshed to the latest rollout policy, and equal coefficients do not enforce equal realized displacement. We therefore do not present it as the exact moving-reference, matched-displacement control requested here.
It nevertheless separates additional gap control from ordinary anchoring: at KL coefficient 0.2, adding DiPOD reduces the on-policy mean gap from 0.037701 to 0.002172 for FPO, and 0.005597 to 0.000713 for SPG. Without KL, SPG and SPG+DiPOD end at nearly identical realized KL (0.523454 and 0.523136) and reward, while their mean gaps differ substantially (0.012307 and 0.002959). These are descriptive comparisons, not enforced displacement matching.
Independently, the exact fixed-output family in G.1 has uniform output policies for every , hence zero marginal KL even to the current rollout policy, while its posterior gap is and can diverge. This establishes why marginal-policy proximity alone is insufficient. It illustrates a broader structural distinction; it does not replace the missing controlled moving-reference experiment or establish that KL never helps.
Thanks for your response. I find most of my concerns have been addressed. Therefore, I will maintain weak acceptance.
We appreciate the reviewer’s careful technical assessment and recognition of the link between variational discrepancy and gradient bias, as well as the promising results across two underlying methods. Thank you for the detailed feedback on the scope of the guarantees and the diagnostics needed to examine the proposed mechanism.
Responses to weaknesses
Theorem C.8's general policy-update bound already allows self-distillation to change the policy. Equation (28) (p. 23) quantifies the reference-weighted policy KL and reference-distribution gap. Remark C.5 (p. 21) explicitly notes that the returned occupancy distribution may differ from the reference. The proof therefore uses the returned policy's own gap, DρθˉkL(θˉk), in Equation (29), which compares J(θk+1) with J(θˉk). Policy change is thus accounted for in the general analysis.
The informal statement compresses these distinctions for readability. Revised Section 3.2 now identifies the post-distillation comparison policy and its own on-policy gap explicitly; the formal theorem and proof are unchanged.
Exact or policy-preserving distillation is a separate sufficient case for the full-cycle comparison, not a necessary condition for net improvement. It makes the pre- and post-distillation returns equal and transfers the reference gap bound to the returned distribution. Without preservation, a full cycle can still improve when the subsequent gain exceeds any distillation-induced return loss.
Algorithm 2 combines both directions in one update and does not automatically inherit the oracle guarantee. We state this scope in Appendix G. A stationary joint direction need not have stationary components; Appendix G.3 reports actual component norms and opposition without asserting separate convergence.
The initialization-only control protocol is also an explicit limitation of the submission. Appendix F.1 (p. 28) states: “The only added component is an initial self-distillation stage before normal FPO++ policy-gradient training,” and leaves a fully interleaved schedule to future work. Those experiments test the benefit of a tighter-bound initialization, not repeated correction throughout motion training. We have reinforced that distinction in F.1 and qualified the finite-step policy-preservation wording. The broader contribution remains useful: the conditional analysis identifies why bound tightness matters, and the language experiments test the practical continual regularizer across two underlying methods. The control experiment has a more limited role.