- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes EnergyOPD, an on-policy distillation framework that expresses teacher supervision, format penalties, and correctness rewards as additive energy terms. It also introduces a smoothing regularizer that encourages similar student log probabilities for original and locally perturbed responses. Experiments on mathematical reasoning and out-of-domain benchmarks report improvements over existing distillation baselines, alongside reduced training fluctuations and modest computational overhead.
Critical strengths
- The method combines dense teacher feedback with verifiable correctness and format constraints, addressing useful objectives beyond imitation alone.
- Experiments include multiple student scales, Qwen and Llama models, out-of-domain evaluations, and computational measurements. Component ablations provide preliminary evidence that both smoothing and additional supervision contribute to the reported gains.
Critical weaknesses
- In the experimental setting, energy is defined as negative log-probability, and the basic free-energy objective reduces to reverse KL. Incorporating correctness rewards and format penalties is also possible within conventional policy-gradient training. The paper therefore does not clearly establish what the energy formulation contributes beyond OPD with auxiliary rewards and regularization.
- Replacing tokens while keeping the suffix fixed does not guarantee semantic or reasoning consistency. Even when the final answer is preserved, the perturbed response may contain invalid reasoning, making it questionable to encourage similar probabilities for both responses. The removal ablation demonstrates empirical usefulness but does not establish that the regularizer addresses realistic rollout drift or explain why this particular invariance is desirable.
Questions for authors
- Does EnergyOPD outperform conventional OPD when both receive the same correctness rewards, format penalties, smoothing term, and tuning budget? This comparison would help establish whether the energy formulation provides benefits beyond its added supervision and regularization.
- Does the isolated benefit of smoothing persist beyond the 1.7B student? Since the component ablations are confined to this scale, can you provide evidence distinguishing a broadly useful regularizer from an effect specific to smaller models’ sensitivity to perturbations?
3 comments
Response to Reviewer F45D [2/2]
Q3: what does the energy formulation add?
A3: We accept the reviewer's premise: with the canonical energy , the partition function is and the free-energy term vanishes, so the optimization target reduces to an advantage-weighted policy-gradient objective. The energy view, however, contributes three things that the scalar-loss parameterization does not define.
First, Eq. (7) gives a constructive composition rule. The sum
defines a single tilted target distribution
\pi_{\mathrm{te}}(\mathbf{x}\mid\mathbf{q})^{\alpha} \exp\!\big(-\beta E_{\mathrm{Exec}}(\tau)+\gamma R(\tau)\big).$$ In conventional policy-gradient training, correctness rewards and format penalties are added locally to the token-level advantage $A_t$; they do not define a coherent target distribution with a normalizer. The energy composition changes the mode structure of the target, not merely the scale of an advantage. Second, Theorem 2 ties the smoothing term to a geometric variance bound. After the revised local-perturbation assumption $|E_S(\tau)-E_S(\tau')|\le\delta$, the estimator variance satisfies $$\operatorname{Var}(\hat g)\le C_1\delta^2+C_2\operatorname{Var}_{\mathrm{global}}.$$ A scalar "OPD + reward + smoothing" baseline has no analogous connection between its smoothing term and a landscape-smoothness quantity, because its regularizer is not tied to an energy gap. Third, Theorem 1 shows that the energy instantiation and sampling measure are the two degrees of freedom behind reverse KL, forward KL, and a geometric Jensen--Shannon variant; this is a design recipe, not a hand-crafted loss combination. The matched-supervision ablation isolates these differences. EnergyOPD improves over "OPD + scalar rewards + smoothing" by $2.3$ Avg@8 on GSM8K and $1.4$ Avg@8 on MATH, while reducing relative gradient variance from $0.78$ to $0.54$. The scalar baseline receives the same signals and smoothing term, so the remaining gap is attributable to the composition mechanism. **Q4: why is local smoothing invariance desirable?** **A4:** We do not claim that $\tau$ and its perturbed neighbor $\tau'$ are semantically equivalent. Local token replacement can produce invalid intermediate reasoning even when the final answer is preserved, and we agree this makes semantic-consistency regularization questionable if the goal were data augmentation. The goal of $\mathcal{L}_{\mathrm{Smooth}}$ is different: it enforces local Lipschitz continuity of the student energy $$\mathcal{L}_{\mathrm{Smooth}}=\mathbb{E}_{\tau\sim P_S,\tau'\sim\mathcal{N}(\tau)} \|E_S(\tau;\theta)-E_S(\tau';\theta)\|^2.$$ A rugged energy landscape is the geometric source of high-variance policy gradients: small changes in the sampled trajectory produce large changes in the advantage. The regularizer directly shrinks the local energy gap, and Theorem 2 bounds the resulting local variance contribution by $\delta^2$. The invariance being encouraged is not "same probabilities for semantically equivalent responses" but "smooth energy across local rollout variations". This local-perturbation family is a proxy for the rollout drift the student actually encounters during on-policy training, where the sampler changes at every iteration. The perturbation sweep in Appendix H.3 shows the regularization is only beneficial in the appropriate regime: at $\rho=0.05,k=16$, validity is $81\%$ and relative gradient variance is $0.54$; increasing $\rho$ to $0.10$ lowers variance only slightly but drops validity to $62\%$ and degrades accuracy. Thus we do not ask the model to be invariant to arbitrary corruptions; we require the aggregate smoothing signal to be useful, and overly aggressive perturbation is penalized by the data. Finally, the removal ablation is not offered as proof of realistic OOD robustness; it is a falsifiable test of the geometric hypothesis. The stronger, scale-extended smoothing ablation shows the same benefit at 0.6B, 1.7B, and 4B, which distinguishes a broadly useful local-smoothness regularizer from a small-model artifact. --- **Finally,** we thank the reviewer again for the careful critique. We believe the matched-supervision comparison and the multi-scale smoothing ablation provide direct, falsifiable evidence for the two points the reviewer raised, and we hope these updates improve the overall evaluation of our work.Looking forward to reply
Dear Reviewer F45D,
With the discussion window closing soon, we gently bring our response and new experimental results in Appendix H to your attention. The other two reviewers have reviewed our follow-up clarifications, acknowledged our revisions, and updated their scores accordingly.
We hope the controlled OPD comparison and multi-scale smoothing ablations can address your main questions. Feel free to share any remaining concerns, and we are ready to respond promptly.
Best regards,
The Authors
Response to Reviewer F45D [1/2]
Thanks for the valuable and critical feedback. We address the two central questions directly: whether the energy formulation contributes beyond OPD with auxiliary rewards and regularization, and whether the smoothing benefit persists beyond the 1.7B student. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score. Please see our responses below one by one.
Q1: Does EnergyOPD outperform conventional OPD when both receive the same correctness rewards, format penalties, smoothing term, and tuning budget?
A1: We have added exactly this comparison. A conventional scalar-advantage baseline receives the same teacher logits and verifiable correctness reward as EnergyOPD, both with and without the smoothing term, under matched tuning budgets (same learning rate, batch size, rollout budget, and optimization steps). The only difference is the composition mechanism: energy addition versus scalar advantage addition, plus the trajectory-level execution-energy term.
On the held-out benchmarks GSM8K and MATH (full), the closest baseline "OPD + scalar rewards + smoothing" reaches 65.1±1.1 and 27.9±0.7 Avg@8, while EnergyOPD reaches 67.4±0.9 and 29.3±0.6 (Appendix H.1, Table 8). Relative gradient variance drops from 0.78 to 0.54. The structural reason is that correctness and format signals define a tilted target energy, πteαexp(−βEExec+γR), rather than being added locally as token-level advantages, and the smoothing regularizer acts on this composed landscape. This is the core evidence that the energy framing is doing work beyond "OPD + rewards + smoothing."
Table 8. Energy fusion vs. scalar advantage shaping with matched supervision (Qwen3-1.7B-Base, mean ± std over 3 seeds).
Q2: Does the isolated benefit of smoothing persist beyond the 1.7B student? Is it a broadly useful regularizer or an artifact of small-model sensitivity?
A2: We agree that the original component ablation was confined to 1.7B and was therefore insufficient to answer this question. We have extended the smoothing ablation to Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B students (Appendix H.4). Adding smoothing improves Avg@8 by +0.9, +1.2, and +1.2 points, and Pass@8 by +1.4, +1.3, and +1.6 points, respectively, while reducing relative gradient variance by 0.18--0.22 in all three cases. The benefit is therefore not specific to the 1.7B scale; it persists, and the Pass@8 gain is largest at 4B. This distinguishes a broadly useful regularizer from a small-model artifact.
Smoothing gain across student scales (Appendix H.4).
On the deeper question of why local smoothing invariance is desirable: the regularizer does not assume that τ and τ′ are semantically equivalent. It asks the student's energy function to vary smoothly over local perturbations of its own rollouts, because a rugged energy landscape is the geometric cause of high-variance gradients. Theorem 2 makes this precise: the variance bound is proportional to δ2, where δ bounds the smoothing loss. The new perturbation sweep in Appendix H.3 further shows that too-aggressive perturbation reduces semantic validity and hurts accuracy, so we do not ask the model to be invariant to arbitrary corruptions.