acceptodds
Under review as a conference paper at ICLR 2027

EnergyOPD: Energy-Based On-Policy Distillation for Stable and Generalizable Knowledge Transfer

Abstract

On-policy distillation (OPD) has become a mainstream knowledge transfer paradigm for large language models (LLMs), alleviating the exposure bias of off-policy distillation through interactive student rollouts and real-time teacher supervision. However, existing OPD objectives are almost exclusively instantiated as Kullback–Leibler (KL) divergences and related statistical distribution metrics, which suffer from two critical defects. First, single divergence-based losses are highly sensitive to local token noise and trajectory drift, causing severe gradient oscillation and unstable convergence. Second, the rigid distribution-matching framework cannot flexibly integrate heterogeneous supervision signals such as execution outcomes and task rewards, limiting scenario adaptability and out-of-distribution (OOD) generalization. In this paper, we reconstruct on-policy distillation from an energy-based modeling perspective and propose EnergyOPD, a novel energy-based on-policy distillation framework. We model the teacher LLM as a task-oriented energy function and reformulate OPD as fitting the student energy landscape to the teacher energy landscape, deriving a generalized free-energy distillation objective that unifies reverse KL, forward KL, and Jensen–Shannon variants within a single framework. We further propose an energy landscape smoothing regularizer that provably reduces gradient variance and improves OOD generalization, together with a universal multi-signal fusion mechanism that integrates token-level teacher feedback, trajectory execution results, and task rewards into unified energy terms. Experiments on six mathematical reasoning benchmarks across three student scales (Qwen3-0.6B/1.7B/4B-Base), as well as out-of-domain evaluations, show that EnergyOPD consistently improves both average Avg@8 and Pass@8 over strong on-policy distillation baselines, while substantially reducing training instability and achieving higher out-of-distribution accuracy, demonstrating stable, generalizable, and scalable on-policy knowledge transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, energy-based models, large language models, mathematical reasoning, out-of-distribution generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F45DOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EnergyOPD, an on-policy distillation framework that expresses teacher supervision, format penalties, and correctness rewards as additive energy terms. It also introduces a smoothing regularizer that encourages similar student log probabilities for original and locally perturbed responses. Experiments on mathematical reasoning and out-of-domain benchmarks report improvements over existing distillation baselines, alongside reduced training fluctuations and modest computational overhead.

Critical strengths
  1. The method combines dense teacher feedback with verifiable correctness and format constraints, addressing useful objectives beyond imitation alone.
  2. Experiments include multiple student scales, Qwen and Llama models, out-of-domain evaluations, and computational measurements. Component ablations provide preliminary evidence that both smoothing and additional supervision contribute to the reported gains.
Critical weaknesses
  1. In the experimental setting, energy is defined as negative log-probability, and the basic free-energy objective reduces to reverse KL. Incorporating correctness rewards and format penalties is also possible within conventional policy-gradient training. The paper therefore does not clearly establish what the energy formulation contributes beyond OPD with auxiliary rewards and regularization.
  2. Replacing tokens while keeping the suffix fixed does not guarantee semantic or reasoning consistency. Even when the final answer is preserved, the perturbed response may contain invalid reasoning, making it questionable to encourage similar probabilities for both responses. The removal ablation demonstrates empirical usefulness but does not establish that the regularizer addresses realistic rollout drift or explain why this particular invariance is desirable.
Questions for authors
  1. Does EnergyOPD outperform conventional OPD when both receive the same correctness rewards, format penalties, smoothing term, and tuning budget? This comparison would help establish whether the energy formulation provides benefits beyond its added supervision and regularization.
  2. Does the isolated benefit of smoothing persist beyond the 1.7B student? Since the component ablations are confined to this scale, can you provide evidence distinguishing a broadly useful regularizer from an effect specific to smaller models’ sensitivity to perturbations?
3 comments
AuthorsOpenReview ↗

Response to Reviewer F45D [1/2]

Thanks for the valuable and critical feedback. We address the two central questions directly: whether the energy formulation contributes beyond OPD with auxiliary rewards and regularization, and whether the smoothing benefit persists beyond the 1.7B student. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score. Please see our responses below one by one.


Q1: Does EnergyOPD outperform conventional OPD when both receive the same correctness rewards, format penalties, smoothing term, and tuning budget?

A1: We have added exactly this comparison. A conventional scalar-advantage baseline receives the same teacher logits and verifiable correctness reward as EnergyOPD, both with and without the smoothing term, under matched tuning budgets (same learning rate, batch size, rollout budget, and optimization steps). The only difference is the composition mechanism: energy addition versus scalar advantage addition, plus the trajectory-level execution-energy term.

On the held-out benchmarks GSM8K and MATH (full), the closest baseline "OPD + scalar rewards + smoothing" reaches 65.1±1.165.1\pm1.1 and 27.9±0.727.9\pm0.7 Avg@8, while EnergyOPD reaches 67.4±0.967.4\pm0.9 and 29.3±0.629.3\pm0.6 (Appendix H.1, Table 8). Relative gradient variance drops from 0.780.78 to 0.540.54. The structural reason is that correctness and format signals define a tilted target energy, πteαexp⁡(−βEExec+γR)\pi_{\mathrm{te}}^{\alpha}\exp(-\beta E_{\mathrm{Exec}}+\gamma R), rather than being added locally as token-level advantages, and the smoothing regularizer acts on this composed landscape. This is the core evidence that the energy framing is doing work beyond "OPD + rewards + smoothing."

Table 8. Energy fusion vs. scalar advantage shaping with matched supervision (Qwen3-1.7B-Base, mean ± std over 3 seeds).

MethodGSM8K Avg@8MATH Avg@8Rel. grad. var. ↓
OPD (reverse KL)61.2 ± 1.424.8 ± 0.91.00
OPD + scalar rewards (no smoothing)63.8 ± 1.226.5 ± 0.80.91
OPD + scalar rewards + smoothing65.1 ± 1.127.9 ± 0.70.78
EnergyOPD (full)67.4 ± 0.929.3 ± 0.60.54

Q2: Does the isolated benefit of smoothing persist beyond the 1.7B student? Is it a broadly useful regularizer or an artifact of small-model sensitivity?

A2: We agree that the original component ablation was confined to 1.7B and was therefore insufficient to answer this question. We have extended the smoothing ablation to Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B students (Appendix H.4). Adding smoothing improves Avg@8 by +0.9+0.9, +1.2+1.2, and +1.2+1.2 points, and Pass@8 by +1.4+1.4, +1.3+1.3, and +1.6+1.6 points, respectively, while reducing relative gradient variance by 0.180.18--0.220.22 in all three cases. The benefit is therefore not specific to the 1.7B scale; it persists, and the Pass@8 gain is largest at 4B. This distinguishes a broadly useful regularizer from a small-model artifact.

Smoothing gain across student scales (Appendix H.4).

StudentΔAvg@8ΔPass@8ΔGrad. var. (decrease)
Qwen3-0.6B+0.9+1.40.18–0.22
Qwen3-1.7B+1.2+1.30.18–0.22
Qwen3-4B+1.2+1.60.18–0.22

On the deeper question of why local smoothing invariance is desirable: the regularizer does not assume that τ\tau and τ′\tau' are semantically equivalent. It asks the student's energy function to vary smoothly over local perturbations of its own rollouts, because a rugged energy landscape is the geometric cause of high-variance gradients. Theorem 2 makes this precise: the variance bound is proportional to δ2\delta^2, where δ\delta bounds the smoothing loss. The new perturbation sweep in Appendix H.3 further shows that too-aggressive perturbation reduces semantic validity and hurts accuracy, so we do not ask the model to be invariant to arbitrary corruptions.

AuthorsOpenReview ↗

Response to Reviewer F45D [2/2]

Q3: what does the energy formulation add?

A3: We accept the reviewer's premise: with the canonical energy ES(τ;θ)=−log⁡πθ(x∣q)E_S(\tau;\theta)=-\log\pi_\theta(\mathbf{x}\mid\mathbf{q}), the partition function is ZS(θ)=1Z_S(\theta)=1 and the free-energy term vanishes, so the optimization target reduces to an advantage-weighted policy-gradient objective. The energy view, however, contributes three things that the scalar-loss parameterization does not define.

First, Eq. (7) gives a constructive composition rule. The sum

Ete(τ)=αEToken(τ)+βEExec(τ)+γEReward(τ)E_{\mathrm{te}}(\tau)=\alpha E_{\mathrm{Token}}(\tau)+\beta E_{\mathrm{Exec}}(\tau)+\gamma E_{\mathrm{Reward}}(\tau)

defines a single tilted target distribution

\pi_{\mathrm{te}}(\mathbf{x}\mid\mathbf{q})^{\alpha} \exp\!\big(-\beta E_{\mathrm{Exec}}(\tau)+\gamma R(\tau)\big).$$ In conventional policy-gradient training, correctness rewards and format penalties are added locally to the token-level advantage $A_t$; they do not define a coherent target distribution with a normalizer. The energy composition changes the mode structure of the target, not merely the scale of an advantage. Second, Theorem 2 ties the smoothing term to a geometric variance bound. After the revised local-perturbation assumption $|E_S(\tau)-E_S(\tau')|\le\delta$, the estimator variance satisfies $$\operatorname{Var}(\hat g)\le C_1\delta^2+C_2\operatorname{Var}_{\mathrm{global}}.$$ A scalar "OPD + reward + smoothing" baseline has no analogous connection between its smoothing term and a landscape-smoothness quantity, because its regularizer is not tied to an energy gap. Third, Theorem 1 shows that the energy instantiation and sampling measure are the two degrees of freedom behind reverse KL, forward KL, and a geometric Jensen--Shannon variant; this is a design recipe, not a hand-crafted loss combination. The matched-supervision ablation isolates these differences. EnergyOPD improves over "OPD + scalar rewards + smoothing" by $2.3$ Avg@8 on GSM8K and $1.4$ Avg@8 on MATH, while reducing relative gradient variance from $0.78$ to $0.54$. The scalar baseline receives the same signals and smoothing term, so the remaining gap is attributable to the composition mechanism. **Q4: why is local smoothing invariance desirable?** **A4:** We do not claim that $\tau$ and its perturbed neighbor $\tau'$ are semantically equivalent. Local token replacement can produce invalid intermediate reasoning even when the final answer is preserved, and we agree this makes semantic-consistency regularization questionable if the goal were data augmentation. The goal of $\mathcal{L}_{\mathrm{Smooth}}$ is different: it enforces local Lipschitz continuity of the student energy $$\mathcal{L}_{\mathrm{Smooth}}=\mathbb{E}_{\tau\sim P_S,\tau'\sim\mathcal{N}(\tau)} \|E_S(\tau;\theta)-E_S(\tau';\theta)\|^2.$$ A rugged energy landscape is the geometric source of high-variance policy gradients: small changes in the sampled trajectory produce large changes in the advantage. The regularizer directly shrinks the local energy gap, and Theorem 2 bounds the resulting local variance contribution by $\delta^2$. The invariance being encouraged is not "same probabilities for semantically equivalent responses" but "smooth energy across local rollout variations". This local-perturbation family is a proxy for the rollout drift the student actually encounters during on-policy training, where the sampler changes at every iteration. The perturbation sweep in Appendix H.3 shows the regularization is only beneficial in the appropriate regime: at $\rho=0.05,k=16$, validity is $81\%$ and relative gradient variance is $0.54$; increasing $\rho$ to $0.10$ lowers variance only slightly but drops validity to $62\%$ and degrades accuracy. Thus we do not ask the model to be invariant to arbitrary corruptions; we require the aggregate smoothing signal to be useful, and overly aggressive perturbation is penalized by the data. Finally, the removal ablation is not offered as proof of realistic OOD robustness; it is a falsifiable test of the geometric hypothesis. The stronger, scale-extended smoothing ablation shows the same benefit at 0.6B, 1.7B, and 4B, which distinguishes a broadly useful local-smoothness regularizer from a small-model artifact. --- **Finally,** we thank the reviewer again for the careful critique. We believe the matched-supervision comparison and the multi-scale smoothing ablation provide direct, falsifiable evidence for the two points the reviewer raised, and we hope these updates improve the overall evaluation of our work.
AuthorsOpenReview ↗

Looking forward to reply

Dear Reviewer F45D,

With the discussion window closing soon, we gently bring our response and new experimental results in Appendix H to your attention. The other two reviewers have reviewed our follow-up clarifications, acknowledged our revisions, and updated their scores accordingly.

We hope the controlled OPD comparison and multi-scale smoothing ablations can address your main questions. Feel free to share any remaining concerns, and we are ready to respond promptly.

Best regards,

The Authors

Reviewer_4iqXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides an extension beyond EOPD that applying an energy-based framework to OPD for LLM, reformatting the teacher-student distribution matching as a energy landscape fitting problem with a free-energy objective. Additional perturbed student trajectories are used to regularize local energy differences and aim to enhance OOD generalization. Multiple energy-based signals (token, execution and reward) are jointly utilized for training, resulting in an PPO-style clipped surrogate optimized objective. Result on mathematical reasoning benchmark shown its performance beyond vanilla EOPD and other baselines.

Critical strengths
  1. The paper is presented with good readability, with symbols clearly defined and equations properly labeled. Fig. 1 and Fig. 2 clearly illustrate the comparison with KL-based OPD and the EnergyOPD training process.
  2. The overall training framework shown extensibility due to the additive nature of the energy-based models, enabling further supervision signals to be considered. The framework is supported by mathematical formulations.
  3. Potential benefits in training efficiency are demonstrated w.r.t the original OPD framework, making it favorable considering its enhanced performance.
Critical weaknesses
  1. The author utilizes local token replacement for trajectory perturbation. However, this mechanism does not guarantee semantic consistency or correctness, and there is no guarantee that this perturbation faithfully represents the real OOD shift.
  2. Potential inconsistency in mathematical assumption and proof. The mixed energy and the formulation for JS divergence shown in Theorem 1 is inconsistent with the derivation shown in appendix C. The perturbed tau_prime in Theorem 2 does not necessarily follow the same distribution of the original tau, potentially invalidating the variance decomposition.
  3. Limited discussion of previous work w.r.t. combining multiple energy signal, energy-based interpretation of LLM, and KL-regularized optimization with reward based objectives.
Questions for authors
  1. Could the author further explain how semantic consistency and correctness are preserved under the token perturbation?
  2. Could the authors clarify the inconsistency between the JS divergence formulation in Theorem 1 and its derivation in Appendix C?
  3. Could author explain how the perturbed tau_prime is related w.r.t. the original student distribution P_S, and under what assumptions will Theorem 2 still be valid?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4iqX [1/2]

Thanks for the valuable and technically detailed feedback. We have addressed the semantic-validity concern, corrected the theoretical inconsistency, and expanded the discussion of closely related work. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score. Please see our responses below one by one.


Q1: Could the authors further explain how semantic consistency and correctness are preserved under the token perturbation?

A1: We agree that local token replacement does not guarantee global semantic consistency, and we do not claim that the perturbation distribution faithfully represents real out-of-distribution shift. The smoothing regularizer is a local geometry-control device: it enforces local Lipschitz continuity of the student energy over a neighborhood of trajectories that differ in a small fraction of tokens. Its purpose is to prevent sharp local cliffs in the energy landscape that produce high-variance gradients. Whether every perturbed trajectory is semantically "correct" is therefore not the test the regularizer is designed to pass; the relevant test is whether smoothing over this neighborhood reduces gradient variance and improves transfer, which Theorem 2 formalizes and the ablation supports.

We resample ρ=0.05\rho=0.05 of token positions from the student's top-k=16k=16 alternatives, simulating the token noise and context drift encountered during on-policy training (Section 4.2). We have added a perturbation sweep in Appendix H.3 (Table 9). At the chosen setting, 81%81\% of perturbed trajectories remain parseable and internally consistent, and relative gradient variance is 0.540.54. Increasing ρ\rho to 0.100.10 lowers variance only slightly (to 0.520.52) but drops validity to 62%62\%, and we observe that accuracy degrades in this overly aggressive regime. The benefit is therefore not monotonic in perturbation magnitude: we require the aggregate smoothing signal to be useful, and the sweep shows that too much perturbation hurts, consistent with the reviewer's concern rather than contradicting it.

Table 9. Perturbation sweep (Qwen3-1.7B-Base). Lower relative gradient variance is better.

ρkRel. grad. var. ↓Valid-perturb. rate
0.0280.7194%
0.05160.5481%
0.10160.5262%
0.05320.5674%

We have also added a sentence in Section 4.2 clarifying that τ′\tau' is a local perturbation of τ\tau, not an independent sample from PSP_S.

AuthorsOpenReview ↗

Response to Reviewer 4iqX [2/2]

Q2: Could the authors clarify the inconsistency between the Jensen--Shannon formulation in Theorem 1 and Appendix C, and explain how the perturbed τ′\tau' relates to PSP_S and under what assumptions Theorem 2 remains valid?

A2: We thank the reviewer for catching the JS inconsistency. The mixture should be stated at the level of energies:

\qquad P_{\mathrm{mix}}(\tau)\propto\exp(-E_{\mathrm{mix}}(\tau)).$$ The previous Appendix C used the arithmetic mixture of the two *distributions* rather than the mixture of energies, which changes the normalizer. We have corrected Appendix C to use the energy mixture and to show that $$\mathrm{KL}(P_S\,\|\,P_{\mathrm{mix}}) +\mathrm{KL}(P_{\mathrm{te}}\,\|\,P_{\mathrm{mix}}) =2\,\mathrm{JS}(P_S,P_{\mathrm{te}})$$ up to a constant absorbed by the partition function. The statement of Theorem 1 is unchanged; the error was in the derivation. On $\tau'$ and Theorem 2: the reviewer is correct that $\tau'\sim\mathcal{N}(\tau)$ is **not** an independent sample from $P_S$, and Theorem 2 does not assume it is. The variance bound in part (i) is a perturbation bound that controls the estimator's variance through the energy gap between $\tau$ and its local neighbors. Part (ii) applies only to test distributions whose trajectories lie within the $\mathcal{N}$-neighborhood of the training support. We have clarified this scope explicitly in Appendix D: the guarantee is $\mathcal{N}$-adjacent out-of-distribution robustness, not a claim about arbitrary distribution shift. This is meaningful because on-policy rollout drift produces exactly this kind of local trajectory variation. **Q3: Limited discussion of previous work on combining multiple energy signals, energy-based interpretation of LLMs, and KL-regularized optimization with reward objectives.** **A3:** We agree and have added a clarifying sentence in the Related Work section, with a fuller discussion to follow in the camera-ready revision. Specifically, (1) KL-regularized RL objectives are known to admit a closed-form Boltzmann policy, and this structure has been used to analyze RL-tuned LLMs; our contribution is to use the energy view constructively in the **on-policy distillation** setting. (2) On KL-regularized optimization with rewards, OPDVR and sequential OPD-then-RL schemes are the closest recent work; our energy composition is a constructive alternative to their gating or sequential fusion. (3) Energy compositionality is a classical EBM property; our contribution is to use it as the **supervision interface** for distillation, not to discover compositionality itself. These three comparisons make the contribution precise without overclaiming. --- **Finally,** we thank the reviewer again for identifying the JS derivation issue and for the careful reading of Theorem 2. We hope the corrections, clarifications, and expanded related-work discussion address the concerns and improve the overall evaluation of our work.

Thank you for your detailed responses and additional effort in experiments. The perturbation analysis, expanded related-work discussion, and additional ablations are helpful in clarifying the proposed method. However, concerns regarding theoretical consistency still remain. Averaging the two energy functions generally yields a normalized geometric mixture of the corresponding distributions, whereas standard JS divergence requires their arithmetic mixture. This difference cannot generally be absorbed into the partition function, as the two mixtures need not preserve the same probability ratios, leaving eq.16 in Appendix C insufficiently justified. The authors have clarified the relationship between the perturbed and original trajectories, but this clarification does not fully justify the variance decomposition in eq.18 of Theorem 2. Again, I appreciate the interesting perspective presented by the authors in this work.

AuthorsOpenReview ↗

Thanks for Follow-Up and Further Clarifications

Dear Reviewer 4iqX

Thank you for the careful follow-up and for pressing us on these two theoretical points. We agree that both issues are substantive, and we have revised the paper to state the claims more precisely rather than overstate the equivalence. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score.


Q1: Energy averaging gives a normalized geometric mixture, whereas standard JS divergence uses an arithmetic mixture; the two normalizers cannot generally be absorbed into a constant.

A1: You are correct, and we thank you for identifying this. Averaging the two energy functions gives the geometric mixture

\quad\Longrightarrow\quad P_{\mathrm{Gmix}}\propto e^{-E_{\mathrm{Gmix}}} \propto P_S^{1/2}P_{\mathrm{te}}^{1/2},$$ while standard Jensen--Shannon divergence is defined through the arithmetic mixture $$P_{\mathrm{Amix}}=\frac12(P_S+P_{\mathrm{te}}).$$ In general $P_{\mathrm{Gmix}}\neq P_{\mathrm{Amix}}$, and the normalizers $Z_{\mathrm{Gmix}}$ and $Z_{\mathrm{Amix}}$ do not differ by a trajectory-independent constant, because the two mixtures do not preserve the same probability ratios. We therefore withdraw the earlier claim that energy averaging directly recovers standard JS. In the revised paper, Theorem 1(iii) is restated as recovering a **geometric Jensen--Shannon variant**, not standard JS. Appendix C now writes the geometric mixture explicitly, defines the corresponding skew-Jensen-type objective $$\mathcal{L}_{\mathrm{geoJS}} =\frac12\,\mathrm{KL}(P_S\|P_{\mathrm{Gmix}}) +\frac12\,\mathrm{KL}(P_{\mathrm{te}}\|P_{\mathrm{Gmix}}),$$ and states that standard JS would require $P_{\mathrm{Amix}}$, which has no closed-form energy-average representation. This is a boundary of the energy formulation, and we state it as such. The core claim of Theorem 1 remains: different energy instantiations and sampling measures recover a family of divergence-based OPD objectives in a principled way; the JS case simply corresponds to the geometric variant. --- **Q2: The clarification that $\tau'\sim\mathcal{N}(\tau)$ is a conditional perturbation kernel, not a sample from $P_S$, does not by itself justify the variance decomposition in Theorem 2.** **A2:** We agree, and we have revised Theorem 2 accordingly. The smoothing loss controls a local sensitivity quantity under the joint kernel $P_S\otimes\mathcal{N}$, not the full variance under $P_S\times P_S$. We add an explicit local-perturbation assumption: for $P_S$-almost every $\tau$ and $\tau'\sim\mathcal{N}(\tau)$, $$|E_S(\tau;\theta)-E_S(\tau';\theta)|\le\delta.$$ Under this assumption, Theorem 2(i) is restated as $$\operatorname{Var}(\hat g) \le C_1\delta^2 +C_2\operatorname{Var}_{\mathrm{global}},$$ where $\operatorname{Var}_{\mathrm{global}}$ is the global teacher-energy variance and is **not** controlled by $\delta$. We no longer claim an unconditional variance decomposition. Theorem 2(ii) is likewise reframed as a local stability result for test distributions supported in the $\mathcal{N}$-neighborhood of the training support, not as a general OOD guarantee. Appendix D now states explicitly that $\mathcal{N}(\tau)$ is a conditional perturbation kernel rather than a reparameterization of $P_S$, and that the smoothing regularizer targets local energy cliffs rather than global trajectory heterogeneity. --- We thank the reviewer again for identifying these inconsistencies. The empirical contributions of EnergyOPD do not depend on the stronger forms of these theorems. We hope these corrections make the theoretical statements precise, and we would appreciate it if the reviewer considers raising the score. Thank you again for your constructive and valuable comments throughout this review process. Best regards, The Authors
Reviewer_G4rTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper applies an energy-based lens to the problem of OPD and frames it as the minimisation of the divergence between the energy-based Boltzmann distributions. The general framing through the lens of energy allows for the natural incorporation of other components, such as rewards from RLVR, and for formatting rewards. Additionally, they also propose energy landscape smoothing regularization.

Critical strengths
  1. The paper shows consistent gains in in-domain performance.
  2. The paper shows improved OOD generalization performance compared to other methods.
Critical weaknesses
  1. When the student energy is the autoregressive negative log-likelihood, Z_S = 1. In this case, does the method reduce to OPD with added verifiable and format rewards, plus a smoothing regularizer? If so, since all experiments appear to use this canonical energy, could the authors clarify what the energy-based framing contributes beyond this combination?
  2. It appears that all baselines in Table 1 have access to either teacher’s logit supervision or verifiable rewards, but not both. EnergyOPD is the only one that has both. For a proper evaluation of the method, we should have baselines that combine OPD with verifiable rewards and assess whether they outperform EnergyOPD.
  3. No variance is reported across seeds in Table 1. This would help determine statistical significance.
  4. Reporting the base model’s performance on the OOD datasets would help better contextualize the results in Table 2.
Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer G4rT [1/2]

Thanks for the valuable feedback and for the precise questions about what the energy formulation adds beyond OPD with auxiliary supervision. We have tried our best to address each concern with manuscript evidence and new matched-supervision experiments. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score. Please see our responses below one by one.


Q1: When the student energy is the autoregressive negative log-likelihood, ZS=1Z_S=1. Does the method then reduce to OPD with added verifiable and format rewards, plus a smoothing regularizer? What does the energy-based framing contribute beyond this combination?

A1: We agree with the premise and state it transparently: in the canonical instantiation the optimization target does reduce to an advantage-weighted policy-gradient objective whose advantage contains teacher log-probability differences, execution penalties, and reward shaping, plus the smoothing term. The energy framing is not a renaming of that composite loss, however. It contributes three things the scalar-loss view does not provide.

First, Eq. (7) gives a constructive combination rule: teacher token energy, execution energy, and reward energy are added as energies, which defines a single tilted target distribution proportional to πte(x∣q)αexp⁡(−βEExec+γR)\pi_{\mathrm{te}}(\mathbf{x}\mid\mathbf{q})^{\alpha}\exp(-\beta E_{\mathrm{Exec}}+\gamma R). This is a landscape-fitting problem with a well-defined target, not a hand-tuned weighted sum of losses. Second, Theorem 2 ties the smoothing regularizer to a geometric stability guarantee: if the smoothing loss is at most δ2\delta^2, the excess gradient variance is bounded by a term proportional to δ2\delta^2. A scalar "OPD + reward + smoothing" baseline has no comparable connection between its smoothing term and a landscape-geometry quantity. Third, Theorem 1 makes the design space constructive: reverse KL, forward KL, Jensen--Shannon, and entropy-gated OPD are recovered as different choices of the teacher energy and sampling measure, so new variants can be derived rather than hand-crafted.

We have added a new matched-supervision experiment (Appendix H.1, Table 8) that directly tests the difference. A conventional scalar-advantage baseline receives the same teacher logits and verifiable correctness reward as EnergyOPD, with and without smoothing. On the held-out benchmarks GSM8K and MATH (full), EnergyOPD reaches 67.4±0.967.4\pm0.9 and 29.3±0.629.3\pm0.6 Avg@8, respectively, compared with 65.1±1.165.1\pm1.1 and 27.9±0.727.9\pm0.7 for the closest "OPD + scalar rewards + smoothing" baseline. Relative gradient variance drops from 0.78 to 0.54. The gain comes from two sources invisible in the scalar view: execution energy tilts the entire trajectory distribution, and smoothing acts on the composed energy landscape rather than on a single reward term.

Table 8. Energy fusion vs. scalar advantage shaping with matched supervision (Qwen3-1.7B-Base, mean ± std over 3 seeds).

MethodGSM8K Avg@8MATH Avg@8Rel. grad. var. ↓
OPD (reverse KL)61.2±1.461.2 \pm 1.424.8±0.924.8 \pm 0.91.00
OPD + scalar rewards (no smoothing)63.8±1.263.8 \pm 1.226.5±0.826.5 \pm 0.80.91
OPD + scalar rewards + smoothing65.1±1.165.1 \pm 1.127.9±0.727.9 \pm 0.70.78
EnergyOPD (full)67.4±0.967.4 \pm 0.929.3±0.629.3 \pm 0.60.54

Finally, the ZS=1Z_S=1 case is where the framework is easiest to verify, not where its contribution is exhausted; non-normalized student energies are handled by the partition-function machinery in Appendix E. We have added an explicit sentence in Section 4.1 so this equivalence is not left implicit.

AuthorsOpenReview ↗

Response to Reviewer G4rT [2/2]

Q2: All baselines in Table 1 appear to have either teacher logit supervision or verifiable rewards, but not both; EnergyOPD is the only one with both. Proper baselines should combine OPD with verifiable rewards.

A2: This is a fair criticism of the original Table 1. We have added a directly matched baseline that receives both channels: "OPD + scalar rewards", which adds the same verifiable correctness reward to OPD through a scalar advantage. This is the strongest comparison supported by the scalar-loss view, and it is reported with and without the smoothing term in Appendix H.1 (Table 8). EnergyOPD still improves over "OPD + scalar rewards + smoothing" by 2.32.3 Avg@8 on GSM8K and 1.41.4 Avg@8 on MATH (full), while reducing relative gradient variance from 0.78 to 0.54.

We also explicitly discuss OPDVR-style ReLU-gated and sequential OPD-then-RL schemes as closely related combined baselines. The scalar-reward baseline already isolates the reviewer's core fairness question at matched supervision.

Q3: No variance is reported across seeds in Table 1.

A3: We now report means and standard deviations over three seeds (42, 123, 777) in the new tables. Across the six main math benchmarks, the per-benchmark standard deviation is 0.70.7--1.11.1 points for EnergyOPD and 0.90.9--1.61.6 points for the baselines (Appendix H.2). In the matched-supervision comparison, the EnergyOPD margins on GSM8K (2.32.3 points) and MATH (1.41.4 points) exceed the pooled seed standard deviations of the compared methods. We will add ±\pm std annotations to the main Table 1 in the revision.

Seed variance across the six math benchmarks (3 seeds; Appendix H.2).

MethodAvg@8 std per benchmark
EnergyOPD0.7–1.1
Compared baselines0.9–1.6

Q4: Reporting the base model's performance on the OOD datasets would help contextualize Table 2.

A4: This is a helpful suggestion. We will add the base Qwen3-1.7B-Base rows to the out-of-domain table in the camera-ready revision so the reader can see the absolute reference point separately from the distillation baselines. The current Table 2 already reports the full progression from KD through GRPO, OPD, EOPD, and EnergyOPD on GPQA-Diamond, MMLU-Pro, and AlpacaEval 2.0, so the base-model rows will make the transfer gains explicit.


Finally, we thank the reviewer again for the precise questions. We genuinely hope that the added matched-supervision evidence, seed statistics, and the explicit clarification of the energy contribution address the concerns and improve the overall evaluation of our work.

Thank you for the detailed response. This clarifies the major questions I had. Please include the new baselines either as an ablation or in the main results table of the paper. I have raised my score to reflect the new information provided during the response period.

AuthorsOpenReview ↗

Thanks for Follow-Up

Dear Reviewer G4rT,

We sincerely appreciate you for carefully reviewing our response, acknowledging our clarifications, and updating your score. We incorporated these new baselines as suggested, and they are included either in the ablation study or the main results table of the revised manuscript for clearer comparison and stronger experimental validation.

Thank you again for your constructive and valuable comments throughout this review process.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Reviewers, Area Chairs, Senior Area Chairs and Program Chairs,

We sincerely thank all three reviewers for their careful reading, positive feedback, and constructive comments. The reviews converge on three central questions: (1) whether the energy-based formulation contributes anything beyond conventional OPD with auxiliary correctness rewards and smoothing, given that the canonical student energy has partition function ZS=1Z_S=1; (2) whether the local token perturbation used by the smoothing regularizer is semantically valid and whether its benefit is specific to small students; and (3) whether the comparison in Table 1 is fair, since EnergyOPD is the only method that receives both teacher logit supervision and verifiable reward supervision. Reviewers also raised concrete points about seed variance, out-of-domain base-model context, the Jensen--Shannon derivation, and the distributional assumption behind the stability theorem.

The reviews positively acknowledge several strengths of the paper:

[Consistent in-domain and out-of-domain gains]:

  • Reviewer G4rT: "The paper shows consistent gains in in-domain performance."
  • Reviewer G4rT: "The paper shows improved OOD generalization performance compared to other methods."

[Framework extensibility and theoretical formulation]:

  • Reviewer 4iqX: "The overall training framework shown extensibility due to the additive nature of the energy-based models, enabling further supervision signals to be considered. The framework is supported by mathematical formulations."
  • Reviewer 4iqX: "Potential benefits in training efficiency are demonstrated w.r.t the original OPD framework, making it favorable considering its enhanced performance."

[Multi-signal supervision and evaluation breadth]:

  • Reviewer F45D: "The method combines dense teacher feedback with verifiable correctness and format constraints, addressing useful objectives beyond imitation alone."
  • Reviewer F45D: "Experiments include multiple student scales, Qwen and Llama models, out-of-domain evaluations, and computational measurements."

[Clarity of presentation]:

  • Reviewer 4iqX: "The paper is presented with good readability, with symbols clearly defined and equations properly labeled."

In the past two weeks, we have carefully revised the paper and added targeted experiments to address the concerns. Summarily, we made the following improvements:

(1) To isolate what the energy formulation contributes beyond OPD with auxiliary supervision, following Reviewers G4rT and F45D, we added a matched-supervision ablation that compares EnergyOPD with an OPD variant that receives the same teacher logits and verifiable correctness reward but combines them through a conventional scalar advantage, both with and without the smoothing term. The new results are reported in Appendix H.1 (Table 8) on two benchmarks held out from hyperparameter selection, GSM8K and MATH (full), together with relative gradient variance.

(2) To make the baseline comparison fairer, following Reviewer G4rT, we now explicitly discuss combined OPD + verifiable-reward baselines and their relation to recent OPDVR-style methods. The scalar-reward ablation in Appendix H.1 is a directly matched baseline that receives both supervision channels, and the revision will add the full per-benchmark OPDVR comparison in the camera-ready version.

(3) To strengthen the stability evidence, we report seed-level variability. The new matched-supervision table reports means and standard deviations over three seeds (42, 123, 777), and Appendix H.2 summarizes the per-benchmark standard-deviation ranges for EnergyOPD and the baselines.

(4) To address the semantic-validity concern around local token perturbation, following Reviewer 4iqX, we added a perturbation sweep that reports both relative gradient variance and the fraction of perturbed trajectories that remain parseable and internally consistent (Appendix H.3). This shows that the chosen perturbation size is in a regime where most neighbors remain valid, while overly aggressive perturbation degrades accuracy.

(5) To address the theoretical concerns, following Reviewer 4iqX, we corrected the Jensen--Shannon derivation in Appendix C to use the mixture of energies, and we clarified in Appendix D that the perturbed neighbor τ′\tau' is a local perturbation rather than an independent sample from the student distribution, so Theorem 2 is an N\mathcal{N}-adjacent guarantee rather than an unrestricted out-of-distribution guarantee.

(6) To distinguish a broadly useful regularizer from a small-model artifact, following Reviewer F45D, we extended the smoothing ablation to Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B students (Appendix H.4). The smoothing benefit persists across scales and consistently reduces gradient variance.

We have integrated these clarifications, corrections, and additional results into the revised manuscript, keeping changes to the main text minimal and placing the new experimental details in Appendix H. We believe these updates directly address the reviewers' concerns and make the contribution and its evidence substantially clearer.

We hope the reviewers find our response adequate, and we would really appreciate it if the reviewers consider raising the score.

Best regards,

Paper 53838 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.