- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies on-policy distillation by considering the combination of forward and reverse KL divergence objectives between teacher and student distributions. It introduces a matrix representation of the entropy and cross-entropy terms, from which the two KL divergences and their gradients are derived. The authors then use properties of these gradients to motivate an adaptive combination, where the relative contributions of forward and reverse KL are determined based on the distributions' entropies and a gradient-bargaining formulation. The proposed method is evaluated experimentally on language-model distillation tasks, demonstrating improvements over several competing approaches.
Critical strengths
-
The proposed approach for adaptively combining forward and reverse KL gradients, rather than relying on fixed weighting, offers a potentially useful way to balance their complementary properties.
-
The paper provides encouraging experimental results supporting the proposed approach, including comparisons with alternative distillation methods and ablation studies.
Critical weaknesses
-
I found the paper relatively difficult to follow, particularly the transition from the theoretical formulation to the proposed algorithm. The exposition places considerable emphasis on the entropy matrix representation and the resulting mathematical properties, while the motivation for some of the methodological choices is less clear.
-
It is not entirely clear how the observations regarding the relationship between forward and reverse KL gradients in Section 4 motivate the specific adaptive weighting and gradient-coordination rules proposed in Section 5. In particular, some of the results appear to follow from established properties of KL divergence and information geometry.
Questions for authors
-
Please explain more explicitly how the theoretical analysis in Sections 3–4 motivates the particular gradient-coordination procedure proposed in Section 5. In particular, which aspects of the algorithm follow directly from the theoretical results, and which are additional design choices?
-
Please clarify the advantages of the proposed entropy-dependent weighting and gradient-bargaining formulation compared with simpler approaches for combining forward and reverse KL gradients. To what extent do the experimental results isolate the contribution of these particular design choices?
-
I suspect that there is a sign issue in the optimization formulation. In Eq. (6), the gradients are defined with respect to the KL objectives, while Eq. (16) introduces a negative sign in the surrogate loss. How does minimizing this surrogate produce the intended descent direction for the KL objectives?
4 comments
Question 3:
We use , , and to denote loss-gradient signals, so the corresponding descent update carries a minus sign. With this convention, Eq. (18) and Algorithm 1, line 16, use the positive-sign surrogate
Its coefficients are signed gradient components. Both KL logit gradients sum to zero, so their coordinated combination satisfies
Using , we obtain
Thus the surrogate produces exactly the intended logit-gradient signal, as derived in Appendix H.1. Appendix H.2 separately examines the changes induced by actual shared-parameter optimizer updates.
We have made the sign convention explicit in Section 5.2: the signed surrogate has logit gradient , and the corresponding direct logit descent step is . The formulas in Eq. (18), Algorithm 1, and Appendix H.1 use this same convention.
Weakness 1:
We have clarified the progression from the two conditional KL corrections to their coordination and update strength in Sections 4–5. The response to Question 1 explains this connection and identifies the role of each stage in Algorithm 1
Weakness 2:
The response to Question 1 distinguishes the compatibility result from the controller's design choices. Question 2 explains the advantages over directly adding the two KL losses and reports controlled ablations of coordination and preference adaptation.
Dear Reviewer,
Thank you so much for your thoughtful initial reviews!
With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.
Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!
Warm regards
I thank the authors for their detailed response and the additional clarifications. I appreciate the explanation of how the theoretical observations motivate different stages of the proposed algorithm, the additional ablation results, and the clarification regarding the sign convention in the optimization.
While these responses address some of my specific questions, my main concerns remain. In particular, I still find the presentation rather difficult to follow, and it is still not entirely convincing that the extensive theoretical development provides sufficiently strong insights into the particular algorithmic choices made in the proposed method. I have to admit that the connection between the theoretical analysis and the methodological contribution remains somewhat less compelling to me. This, I believe that my original assessment still stands, and I will maintain my recommendation.
We thank the reviewer for the constructive feedback and address the concerns below.
Question 1:
The analysis connects two complementary corrections to a feasible coordinated update. At each student-generated prefix, the source-centered contrasts in Eq. (4) recover reverse KL, R=DKL(q∥p), and forward KL, F=DKL(p∥q). Their logit gradients in Eq. (6) explain their different roles: RKL refines the student's current probability allocation, while FKL retains a direct correction toward teacher-supported candidates that the student underrepresents. For a fixed positive teacher distribution and qj→0, gR,j→0, whereas gF,j→−pj. This motivates retaining both corrections.
Theorem 1 rules out exactly opposing gradients and guarantees a nonempty set of directions d satisfying gR⊤d>0 and gF⊤d>0. A sufficiently small update decreases both conditional KL objectives at the fixed prefix, including when the Euclidean angle between the gradients is obtuse. This is the theoretical basis for coordinating the two corrections.
Section 5 specifies how to select and scale a direction within this feasible set. The entropy preference map, weighted log-utility criterion, and update-strength rule are design choices with distinct purposes. The entropy map sets the relative priorities of refinement and coverage at the current prefix. Weighted log-utility bargaining realizes these priorities while maintaining positive local gains for both objectives. Once this criterion is specified, its unique solution and closed-form expression follow from Appendix D. The magnitude anchor restores a discrepancy-dependent scale to the unit direction, and the position gate attenuates supervision where teacher and student already agree closely. Appendices E–F characterize the resulting boundedness, teacher-deficit correction, and near-agreement attenuation.
This gives a direct mapping to Algorithm 1: lines 7–10 construct the observations and conditional KL gradients, line 11 sets the preferences, line 12 computes the coordinated direction, and line 13 sets its magnitude. Lines 16–17 realize the resulting gradient signal through the student loss and update. The revised Sections 4–5 explain this progression, and Appendix A.2 distinguishes the specific preference map from the compatibility and bargaining guarantees, which hold for every positive preference pair.
Question 2:
For a fixed-weight combination
Lmix=λR+(1−λ)F,
the gradient is λgR+(1−λ)gF. Reducing this weighted sum can improve one KL while worsening the other when their gradients conflict. Their relative influence also depends on their gradient norms, so fixed loss coefficients can allow one correction to dominate regardless of the intended supervision priority. During on-policy training, teacher–student uncertainty and gradient geometry vary across prefixes, making a single fixed weighting insufficient to control this interaction.
OC-OPD addresses these issues through state-dependent preferences and geometric coordination. The entropy rule determines which correction should receive greater priority at the current prefix: a diffuse teacher increases the preference for coverage, while a concentrated teacher paired with an uncertain student increases the preference for refinement. Gradient normalization separates these priorities from the raw gradient scales. Bargaining then accounts for the current angle between the gradients and selects a direction with positive local gains for both objectives. For distinct positive teacher and student distributions, a sufficiently small step along the resulting descent direction decreases both conditional KLs in fixed-prefix logit space. The discrepancy gate further adjusts supervision strength according to the remaining teacher–student mismatch.
Table 4 and Appendix I.6 isolate the contributions of preference adaptation and bargaining through matched controls. The adaptive linear-mixture variant retains the same entropy preferences, gradient normalization, position gate, and final magnitude w∥gF∥2, and replaces bargaining with a normalized weighted sum. A separate variant fixes the preferences while retaining bargaining and the update-strength rule.
OC-OPD also improves Pass@32 over both variants on all three benchmarks. The first comparison supports the benefit of coordinating the gradients under the same adaptive preferences and strength rule; the second supports the benefit of adapting supervision priorities to the current teacher–student state. These results provide separate empirical support for the two components.