acceptodds
Under review as a conference paper at ICLR 2027

OC-OPD: Observation-coordinated Supervision for On-Policy Distillation

Abstract

Reverse and forward Kullback–Leibler (KL) divergences provide complementary supervision in on-policy distillation: reverse KL refines the student's active token rankings, and forward KL allocates mass to teacher-supported candidates. At student-generated prefixes, we form a four-observation matrix by treating the student and teacher as both candidate sources and evaluators. The diagonal self-observations measure uncertainty, while source-centered cross-observations recover the two KL objectives. We establish an exact identity equating the Fisher inverse-metric coupling of their logit gradients with the sum of these contrasts, yielding a common local descent direction for distinct positive distributions. Observation-coordinated on-policy distillation (OC-OPD) uses observations to set preferences, closed-form bargaining to coordinate gradients, and distributional discrepancy to scale the update. Across three heterogeneous student settings (non-thinking generation, thinking generation, and base-model initialization), OC-OPD achieves strong gains in mathematical reasoning and transfers effectively to out-of-distribution scientific question answering and code generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models, adaptive supervision, gradient coordination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rP8EOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies on-policy distillation by considering the combination of forward and reverse KL divergence objectives between teacher and student distributions. It introduces a matrix representation of the entropy and cross-entropy terms, from which the two KL divergences and their gradients are derived. The authors then use properties of these gradients to motivate an adaptive combination, where the relative contributions of forward and reverse KL are determined based on the distributions' entropies and a gradient-bargaining formulation. The proposed method is evaluated experimentally on language-model distillation tasks, demonstrating improvements over several competing approaches.

Critical strengths
  • The proposed approach for adaptively combining forward and reverse KL gradients, rather than relying on fixed weighting, offers a potentially useful way to balance their complementary properties.

  • The paper provides encouraging experimental results supporting the proposed approach, including comparisons with alternative distillation methods and ablation studies.

Critical weaknesses
  • I found the paper relatively difficult to follow, particularly the transition from the theoretical formulation to the proposed algorithm. The exposition places considerable emphasis on the entropy matrix representation and the resulting mathematical properties, while the motivation for some of the methodological choices is less clear.

  • It is not entirely clear how the observations regarding the relationship between forward and reverse KL gradients in Section 4 motivate the specific adaptive weighting and gradient-coordination rules proposed in Section 5. In particular, some of the results appear to follow from established properties of KL divergence and information geometry.

Questions for authors
  • Please explain more explicitly how the theoretical analysis in Sections 3–4 motivates the particular gradient-coordination procedure proposed in Section 5. In particular, which aspects of the algorithm follow directly from the theoretical results, and which are additional design choices?

  • Please clarify the advantages of the proposed entropy-dependent weighting and gradient-bargaining formulation compared with simpler approaches for combining forward and reverse KL gradients. To what extent do the experimental results isolate the contribution of these particular design choices?

  • I suspect that there is a sign issue in the optimization formulation. In Eq. (6), the gradients are defined with respect to the KL objectives, while Eq. (16) introduces a negative sign in the surrogate loss. How does minimizing this surrogate produce the intended descent direction for the KL objectives?

4 comments

We thank the reviewer for the constructive feedback and address the concerns below.

Question 1:

The analysis connects two complementary corrections to a feasible coordinated update. At each student-generated prefix, the source-centered contrasts in Eq. (4) recover reverse KL, R=DKL(q∥p)R=D_{\mathrm{KL}}(q\Vert p), and forward KL, F=DKL(p∥q)F=D_{\mathrm{KL}}(p\Vert q). Their logit gradients in Eq. (6) explain their different roles: RKL refines the student's current probability allocation, while FKL retains a direct correction toward teacher-supported candidates that the student underrepresents. For a fixed positive teacher distribution and qj→0q_j\to 0, gR,j→0g_{R,j}\to 0, whereas gF,j→−pjg_{F,j}\to -p_j. This motivates retaining both corrections.

Theorem 1 rules out exactly opposing gradients and guarantees a nonempty set of directions dd satisfying gR⊤d>0g_R^{\top}d > 0 and gF⊤d>0g_F^{\top}d > 0. A sufficiently small update decreases both conditional KL objectives at the fixed prefix, including when the Euclidean angle between the gradients is obtuse. This is the theoretical basis for coordinating the two corrections.

Section 5 specifies how to select and scale a direction within this feasible set. The entropy preference map, weighted log-utility criterion, and update-strength rule are design choices with distinct purposes. The entropy map sets the relative priorities of refinement and coverage at the current prefix. Weighted log-utility bargaining realizes these priorities while maintaining positive local gains for both objectives. Once this criterion is specified, its unique solution and closed-form expression follow from Appendix D. The magnitude anchor restores a discrepancy-dependent scale to the unit direction, and the position gate attenuates supervision where teacher and student already agree closely. Appendices E–F characterize the resulting boundedness, teacher-deficit correction, and near-agreement attenuation.

This gives a direct mapping to Algorithm 1: lines 7–10 construct the observations and conditional KL gradients, line 11 sets the preferences, line 12 computes the coordinated direction, and line 13 sets its magnitude. Lines 16–17 realize the resulting gradient signal through the student loss and update. The revised Sections 4–5 explain this progression, and Appendix A.2 distinguishes the specific preference map from the compatibility and bargaining guarantees, which hold for every positive preference pair.

Question 2:

For a fixed-weight combination

Lmix=λR+(1−λ)F,L_{\mathrm{mix}}=\lambda R+(1-\lambda)F,

the gradient is λgR+(1−λ)gF\lambda g_R+(1-\lambda)g_F. Reducing this weighted sum can improve one KL while worsening the other when their gradients conflict. Their relative influence also depends on their gradient norms, so fixed loss coefficients can allow one correction to dominate regardless of the intended supervision priority. During on-policy training, teacher–student uncertainty and gradient geometry vary across prefixes, making a single fixed weighting insufficient to control this interaction.

OC-OPD addresses these issues through state-dependent preferences and geometric coordination. The entropy rule determines which correction should receive greater priority at the current prefix: a diffuse teacher increases the preference for coverage, while a concentrated teacher paired with an uncertain student increases the preference for refinement. Gradient normalization separates these priorities from the raw gradient scales. Bargaining then accounts for the current angle between the gradients and selects a direction with positive local gains for both objectives. For distinct positive teacher and student distributions, a sufficiently small step along the resulting descent direction decreases both conditional KLs in fixed-prefix logit space. The discrepancy gate further adjusts supervision strength according to the remaining teacher–student mismatch.

Table 4 and Appendix I.6 isolate the contributions of preference adaptation and bargaining through matched controls. The adaptive linear-mixture variant retains the same entropy preferences, gradient normalization, position gate, and final magnitude w∥gF∥2w\lVert g_F\rVert_2, and replaces bargaining with a normalized weighted sum. A separate variant fixes the preferences while retaining bargaining and the update-strength rule.

Controlled variantAvg@32 improvement on AIME24 / AIME25 / AIME26
Adaptive linear mixture+0.73 / +3.65 / +4.58 pp
Fixed preferences+2.29 / +3.13 / +1.87 pp

OC-OPD also improves Pass@32 over both variants on all three benchmarks. The first comparison supports the benefit of coordinating the gradients under the same adaptive preferences and strength rule; the second supports the benefit of adapting supervision priorities to the current teacher–student state. These results provide separate empirical support for the two components.

Question 3:

We use gRg_R, gFg_F, and gs∗g_s^{\ast} to denote loss-gradient signals, so the corresponding descent update carries a minus sign. With this convention, Eq. (18) and Algorithm 1, line 16, use the positive-sign surrogate

ℓ~s(θ)=∑isg⁡[gs,i∗]log⁡qθ(i∣s).\widetilde{\ell}_s(\theta)=\sum_i\operatorname{sg}[g_{s,i}^{\ast}]\log q_\theta(i\mid s).

Its coefficients are signed gradient components. Both KL logit gradients sum to zero, so their coordinated combination satisfies

∑igs,i∗=0.\sum_i g_{s,i}^{\ast}=0.

Using ∂log⁡qi/∂zj=δij−qj\partial\log q_i/\partial z_j=\delta_{ij}-q_j, we obtain

∂ℓ~s∂zj=∑igs,i∗(δij−qj)=gs,j∗−qj∑igs,i∗=gs,j∗.\begin{aligned}\frac{\partial\widetilde{\ell}_s}{\partial z_j}&=\sum_i g_{s,i}^{\ast}(\delta_{ij}-q_j)\\\\&=g_{s,j}^{\ast}-q_j\sum_i g_{s,i}^{\ast}\\\\&=g_{s,j}^{\ast}.\end{aligned}

Thus the surrogate produces exactly the intended logit-gradient signal, as derived in Appendix H.1. Appendix H.2 separately examines the changes induced by actual shared-parameter optimizer updates.

We have made the sign convention explicit in Section 5.2: the signed surrogate has logit gradient gs∗g_s^{\ast}, and the corresponding direct logit descent step is −ηgs∗-\eta g_s^{\ast}. The formulas in Eq. (18), Algorithm 1, and Appendix H.1 use this same convention.

Weakness 1:

We have clarified the progression from the two conditional KL corrections to their coordination and update strength in Sections 4–5. The response to Question 1 explains this connection and identifies the role of each stage in Algorithm 1

Weakness 2:

The response to Question 1 distinguishes the compatibility result from the controller's design choices. Question 2 explains the advantages over directly adding the two KL losses and reports controlled ablations of coordination and preference adaptation.

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

I thank the authors for their detailed response and the additional clarifications. I appreciate the explanation of how the theoretical observations motivate different stages of the proposed algorithm, the additional ablation results, and the clarification regarding the sign convention in the optimization.

While these responses address some of my specific questions, my main concerns remain. In particular, I still find the presentation rather difficult to follow, and it is still not entirely convincing that the extensive theoretical development provides sufficiently strong insights into the particular algorithmic choices made in the proposed method. I have to admit that the connection between the theoretical analysis and the methodological contribution remains somewhat less compelling to me. This, I believe that my original assessment still stands, and I will maintain my recommendation.

Reviewer_p5dNOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Observation-Coordinated On-Policy Distillation (OC-OPD), a method that frames aligned student and teacher next-token distributions into a 2x2 matrix of self- and cross-entropies. By centering these observations, it recovers reverse and forward KL divergence gradients, leveraging a Fisher-geometric identity to certify a common local descent direction. A closed-form bargaining rule is then used to coordinate these gradients and update the student model.

Critical strengths
  • The source-evaluator matrix is an interesting conceptual view, and the exact Fisher-geometric identity gRTJ(q)+gF=DKL(q∥p)+DKL(p∥q)g_R^T J(q)^+ g_F = D_{KL}(q\Vert{}p) + D_{KL}(p\Vert{}q) provides a clear rationale for coordinating descent directions.
Critical weaknesses
  • The paper is extremely difficult to follow. The introduction is overloaded with heavy mathematical notation rather than intuitively explaining the core problem and intuition. Furthermore, Sections 5.1 and 5.2 present dense mathematical derivations (e.g., gradient bargaining rules and position strength equations) with almost no explanation of how they connect to or implement the proposed method in practice.

  • The theoretical guarantees rely heavily on the assumption of strictly positive support for both teacher and student distributions. In practice, evaluating large vocabularies requires truncation (e.g., top-kk) for efficiency, which explicitly violates these assumptions and potentially breaks the coordination guarantees.

  • The forward-KL formulation appears to require full-vocabulary teacher probabilities at student prefixes. The paper entirely omits the massive latency and memory implications this overhead would cause at scale.

  • The evaluation is very limited. I highly suggest the authors select more I.D. math benchmarks aside from AIME and do experiments with a different model family to validate their proposed method.

Questions for authors
  • Can you explicitly connect the mathematical derivations in Sections 5.1 and 5.2 to the practical implementation steps of your method?

  • How do you reconcile the strict positivity assumptions in your theoretical guarantees with the practical reality that models use truncated (top-kk) distributions for efficiency?

  • What are the precise wall-clock time and GPU memory overheads for computing full-vocabulary forward KL per prefix compared to standard reverse KL?

5 comments

We thank the reviewer for the constructive feedback and address the concerns below.

Enlarged the evaluation

We have expanded the evaluation to MATH500 and OlympiadBench for non-thinking and base-initialized students, and to OlympiadBench for thinking students. We also evaluate an OLMo teacher–student pair with OPD as the baseline. All scores below are percentages.

1. Additional mathematical benchmarks with non-thinking students. Table 1 of the revised manuscript reports both Qwen3-1.7B and Qwen3-4B:

StudentMethodMATH500 Avg@8OlympiadBench Pass@8
Qwen3-1.7BOPD85.8460.77
Qwen3-1.7BOC-OPD87.4067.40
Qwen3-4BOPD (RKL)87.4073.04
Qwen3-4BOC-OPD94.4380.86

2. Additional mathematical benchmarks from base-model initialization. Table 2 reports results for Qwen3-4B-Base:

MethodMATH500 Avg@8MATH500 Pass@8OlympiadBench Pass@8
Initial48.6085.2052.52
OPD78.4090.4058.80
OC-OPD81.7593.2063.20

These results show gains over the initial model and OPD across metrics. Figures 2–3 also compare Pass@kk on AIME24 and AIME25, examining performance across inference sampling budgets.

3. Additional math evaluation with thinking model Table 3 reports OlympiadBench results for thinking model at both model sizes:

Thinking studentMethodOlympiadBench Avg@8OlympiadBench Pass@8
Qwen3-1.7BOPD58.9179.08
Qwen3-1.7BOC-OPD59.7080.12
Qwen3-4BOPD69.8885.72
Qwen3-4BOC-OPD71.9088.09

OC-OPD improves both Avg@8 and Pass@8 over the initial model and OPD at both sizes, extending the additional benchmark results to thinking generation.

4. Evaluation with a second model family. Appendix I.5 and Table 10 use Olmo-3.1-32B-Instruct as the teacher and Olmo-3-7B-Instruct-SFT as the larger student.

BenchmarkkkMethodAvg@kkPass@kk
MATH5008OPD69.2092.00
MATH5008OC-OPD81.8395.40
AIME2432OPD26.4553.33
AIME2432OC-OPD30.0466.67
AIME2532OPD25.3046.67
AIME2532OC-OPD28.5460.00

On MATH500, OC-OPD improves Avg@8 and Pass@8 over OPD by 12.63 and 3.40 percentage points, respectively. On AIME24, the improvements are 3.59 points in Avg@32 and 13.34 points in Pass@32; on AIME25, they are 3.24 and 13.33 points, respectively. These results extend the improvements over OPD to the OLMo family across all three benchmarks.

Question 1 & Weakness1:

This paper presents an analytical framework for OPD-style methods that connects KL divergence and characterizes the teacher-student relationship of models, and can cover relevant existing OPD works (Appendix G). Building on this framework, we further propose the OC-OPD method. And we have clarified this connection in Sections 4–5, Algorithm 1, and Appendix H.

1. Compute the two corrections at each student-generated prefix.

The teacher and student evaluate the same prefix, producing aligned distributions pp and qq. Their conditional KL logit gradients are

gR=q⊙(log⁡(q/p)−R1),gF=q−p,g_R=q\odot\left(\log(q/p)-R\mathbf{1}\right), \qquad g_F=q-p,

where R=DKL(q∥p)R=D_{\mathrm{KL}}(q\Vert p). Both gradients are formed analytically before backward pass.

2. Section 5.1 determines their relative priorities and coordinated direction. The student and teacher self-entropies determine positive preferences through Eq.8. A diffuse teacher raises the coverage preference; a concentrated teacher paired with a diffuse student raises the refinement preference. We normalize the gradients and substitute the preferences and their current angle into the closed-form solution in Eq.10. Theorem1 establishes compatibility of the two logit gradients. Bargaining selects d⋆d^{\star} with positive inner products with both gradients, so −d⋆-d^{\star} is a common local descent direction in fixed-prefix logit space. Implementation evaluates the closed-form vector expression.

3. Section 5.2 determines how strongly each prefix contributes. The final logit signal is

g_s^{\star}=w_s\lVert g_{F,s}\rVert_2d_s^{\star}, \qquad w_s=\min\\!\left\\{ \frac{1}{2\tau}\sum_i(\sqrt{p_i}-\sqrt{q_i})^2,\\;1 \right\\}.

The factor ∥gF,s∥2=∥q−p∥2\lVert g_{F,s}\rVert_2=\lVert q-p\rVert_2 provides a discrepancy-dependent scale for the unit direction. The gate wsw_s attenuates supervision where the distributions already agree closely. This construction gives ∥gs⋆∥2≤2\lVert g_s^{\star}\rVert_2\le\sqrt{2}.

4. Algorithm1 realizes this signal through the training loss.

Appendix H.1 derives this loss realization, and Appendix H.2 examines the KL changes induced by actual shared-parameter optimizer updates.

Question 2 & Weakness 2:

Our implementation computes the distillation statistics and KL gradients over the full vocabulary. As described in Appendix I.3, we process 128 response positions at a time, with every position retaining the entire vocabulary. This chunking changes the computation schedule while preserving the distributions used by the KL objectives.

The strict-positivity assumption applies to these conditional loss distributions. For finite, unmasked logits, the mathematical softmax distributions have strictly positive coordinates. Theorem 1 applies throughout the open probability simplex, including arbitrarily small positive probabilities; it requires no uniform positive lower bound on token probabilities.

Rollout filtering concerns which prefixes are visited. If top-k filtering is applied during generation, the local compatibility result continues to apply to the full teacher and student loss distributions evaluated at each visited prefix. Hard top-k masks inside the KL computation can introduce support mismatches and fall outside the stated theorem.

The full-vocabulary construction preserves the support assumed by the analysis. Appendix H describes the sampling and loss realization, and Appendix I.3 reports the computation schedule and its measured time and memory costs.

Question 3 & Weakness 3:

Appendix I.3 now reports the incremental cost of the complete OC-OPD update, including its full-vocabulary computations and gradient coordination, relative to standard reverse KL.

GPU memory. Full-vocabulary statistics are computed in chunks of 128 response positions. In the profiled Qwen3 setup, each position retains all 151,936 vocabulary entries. A single FP32 tensor for one chunk occupies

128×151,936×4bytes=74.19MiB.128\times151{,}936\times4\\ \text{bytes} =74.19\\ \text{MiB}.

Processing the statistics in chunks limits the size of intermediate tensors while retaining full vocabulary coverage. On the 10,875-response-token batch, the largest per-GPU increase in PyTorch peak allocated memory is 0.435 GiB relative to standard reverse KL.

Recorded OLMo training-run wall-clock time. Table 8 separately reports complete training-run durations for the Olmo-3.1-32B-Instruct teacher and Olmo-3-7B-Instruct SFT student:

MethodCompleted parameter updatesRecorded wall-clock time
OPD / RKL2506 h 54 min 29 s
OC-OPD2206 h 36 min 12 s

These totals describe the OLMo runs with their respective update counts.

Parameter-update time. We profile Qwen3-4B-Base distilled from Qwen3-30B-A3B-Instruct-2507 on four A800-80GB GPUs. Two matched fixed rollout batches contain 10,875 and 18,043 response tokens. We perform fifteen timed repetitions per batch, then average the batch-wise median timings. Timing includes teacher and student forward passes, backward propagation, synchronization, and the optimizer step, using already collected rollouts.

Under this protocol, OC-OPD adds 0.23 seconds per parameter update, corresponding to a 2.14% increase over standard reverse KL. Both KL logit gradients are computed analytically from the teacher and student distributions.

Thank you for the response and revisions. The explanations, additional benchmarks and model-family evaluation, and computational-overhead measurements address some of my original concerns. I am inclined to raise my score to weak reject. However, several unresolved issues limit my confidence in the experimental conclusions:

1. Unexplained changes to the experimental configuration and existing results. Between versions, the global batch size changes from 128 to 8, the mini-batch size from 32 to 8, and the number of mini-batches from 4 to 1. Several previously reported AIME scores also change, including the base student’s AIME24/25 Pass@8 increasing from 36.67 to 40.00. It is unclear whether these changes reflect corrected documentation, reevaluation, different checkpoints, or new training runs. Please explain these changes and identify the configuration used for each experiment and its baselines.

2. Insufficient detail to assess the OLMo comparison. Appendix I.5 evaluates OC-OPD at its best observed checkpoint, step 75, but does not specify the OPD checkpoint or the checkpoint-selection dataset and criterion. Consequently, I cannot verify whether both methods receive comparable checkpoint selection. Please clarify the procedure for both methods and whether selection is separate from the reported test evaluations.

3. Unclear statistical reliability of the reported improvements. Appendix J mentions six runs but does not explain whether these are independent training runs or repeated decoding evaluations. The construction of the error bars and the rationale for reporting the mode are also unclear. These details matter when assessing the smaller improvements over stronger baselines and the ablation variants. Please clarify the protocol and either substantiate or soften the statistical-significance claims.

4. Overstated interpretation of the end-to-end timing results. The controlled per-update profiling is useful, but Table 8 compares 220 OC-OPD updates with 250 OPD updates. The shorter recorded duration therefore does not establish an efficiency advantage at matched work. Please qualify the “faster wall-clock time” statement and distinguish it from the controlled overhead measurement.

Please address these points in the revised manuscript and mark all changes relative to the original submission in a different color, including modified numerical results and hyperparameters.

Reviewer_apT9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OC-OPD, a method that adaptively combines reverse and forward KL supervision at student-generated prefixes. Based on gradient bargaining, a common gradient direction is found to simultaneously minimizes the forward and reverse KL. Based on this, a closed form gradient is derived as a weighted combination of the gradients for forward and reverse KL divergences, and is subsequenctly used in the distillation loss. Experimental results demonstrate the superior performance of the proposed method.

Critical strengths
  1. The common gradient direction method is novel and principled for adaptive KL combination.
  2. The paper provides fruitful discussion on preliminaries and background.
Critical weaknesses
  1. Equation 2 doesn't provide the correct gradient for the sequence level reverse KL objective in equation 1.
  2. Only two teacher-student pairs within the same model family are evaluated, which limits the breadth of empirical evidence.
Questions for authors
  1. Eq. (8) appears ad hoc. Why should ωj\omega_j be computed in the proposed way instead of simpler alternatives?
  2. In section 5.2, no justification/motivation is provided before BB and ww are defined. Why are they defined in the given way?
4 comments

We thank the reviewer for the constructive feedback and address the concerns below.

Question 1:

Eq. (8) is a design choice that translates the observed teacher and student uncertainties into priorities for two complementary corrections. The normalization h=H/(H+h0)h=H/(H+h_0) maps entropy monotonically to [0,1)[0,1). The preferences

νF=ϵ+hT,νR=ϵ+(1−hT)hS\nu_F=\epsilon+h_T,\qquad\nu_R=\epsilon+(1-h_T)h_S

give coverage greater priority when the teacher distributes probability across several candidates, and refinement greater priority when the student is diffuse while the teacher is concentrated. The multiplicative refinement term expresses this joint condition. Thus, increasing teacher entropy shifts preference toward coverage, while increasing student entropy at fixed teacher entropy shifts preference toward refinement.

The positive floor keeps both corrections active. Since hT+(1−hT)hS≤1h_T+(1-h_T)h_S\le 1, the normalized preferences satisfy

ωj≥ϵ1+2ϵ>0.\omega_j\ge\frac{\epsilon}{1+2\epsilon} > 0.

For distinct positive teacher and student distributions, Theorem 1 establishes a nonempty common-descent region. The positive preferences allow the bargaining criterion to select a direction with positive local gains for both conditional KL objectives. Appendix E further shows that, whenever qi<piq_i < p_i, the resulting logit correction satisfies

−gi∗≥wωF(pi−qi)≥wϵ1+2ϵ(pi−qi)>0.-g_i^{\ast}\ge w\omega_F(p_i-q_i)\ge w\frac{\epsilon}{1+2\epsilon}(p_i-q_i) > 0.

This property preserves a correction toward teacher-supported candidates that the student underrepresents. It concerns the local logit signal.

The update also has consistent limiting behavior. When the two gradients align, it reduces to w(q−p)w(q-p) independently of the preferences. Near agreement with a fixed interior teacher distribution, preference-dependent effects enter only at higher order (Appendices D.1 and F).

Table 4 isolates the contribution of preference adaptation by fixing ωR=ωF=1/2\omega_R=\omega_F=1/2 while retaining the bargaining procedure and update-strength rule. This control lowers Avg@32\mathrm{Avg}@32 by 2.29, 3.13, and 1.87 points on AIME24, AIME25, and AIME26, respectively, and also lowers Pass@32\mathrm{Pass}@32 on all three benchmarks. These results support adapting preferences beyond the equal-preference control in the evaluated setting. We have added the motivation before Eq. (8); Appendices A.2 and E give the preference construction and its positive-floor properties.

Question 2:

Section 5.1 selects the correction direction; Section 5.2 determines how strongly each prefix contributes. Entropy alone cannot determine this strength: two concentrated distributions can agree or favor different tokens while having identical entropies. We therefore measure their discrepancy using squared Hellinger distance,

B=12∑i(pi−qi)2.B=\frac{1}{2}\sum_i\left(\sqrt{p_i}-\sqrt{q_i}\right)^2.

It is symmetric, bounded in [0,1][0,1], and vanishes exactly at agreement. For a fixed interior teacher distribution and q→pq\to p, it also shares the leading quadratic geometry of the KL objectives:

B=18(R+F)+O(∥q−p∥23).B=\frac{1}{8}(R+F)+O\left(\lVert q-p\rVert_2^3\right).

We use w=min⁡(B/τ,1)w=\min(B/\tau,1) to increase supervision with discrepancy and cap its strength. This gate admits the allocation interpretation

w=arg⁡max⁡0≤u≤1(uB−τ2u2).w=\underset{0\le u\le 1}{\arg\max}\left(uB-\frac{\tau}{2}u^2\right).

The linear term rewards allocating supervision to larger discrepancies, and the quadratic term assigns increasing marginal cost to stronger supervision. Under this chosen criterion, strict concavity gives the unique clipped-linear gate: strength rises with disagreement and reaches its full value at B≥τB\ge\tau (Appendix A.2).

The final logit signal

g∗=w∥q−p∥2d∗g^{\ast}=w\lVert q-p\rVert_2d^{\ast}

combines the coordinated unit direction with a discrepancy-dependent magnitude. It satisfies ∥g∗∥2=w∥q−p∥2≤2\lVert g^{\ast}\rVert_2=w\lVert q-p\rVert_2\le\sqrt{2}. For distinct positive p,qp,q, the positive gate preserves the common-descent signs in fixed-prefix logit space. Near agreement with a fixed interior teacher, the update has cubic attenuation in ∥q−p∥2\lVert q-p\rVert_2, progressively reducing supervision at closely matched positions (Appendices E–F).

Table 4 isolates this component by setting w=1w=1 while retaining the preference rule and coordinated direction. Removing the gate lowers Pass@32\mathrm{Pass}@32 by approximately 3.33 percentage points on each AIME benchmark. Section 5.2 now explains the direction–magnitude separation and the discrepancy gate; the allocation interpretation and update properties appear in Appendices A.2 and E–F.

Weakness 1:

We have provided the exact sequence-level gradient and its derivation in Appendix A and clarified this current-point interpretation after Eq. (2) in Section 3.1.

Eq. (2) implements a token-local surrogate. The exact sequence-level gradient of Eq. (1) also accounts for how each sampled action changes subsequent prefixes. Define

ct(θ)=log⁡qθ(yt∣st)pT(yt∣st).c_t(\theta)=\log\frac{q_\theta(y_t\mid s_t)}{p_T(y_t\mid s_t)}.

Then

∇θJOPD=Ey∼qθ[∑t=1∣y∣∇θlog⁡qθ(yt∣st)∑u=t∣y∣cu(θ)].\nabla_\theta J_{\mathrm{OPD}}=\mathbb{E}_{y\sim q_\theta}\left[\sum_{t=1}^{|y|}\nabla_\theta\log q_\theta(y_t\mid s_t)\sum_{u=t}^{|y|}c_u(\theta)\right].

The cumulative term credits each action for its current discrepancy and the future discrepancies encountered through subsequent prefixes. Eq. (2) retains the current-token credit ctc_t. Freezing the teacher leaves this dependence of future prefixes on student actions intact. We have revised this part in the updated manuscript for improved clarity.

Our update optimizes conditional KL objectives at the collected student prefixes. At iteration kk, we collect SkS_k using qk=qθkq_k=q_{\theta_k} and hold these prefixes fixed during differentiation. The reverse-KL component can be written as

J~R,k(θ)=1∣Sk∣∑s∈SkDKL(qθ(⋅∣s)∥pT(⋅∣s)).\widetilde{J}_{R,k}(\theta)=\frac{1}{|S_k|}\sum_{s\in S_k}D_{\mathrm{KL}}\left(q_\theta(\cdot\mid s)\Vert p_T(\cdot\mid s)\right).

At each fixed prefix, the expected token-local gradient in Eq. (2) equals the conditional KL gradient at the current parameters:

∇θDKL(qθ∥pT)∣θk=Ea∼qk[log⁡qk(a)pT(a)∇θlog⁡qθ(a)∣θk],\left.\nabla_\theta D_{\mathrm{KL}}(q_\theta\Vert p_T)\right|_{\theta_k}=\mathbb{E}_{a\sim q_k}\left[\log\frac{q_k(a)}{p_T(a)}\left.\nabla_\theta\log q_\theta(a)\right|_{\theta_k}\right],

where all distributions condition on the same prefix. Full-vocabulary evaluation computes this conditional gradient directly.

Algorithm 1 coordinates the conditional reverse- and forward-KL logit gradients, applies one student update, and collects fresh rollouts for the next iteration. On-policy sampling determines the training prefixes, while conditional KL optimization supplies supervision at those prefixes. The identities and common-descent result in Section 4 concern these fixed-prefix logit gradients.

Weakness 2:

We have extended the evaluation to an OLMo teacher–student pair, using Olmo-3.1-32B-Instruct as the teacher and the SFT initialization of Olmo-3-7B-Instruct as the student. Table 10 and Appendix I.5 report comparisons with OPD on MATH500, AIME24, and AIME25. Additionally, we have added more empirical results in Tables 1–3 and Figures 2–3. OC-OPD is evaluated at its best observed checkpoint, obtained at step 75. MATH500 uses eight responses per problem; AIME24 and AIME25 use 32. All scores below are percentages.

BenchmarkkkMethodAvg@k\mathrm{Avg}@kPass@k\mathrm{Pass}@k
MATH5008OPD69.2092.00
MATH5008OC-OPD, step 7581.8395.40
AIME2432OPD26.4553.33
AIME2432OC-OPD, step 7530.0466.67
AIME2532OPD25.3046.67
AIME2532OC-OPD, step 7528.5460.00

Relative to OPD, the Avg@k\mathrm{Avg}@k improvements are 12.63, 3.59, and 3.24 percentage points on MATH500, AIME24, and AIME25, respectively. The corresponding Pass@k\mathrm{Pass}@k improvements are 3.40, 13.34, and 13.33 points, using the displayed scores. These comparisons extend the observed gains over OPD to a second model family and an additional mathematical benchmark.

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

We thank the authors for their additional results and clarifications. However, we echo Reviewer rP8E's view that the overall presentation is somewhat unclear, and some design choices are not adequately justified. Therefore, we decide to maintain the score.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback and the AC for overseeing the review process.

We appreciate that the reviewers recognized (1) the novel and principled coordination of KL gradients, (2) the interesting entropy-matrix framework and its Fisher-geometric foundation for common descent, and (3) the encouraging empirical results and useful efficiency studies. These observations highlight the theoretical and methodological contributions of OC-OPD.

To address concerns, we have substantially revised the manuscript and highlight the major improvements in three aspects, a revised PDF (changes in blue) is uploaded:

Theoretical clarification. We clarify the connection between our theoretical analysis and practical algorithm, and provide further justification for entropy-dependent preferences, gradient bargaining. Our explanations and additional evaluations have addressed some concerns raised by reviewers, but reviewer rP8E still finds our work difficult to follow, and reviewer apT9 echoes this view.

Expanded experimental evaluation. We substantially extend our evaluation to MATH500 and OlympiadBench. We further introduce cross-family experiments with OLMo models, demonstrating consistent improvements over baseline.

Computational efficiency. We provide a detailed analysis of vocabulary computation, GPU memory consumption, and training time. OC-OPD introduces only 2.14% additional per-update time and a maximum measured 0.435 GiB increase in peak GPU memory compared with standard reverse-KL distillation under the profiled setup,with faster wall-clock time to convergence.

These revisions improve the theoretical clarity, empirical generality, and practical efficiency characterization of OC-OPD. We appreciate the reviewers' comments and believe the revised manuscript provides stronger support for our contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.