acceptodds
Under review as a conference paper at ICLR 2027

Decoupled On-Policy Distillation for LLM Reasoning

Abstract

On-policy distillation (OPD) commonly relies on forward or reverse KL divergence (FKL/RKL) to align the student with the teacher. However, FKL has a particular limitation in OPD: its per-token loss and gradient concentrate on a small set of candidate tokens on which the teacher probability far exceeds the student probability . We call this high-disparity set of candidates the Mismatched Region, where exceeds by one to two orders of magnitude, and this concentration makes OPD prone to response-length inflation, unreasonable truncation, and accuracy collapse. RKL, in contrast, avoids this domination through its -self-weighting but can under-cover teacher modes through its mode-seeking behavior. To address these issues, we propose a simple but effective optimization objective, Decoupled-KL Distillation (DKLD), a loss that decomposes the KL into two contributors, the Matched Region and the Mismatched Region. In detail, we simply use forward KL in the Matched Region, and Reverse KL with -self-weighting takes responsibility within the Mismatched Region. Furthermore, the student rollouts required by OPD simultaneously provide the group-relative advantages for GRPO, but their gradients can conflict. To reduce this interference, we propose an advantage-weighted distillation module that balances these two objectives and combines OPD with GRPO in a single framework. Through comprehensive experiments, across mathematical reasoning and code generation tasks, DKLD reaches state-of-the-art performance in all configurations we ran and drastically mitigates the length-inflation collapse.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, OPD, FKL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cbHROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies instability in on-policy distillation and proposes DKLD, which separates supervision of region probability masses from supervision within each region. It retains forward KL on the region masses and matched candidates, and uses student-mass-weighted reverse KL within the mismatched region. The method combines this loss with GRPO and advantage-weighted distillation (AWD) on correct rollouts. Experiments report improvements across three Qwen3 student sizes and five math benchmarks, with an additional coding study.

Critical strengths

(1) The math results are consistent across student sizes. Table 2 reports average gains over ToDi of 2.4, 2.5, and 2.7 points for the 0.6B, 1.7B, and 4B models. The use of three training seeds and validation-selected checkpoints gives these comparisons some substance. The gains on the harder math sets are particularly encouraging.

(2) The ablations make a useful case for the loss design. Removing the binary term reduces Hard_mini150 accuracy from 73.1 to 60.7 (Table 3). Table 8 also shows that DKLD remains effective without AWD, whereas FKL collapses even with AWD. This is meaningful evidence that the stability improvement comes at least partly from the distillation objective.

Critical weaknesses

Major concerns

(1) The benefit of AWD is not fully separated from a reduction in distillation strength. Its weight is at most 0.5 on correct rollouts, while the main baselines use weight 1 at the same nominal KD coefficient. Tables 3 and 8 help address this for the fixed-direction losses, but the strongest alternatives, AKL and ToDi, are not tested with AWD or a tuned constant weight. Such controls would make the claimed margin over existing adaptive losses more convincing.

(2) The gradient interpretation in Section 3.1 needs correction or clarification. For a fixed partition, let a and b denote the teacher and student masses in G, and D the conditional reverse KL there. Differentiating Eq. (10) along a common shift of all student logits in G gives b - a + b(1 - b)D. Thus the total push toward G is not unchanged unless the student-mass multiplier is detached. This does not invalidate the empirical gains, but it matters for the proposed explanation of them.

(3) The coding experiment uses held-out test cases from problems that are also used for training (Section 4.1). This measures whether solutions pass additional tests on familiar problems; it does not establish transfer to unseen coding problems. I would give this result limited weight in assessing generality. A problem-disjoint split would be considerably more informative.

Minor concerns

(1) Algorithm 1 renormalizes the teacher top-256 logits but leaves the student normalization unclear. Please specify whether student probabilities are renormalized on the same support and how omitted vocabulary mass is handled, since both affect the binary term and region assignment.

(2) Table 4 mixes fixed absolute thresholds with the main configuration, which uses a relative teacher threshold and an annealed student threshold. A sweep that holds the student schedule fixed and varies only the teacher ratio would provide a cleaner sensitivity check.

Questions for authors

(1) Does DKLD retain its advantage over AKL and ToDi when all three use AWD, or when constant KD weights are selected on validation? How much does AWD add after matching average distillation strength?

(2) Is the student-mass multiplier in Eq. (10) detached in the implementation? Please reconcile the implemented gradient with the claim in Section 3.1 that the total push toward G is unchanged.

(3) Can the authors evaluate on LiveCodeBench problems excluded from training and report numerical pass@4 and truncation rates with variation across seeds?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cbHR (Part 1 / 2): W1–W3

We thank you for your comments and suggestions. The following are our responses to your concerns.

W1: AWD vs. reduced distillation strength

Thank you for this suggestion. We ran new experiments to compare AKL, ToDi and DKLD under the same weight in both settings.

Table R1: Qwen3-1.7B

KD weightSeedsHard_mini150MATH-500 (step 80)
AKL1365.8±5.065.8 \pm 5.076.6±1.276.6 \pm 1.2
ToDi1367.6±4.267.6 \pm 4.276.6±0.776.6 \pm 0.7
DKLD1368.9±3.968.9 \pm 3.977.5±0.677.5 \pm 0.6
AKLAWD gate166.077.4
ToDiAWD gate168.776.0
DKLD (ours)AWD gate373.1±3.7\mathbf{73.1 \pm 3.7}81.5±1.5\mathbf{81.5 \pm 1.5}

Due to the time limit, we could run only one seed for AKL and ToDi with the AWD gate.

  1. Same constant weight. At w=1w = 1 no method uses the gate, and DKLD is still the best.
     

  2. Same AWD gate. Following your suggestion, we trained AKL and ToDi with the AWD gate. The gate changes their scores by at most 1.1 points, which is within the variation across seeds. Under the same gate, DKLD leads AKL by 7.1 and 4.1 points and ToDi by 4.4 and 5.5 points.

We have added these changes to the appendix.

W2: Gradient interpretation in Section 3.1

You are right. The multiplier rGSr_G^S is not detached in our implementation, so the push towards GG is b−a+b(1−b)Db - a + b(1 - b)D, and our statement in Section 3.1 is not exact. We have corrected it in the revision.

Also, we measured how large this term is in our final objective. It is very small and almost negligible.

Specifically, we measured it with the released loss code on 256 training rollouts of the initial Qwen3-1.7B student with the Qwen3-8B teacher, at the first and last values of εS\varepsilon_S in our schedule.

εS\varepsilon_Saabbb⋅Db \cdot Dextra term / ∣b−a∣\lvert b - a \rvert
0.010.2550.00375.8×10−45.8 \times 10^{-4}0.23%
0.0010.2420.000324×10−54 \times 10^{-5}0.02%

Here aa, bb and b⋅Db \cdot D are averaged over the positions where GG is non-empty, and the last column over all positions. The push towards GG is therefore dominated by b−ab - a, so the statement in Section 3.1 holds as a close approximation, and we now present it as such.

W3: Coding experiment split

Thank you for this comment. We agree. The split in Section 4.1 is on test cases, not problems, so it does not measure transfer to unseen problems. Following your suggestion, we retrained the model under the problem split configuration. We train on 737 LiveCodeBench problems released before 2025-02-01, and test on the same 131 LiveCodeBench v6 problems as in the paper (released February to April 2025), with all of their test cases. All other settings follow Section 4.1.

Table R2: Qwen3-1.7B student, Qwen3-8B teacher, 131 LiveCodeBench v6 problems

Seedspass@4, test-case split (paper)pass@4, problem-disjoint splittrunc., problem-disjoint split
Base (no training)–19.918.38.6%
GRPO127.923.721.9%
AKL129.028.28.2%
DKLD (ours)131.331.33.6%

On problems never seen in training, DKLD still gives the best pass@4 and the lowest truncation rate, while GRPO drops the most and truncates over a fifth of its responses. We have replaced the coding experiment in Section 4.1 with this split.


AuthorsOpenReview ↗

Response to Reviewer cbHR (Part 2 / 2): W4–W5, Q1–Q3

W4 (Minor 1): Student normalization in Algorithm 1

Thank you for this question. The student is renormalized in the same way as the teacher. We take the student's logits at the teacher's top-256 token ids and apply a softmax over just those tokens, so the two region masses sum to one for each model, and both the binary term and the region assignment use these renormalized probabilities. We have clarified this in Algorithm 1.

W5 (Minor 2): Threshold sweep in Table 4

Thank you for raising this, and Reviewer vv2E shared the same concern.

Time during the rebuttal was tight, so instead of a full sweep we trained DKLD with two more teacher ratios, one below and one above our default. The student threshold keeps the same annealed schedule in every row, so only ρ\rho changes.

Table R3: 1.7B, MATH-500

ρ\rhoεS\varepsilon_S scheduleSeedspass@1
DKLD0.050.01→0.0010.01 \to 0.001181.8
DKLD (paper)0.10\mathbf{0.10}0.01→0.0010.01 \to 0.001383.1±0.8\mathbf{83.1 \pm 0.8}
DKLD0.200.01→0.0010.01 \to 0.001182.0

We have also marked the main row of Table 4 clearly, since it follows a different rule from the fixed-threshold rows.

Q1: DKLD vs. AKL and ToDi under matched AWD / KD weight

Thank you. Yes, DKLD keeps its lead in both settings (Table R1 in our response to W1). With the AWD gate on all three, DKLD beats AKL by 7.1 and 4.1 points and ToDi by 4.4 and 5.5 points on Hard_mini150 and MATH-500. With a constant weight of 1, DKLD is still the best, by a smaller margin. The gate adds about 4 points for DKLD, and a constant of 0.01, below the gate's range, also falls short of it. For AKL and ToDi the gate changes the score by at most 1.1 points.

Q2: Is the student-mass multiplier detached

Thank you for this question. No, the multiplier is not detached. The implemented gradient along a common shift in GG is b−a+b(1−b)Db - a + b(1 - b)D, as you derived. We have corrected the claim in Section 3.1 (see our response to W2).

Q3: Problem-disjoint LiveCodeBench evaluation

Thank you. Yes, Table R2 in our response to W3 reports pass@4 and truncation rates on 131 LiveCodeBench problems that are fully excluded from training. Because of the time limit we could only train one seed per method during the rebuttal.

We appreciate your careful review. If any point needs more detail, please do not hesitate to tell us.

Sincerely,
Authors

Reviewer_7mx3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DKLD, which partitions the vocabulary into “matched” and “mismatched” regions based on the teacher’s and student’s probability distributions. DKLD retains forward KL supervision on the region masses and within the matched region, while using reverse KL within the mismatched region to reduce preference signal to excessively imitate teacher-preferred tokens. The paper further combines this objective with GRPO and introduces advantage-weighted distillation to reduce interference between OPD and GRPO. The motivation is reasonably clear, and the paper provides component ablations to partially support its claims.

Critical strengths
  1. The paper has a clear motivation and attempts to address the optimization bias of forward KL in OPD.
Critical weaknesses
  1. Alignment between motivation and contribution. Some of the claimed contributions are not fully aligned with the stated motivation. Specifically, the paper tries to address two relatively distinct directions: modifying the KL objective in OPD and dynamically balancing OPD with GRPO. The necessity of the second direction does not clearly follow from the initial motivation, yet the main experimental results evaluate the combined framework.

  2. Validation of KL revision. Although the paper provides gradient analysis and component ablations, these do not sufficiently explain why the modified KL objective improves downstream performance, particularly compared with existing mixed-KL objectives.

  3. Insufficient evidence for OPD-GRPO adaptation The experiments do not sufficiently establish the proposed mechanism for coordinating OPD and GRPO. In particular, it remains unclear whether the advantage-weighted gate helps by reducing gradient interference or simply by lowering the overall strength of distillation.

Questions for authors
  1. How does the decision to combine OPD with GRPO follow from the initial motivation concerning FKL’s behavior in mismatched regions?

  2. Does the advantage-weighted distillation used in DKLD outperform a constant weight with the same average distillation strength?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7mx3 (Part 1 / 2): W1–W3

We first thank you for the comments that made the paper clearer. The following are our one-by-one responses to your concerns.

W1: Alignment between motivation and contribution

Thank you for this question. We agree that the alignment between motivation and contribution should be clearer, and we have revised the introduction.

  1. The main reason we take GRPO into our framework is that the student rollouts required by OPD simultaneously provide the group-relative advantages for GRPO. Every KD baseline in Table 2 is also trained with GRPO, so GRPO is the shared setting and the KL objective is our main contribution. AWD is a small module that reduces the conflict between the two gradients in this setting.
     

  2. We have run the corresponding ablation experiment in our manuscript. Without AWD, DKLD is still the best KD objective (68.9 on Hard_mini150 and 77.5 on MATH-500 at step 80; Tables 3 and 8, compared with the baselines in Tables 2 and 7).

W2: Validation of KL revision

Thank you for pointing this out. We have stated the difference between our method and the other baselines more clearly in the revision.

We explain the advantage of DKLD from two perspectives.

  1. From the loss to the accuracy. FKL concentrates on a few Mismatched-Region tokens. The responses then grow longer and get truncated. DKLD has the lowest truncation and the highest accuracy.

    Table R1: 1.7B, MATH-500, step 80

    pass@1trunc.
    FKL13.7±0.613.7 \pm 0.691.2%
    AKL76.6±1.276.6 \pm 1.221.1%
    ToDi76.6±0.776.6 \pm 0.718.2%
    DKLD (ours)81.5±1.5\mathbf{81.5 \pm 1.5}14.6%

     

  2. Difference from mixed-KL objectives. AKL and ToDi mix FKL and RKL with continuous weights. AKL uses one weight for the whole position, and ToDi gives FKL its largest weight on Mismatched-Region tokens, so the gradient concentration still exists. DKLD instead uses an exact decomposition (Proposition 2) and keeps forward KL on the region masses through the binary term, which largely alleviates response truncation and improves model performance.

W3: Insufficient evidence for OPD-GRPO adaptation

Thank you for pointing this out. As we mentioned in W1, the rollouts generated by the student model can be used to calculate the advantage of GRPO without extra cost. We understand that you might be concerned about the clear evidence for combining OPD and GRPO.

Regarding "In particular, it remains unclear whether the advantage-weighted gate helps by reducing gradient interference or simply by lowering the overall strength of distillation", we set the strength of distillation to a very low constant (0.01):

Table R2: 1.7B, DKLD

KD weightSeedsHard_mini150MATH-500 (step 80)
Uniform1368.9±3.968.9 \pm 3.977.5±0.677.5 \pm 0.6
Low constant0.01166.775.0
AWD gate (ours)σ(−A^/τ)\sigma(-\hat{A}/\tau)373.1±3.7\mathbf{73.1 \pm 3.7}81.5±1.5\mathbf{81.5 \pm 1.5}

The table shows that a weight of 1 and a low constant weight both score below our dynamic weight. This is intuitive. A large constant lets KD compete with GRPO on rollouts with a high advantage, and a small constant removes the teacher signal where GRPO gives no gradient. The gate avoids both.


AuthorsOpenReview ↗

Response to Reviewer 7mx3 (Part 2 / 2): Q1–Q2

Q1: Why combine OPD with GRPO

Thank you for this question. Combining OPD with GRPO is a common setting in recent work on reasoning distillation [A–C], and the survey [D] lists GRPO with a token-level KL as its own category. We follow this setting, and every KD baseline in Table 2 is trained with GRPO as well. The KL objective does not depend on GRPO. DKLD without GRPO reaches only 66.9±2.566.9 \pm 2.5 on Hard_mini150 (Table 3), and adding GRPO raises it to 73.1±3.773.1 \pm 3.7.


Q2: AWD vs. a constant weight of the same average strength

Thank you for raising this question. We have experimented with constant weights in our response to W3 (Table R2). The gate gives each correct rollout a weight between 0.08 and 0.5, and we tested one constant above this range (1) and one below it (0.01). Both score below the AWD gate, by 4.2 and 6.4 points on Hard_mini150 and by 4.0 and 6.5 points on MATH-500.

Thank you again for your time. If you have any further questions or concerns, we would be happy to answer them.

Best regards,
Authors


AuthorsOpenReview ↗

Kindly Following Up on Our Responses

Dear Reviewer 7mx3,

We hope you are doing well.

As the discussion period is coming to an end, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns

If there are any remaining questions, we would be happy to clarify further Thank you again for your time and valuable feedback

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of New Experiments and Revisions

Dear Respected Reviewer 7mx3

We would like to kindly follow up on our response.

To address your concerns, we added:

  • DKLD with a low constant KD weight (0.01). It scores below the AWD gate, so the gate does not help only by weakening distillation.
  • AKL and ToDi with the same AWD gate. DKLD is still the best.
  • A revised introduction that clarifies the link between our motivation and contributions.

We would greatly appreciate it if you could let us know whether these address your concerns.

Best regards,

Authors

Reviewer_vv2EOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that forward KL fails specifically in on-policy distillation, because its per-token loss and logit gradient pile up on a small set of candidates where the teacher probability exceeds the student's by one to two orders of magnitude. The authors call this set the Mismatched Region and link the concentration to length inflation, truncation and accuracy collapse. They give an exact three-term decomposition of forward KL: a binary KL between the aggregate masses of the Matched and Mismatched regions, plus one within-region KL each. They keep forward KL on the binary and Matched terms and swap the within-Mismatched term for a student-mass-weighted reverse KL. On top of that they add an advantage-weighted gate that restricts KD to correct rollouts and scales it by σ(−Â/τ) to cut interference with GRPO. Experiments distill Qwen3-8B into 0.6B and 1.7B and Qwen3-14B into 4B, over GSM8K, MATH-500, Hard_mini150, a new CompMath-700, AIME 2025 and LiveCodeBench v6, with three seeds per configuration.

Critical strengths
  1. Proposition 2 is an exact identity and the method follows from it rather than being bolted on afterwards. The ablations make this count: swapping the two KL directions costs 24.2 points (73.1 to 48.9), and dropping the binary term costs 12.4 points, landing below plain FKL (60.7 vs 61.8). Together these say the gain comes from the specific construction, and in particular from the binary term continuing to supervise region mass. That is what separates this from generic FKL/RKL interpolation such as AKL and ToDi.

  2. The empirical work is careful by the standards of this area: best on all five math benchmarks at three student scales, three seeds with standard deviations, a code-generation result, and a released contamination-controlled eval set (CompMath-700) alongside code and weights.

Critical weaknesses
  1. The motivating collapse and the actual baseline do not match. Figure 1 sells the paper on FKL collapsing to 13.7% pass@1 with 91.2% truncation at step 80, but the FKL row in Table 2 is a healthy 62.6 average at 1.7B, because checkpoints there are "selected on validation". Early stopping already neutralises the pathology. The honest headline comparison is 67.1 vs 62.6. Before I accept "FKL is broken in OPD" as the premise, I would want to see that the collapse survives a tuned λKD and standard length normalisation, rather than being a property of one unregularised setup.

  2. The main table is confounded by the AWD gate, and the privileged teacher is never ablated. Only DKLD uses the gate in Table 2; all KD baselines use w=1, and the gate is worth 4.2 points by the paper's own ablation. The matched comparison does exist (DKLD at uniform weight still beats ToDi, 68.9 vs 67.6) but only at 1.7B on one benchmark. That comparison belongs in the headline table. Separately, the teacher receives an answer-aware system prompt in every teacher-based method and there is no ablation removing it, so some unknown share of the improvement comes from a privileged teacher rather than from the divergence design.

  3. Table 4 does not establish the threshold claim. The winning row (εT = 0.10†) uses the dynamic annealed rule while every other row uses fixed absolute thresholds. The sweep is therefore comparing two different mechanisms and cannot show that 0.10 is the right operating point. The εS annealing schedule (ε0 = 0.01 to ε1 = 0.001), which is two of the three new hyperparameters, is not ablated at all.

Questions for authors
  • How much of the gain survives if the teacher does not get the answer in its system prompt?
  • Does |G| or rSG shrink as the student improves? A trace of region size over training would show whether the mechanism is transient or persistent.
  • Is the correct-rollout mask doing work independently of the AWD gate, i.e. what happens with KD on all rollouts plus AWD?
4 comments
AuthorsOpenReview ↗

Response to Reviewer vv2E (Part 1 / 2): W1–W3

Before answering your comments, we would first like to thank you for your suggestions, which help make the paper better. The following are our one-by-one responses:

W1: Motivating collapse vs. the actual FKL baseline

Thank you for this careful comment. We agree that the headline comparison uses the checkpoints only at step 80, and we have stated this in Figure 1 and the introduction.

  1. Early stopping. The FKL in Table 2 is the checkpoint at step 10, and FKL degrades from step 15 (Appendix E). Validation selection therefore keeps only its first 10 steps.
     

  2. Length normalisation. The KD term in Eq. 14 is already averaged over tokens (1/∣y∣1/\lvert y \rvert) for every KD method.
     

  3. Tuned λKD\lambda_{\mathrm{KD}}. Table 8 tests a smaller KD weight. The AWD gate scales the weight of FKL by at most 0.5, so its effective λKD\lambda_{\mathrm{KD}} is at most 0.05. Following your suggestion, we also ran FKL with λKD=0.01\lambda_{\mathrm{KD}} = 0.01. FKL collapses at every weight, while DKLD stays stable.

    Table R1: 1.7B, MATH-500, step 80

    λKD\lambda_{\mathrm{KD}}Seedspass@1trunc.
    FKL0.1313.7±0.613.7 \pm 0.691.2%
    FKL + AWD≤0.05\le 0.05315.3±0.915.3 \pm 0.990.7%
    FKL0.01115.889.4%
    DKLD (ours)0.1381.5±1.5\mathbf{81.5 \pm 1.5}14.6%

    Due to the time limit, we could run only one seed for FKL with λKD=0.01\lambda_{\mathrm{KD}} = 0.01.

    We found that some outputs get the correct answer even though they are truncated, as we extract the answer not only from \boxed{} but also from "final answer is", "the answer is" and the last number in the text.

W2: AWD gate confound and the privileged teacher

Thank you for this valuable comment. We have added a DKLD (w=1w = 1) row to Table 2.

  1. AWD gate. At w=1w = 1, DKLD is still the best KD objective (data from Tables 2, 3, 7 and 8 of our paper).

    Table R2: 1.7B, w=1w = 1, 3 seeds

    Hard_mini150MATH-500 (step 80)
    AKL65.8±5.065.8 \pm 5.076.6±1.276.6 \pm 1.2
    ToDi67.6±4.267.6 \pm 4.276.6±0.776.6 \pm 0.7
    DKLD (ours)68.9±3.9\mathbf{68.9 \pm 3.9}77.5±0.6\mathbf{77.5 \pm 0.6}

     

  2. Answer-aware teacher. We follow Zhao et al. (2026). Every KD baseline uses the same teacher and the same prompt, so the comparison is fair.

W3: Threshold claim in Table 4

Thank you for this careful reading. We agree that Table 4 mixes the two rules a little, and we have stated this in the table. Actually, we do not claim that 0.10 is optimal, but we set it as a default hyper-parameter that we use for all three students without carefully selecting it for each model.

Following your comment, we ran an extra ablation on the annealing schedule.

Table R3: 1.7B, MATH-500

ρ\rhoεS\varepsilon_SSeedspass@1
With annealing (paper)0.100.01→0.0010.01 \to 0.001383.1±0.883.1 \pm 0.8
No annealing0.100.005181.8

Under the fixed thresholds shown in Table 4 of the paper and in the new experiment above, the scores stay between 81.2 and 82.2, so DKLD is not sensitive to the teacher threshold.


AuthorsOpenReview ↗

Response to Reviewer vv2E (Part 2 / 2): Q1–Q3

Q1: Teacher without the answer in its system prompt

Thank you for this question. We trained DKLD with a teacher that does not see the answer.

Table R4: 1.7B, MATH-500

Teacher sees the answerSeedspass@1
GRPO (no teacher)–378.2±0.978.2 \pm 0.9
ToDiyes381.3±1.181.3 \pm 1.1
DKLDno182.2
DKLDyes383.1±0.883.1 \pm 0.8

Without the answer, our method still outperforms the other baselines. Note that all baselines also receive the answer in their teacher prompt.

Q2: Region size over training

Thank you for this question. We measured the Mismatched Region with the same code as in our response to Reviewer cbHR (W2), on 256 on-policy rollouts per checkpoint with the Qwen3-8B teacher at εS=0.01\varepsilon_S = 0.01.

Table R5: Qwen3-8B teacher, εS=0.01\varepsilon_S = 0.01

StudentCheckpointPositions with non-empty GGTeacher mass in GGStudent mass in GG
Qwen3-1.7Binitial11.7%0.2460.0039
Qwen3-0.6Binitial10.0%0.2160.0054
Qwen3-0.6Bafter DKLD training13.0%0.2530.0045

GG does not shrink after training. For the 0.6B student, the share of positions with a non-empty GG grows from 10.0% to 13.0%, the teacher mass in GG stays around 0.25, and the student mass in GG stays below 0.006. The mismatch is therefore persistent rather than transient. Because of the time limit we could compare only the initial and final checkpoints, and we will add a full trace over training in the revision.

Q3: Correct-rollout mask vs. AWD gate

Thank you for this question. The mask is shared by all KD methods in Table 2, so it does not favour DKLD.

The mask and the gate also depend on each other. Our reward is binary, so an incorrect rollout has A^≤0\hat{A} \le 0 and its gate weight σ(−A^/τ)\sigma(-\hat{A}/\tau) is at least 0.5, while a correct rollout has a weight of at most 0.5. With KD on all rollouts plus AWD, the incorrect rollouts would get the largest KD weight. The mask removes them, and the gate then acts among the correct ones.

We hope these responses address your concerns. If anything is still unclear, please let us know and we will be glad to discuss it further.


Thank you very much,
Authors

Thank you for the authors' response. I will maintain my positive rating.

Thank you very much for taking the time to read our paper and response. We truly appreciate your feedback

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and AC,

We thank you for the time and effort you spent reviewing our paper. Below we summarize our contributions, the reviewers' main concerns and our responses. We then answer each reviewer one by one.

For simplicity, we refer to Reviewers vv2E, 7mx3 and cbHR as R1, R2 and R3 in this response.

Core Contributions of Our Work

  • Define a limitation about FKL in OPD. The FKL loss and its gradient concentrate on a small set of tokens. It makes the responses longer, causes truncation, and leads to accuracy collapse.
  • A new KD objective, DKLD. We decompose the KL into a Matched Region and a Mismatched Region. We use forward KL in the Matched Region and reverse KL in the Mismatched Region.
  • A framework to combine OPD with GRPO. The student rollouts of OPD also give the advantages for GRPO without extra computation needed, but the two gradients can conflict. The AWD gate reduces this conflict.

New Experiments during Rebuttal

  • FKL with a smaller KD weight (R1-W1). FKL with λKD=0.01\lambda_{\mathrm{KD}} = 0.01 still collapses (15.8 pass@1, 89.4% truncation) and DKLD keeps stable.
  • Controls for the AWD gate (R2-W3, R2-Q2, R3-W1, R3-Q1). A constant weight of 0.01 for DKLD scores below the AWD gate on both benchmarks. The gate changes AKL and ToDi by at most 1.1 points, and under the same configuration, DKLD is still the best.
  • Teacher without the answer (R1-Q1). We trained DKLD with a teacher whose prompt does not contain the answer. It reaches 82.2, which is still higher than the baselines, whose teachers do see the answer.
  • Different Thresholds (R1-W3, R3-W5). Due to time limitation in the rebuttal, we made an experiment that DKLD fixes εS=0.005\varepsilon_S = 0.005 and it reaches 81.8, compared with 83.1±0.883.1 \pm 0.8 with annealing. We also trained DKLD with two more teacher ratios.
  • Coding problem split (R3-W3, R3-Q3). Considering the concern about the split of dataset of LiveCodeBench, we retrained the methods on LiveCodeBench problems that do not overlap with the test problems.

New Discussions during Rebuttal

  • Motivation and contribution (R2-W1, R2-Q1). We added some discussion to further clarify our contribution in the manuscript. Overall, the KL objective is our main contribution. GRPO is the shared setting of all KD baselines, and AWD is a small module for this setting. We have revised the introduction.
  • Gradient statement in Section 3.1 (R3-W2, R3-Q2). R3 carefully observed that the multiplier is not detached in our implementation, and we have corrected it in the revision.
  • Headline comparison (R1-W1). The comparison in Figure 1 uses the checkpoints at step 80. We have stated this in Figure 1 and the introduction.
  • Mask and gate (R1-Q3). The correct-rollout mask is shared by all KD methods, and we explain how the mask and the gate work together.
  • Threshold rules in Table 4 (R1-W3, R3-W5). Table 4 mixes two rules. We have marked the main row in the table.
  • Student normalization in Algorithm 1 (R3-W4). The student is renormalized as the teacher. We have clarified this in Algorithm 1.

Thank you very much,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.