acceptodds
Under review as a conference paper at ICLR 2027

Junctions and Basins: Trajectory-Sensitive On-Policy Distillation for Language Models

Abstract

On-policy distillation (OPD) is an effective paradigm for transferring knowledge between language models, where a student learns from dense token-level supervision along its own trajectories. However, existing OPD methods typically apply uniform supervision across token positions, overlooking the substantial variation in their downstream consequences. We find that student trajectories exhibit two characteristic regimes: Junctions, where local variations strongly influence subsequent generation, and Basins, where plausible alternatives induce only limited changes to the future trajectory. To identify these regimes, we introduce a lightweight gradient-based future-sensitivity estimator that measures the directional sensitivity of the sampled future trajectory at each token position, together with a two-component mixture model that adaptively captures low- and high-sensitivity regions. Building on this structure, we propose Junction–Basin On-Policy Distillation (JB-OPD), which applies mode-seeking distillation at Junction-like positions while using mode-covering supervision with relational preservation at Basin-like positions. Across six mathematical reasoning benchmarks, JB-OPD consistently outperforms the state-of-the-art OPD baseline across Qwen3-0.6B, 1.7B, and 4B students, with average Pass@8 gains of 3.65%, 4.46%, and 4.39%, respectively. Further analysis reveals clear heterogeneity in downstream influence and shows that adapting distillation according to the inferred Junction–Basin roles consistently improves reasoning performance, supporting future trajectory sensitivity as an effective signal for allocating on-policy supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Large Language Models, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fQNAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Junction–Basin On-Policy Distillation (JB-OPD), which adapts token-level supervision according to an estimated token’s influence on the subsequent student-generated trajectory. It uses a gradient-based directional sensitivity score and a two-component mixture model to assign a soft Junction–Basin weight. High-sensitivity positions receive more reverse-KL supervision; low-sensitivity positions receive more forward-KL supervision and a teacher-preference regularizer.

Critical strengths

The paper asks a worthwhile question: whether supervision should depend on a token’s downstream role rather than only its local predictive distribution. The proposed score can be computed without a separate rollout for every token. The experiments cover three Qwen student sizes, a Llama teacher–student pair, and several out-of-domain tasks.

Critical weaknesses

The central interpretation is not yet established. The score measures how the likelihood of a fixed, already-sampled continuation changes under a small movement in embedding space; it does not measure what the model would actually generate after replacing a token, or whether the final answer would change.

More seriously, apparently equivalent no-relational-regularization settings report different results: Table 2’s reverse-KL/forward-KL row differs from Table 4’s corresponding ablation, while AMC23 Pass@8 is 77.50 in Table 4 but 75.00 for λR = 0 in Table 9. These discrepancies require an explanation before the ablation evidence can be relied upon.

The paper also reports no training-seed variability or confidence intervals; this matters particularly for AIME results reported in increments of roughly 3.33 percentage points.

Questions for authors

see the weakness above

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

W1 Validation of Future Sensitivity Through Explicit Token Swapping.

We thank the reviewer for this insightful comment. To further validate the interpretation of our future-sensitivity score, we conduct an additional comparison with explicit token swapping. Specifically, we replace each selected token with its highest-probability alternative and assess its actual influence on future generation.

As shown in Table 1, our gradient-based approach achieves comparable performance to explicit token swapping while avoiding the substantial computational overhead. Furthermore, we measure the Spearman rank correlation between gradient-based sensitivity scores and the actual effects of token swapping. As reported in Table 2, the two measures exhibit a strong positive correlation (average ρ≈0.75\rho \approx 0.75), suggesting that our estimator effectively captures the relative influence of token substitutions, despite relying on a computationally efficient first-order approximation.

Table 1: Comparison between explicit token swapping and gradient-based future-sensitivity estimation.

Sensitivity EstimationMATH500 Avg@8Pass@8AMC23 Avg@8Pass@8AIME24 Avg@8Pass@8AIME25 Avg@8Pass@8
Explicit Token Swapping70.3890.2043.5280.6311.4629.177.6022.50
Gradient-Based (Ours)70.2590.4043.7580.0011.6730.007.5023.33

Table 2: Spearman correlation between gradient-based sensitivity and explicit token swapping.

BenchmarkSpearman's ρ\rho ↑
MATH5000.78
AMC230.76
AIME240.72
AIME250.74
Average0.75

These findings provide complementary evidence that the proposed gradient-based score serves as an effective and efficient proxy for the relative impact of token substitutions. While it does not directly model the discrete regeneration process or guarantee changes in the final answer, its strong rank correlation with explicit token swapping and comparable downstream performance support its use for identifying influential positions in on-policy distillation.

W2 Clarification and Correction of Ablation Results.

We sincerely apologize for the inconsistencies in the reported ablation results and thank the reviewer for carefully identifying these issues. These discrepancies arose from errors in reporting the experimental results. We have carefully rechecked the experimental configurations and corrected the corresponding results in Tables 2, 4, and 9 to ensure consistency across equivalent settings, particularly regarding relational regularization. Importantly, these corrections do not affect our main conclusions regarding the effectiveness of Junction--Basin KL routing.

W3 Multi-Seed Evaluation and Statistical Reliability.

We thank the reviewer for emphasizing the importance of statistical reliability. We additionally conduct four independent training runs and report the mean and standard deviation of Avg@8 and Pass@8 across different seeds. As shown in Table 1 of the revised manuscript, JB-OPD maintains its overall performance advantage across multiple runs, demonstrating the robustness of our approach.

I have read the rebuttal and i keep the decision of Weak rejection

AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank the reviewer for taking the time to read our rebuttal and reconsider our submission. We respect your assessment and greatly appreciate your constructive feedback.

We would like to briefly summarize the additional efforts we have made to address your three main concerns:

  1. Validation of future sensitivity (W1): We conducted explicit token-swapping experiments to directly evaluate the effects of token substitutions and compared them with our gradient-based estimator. The results show a strong positive Spearman rank correlation (0.75 on average), together with comparable downstream performance, providing additional empirical support for our sensitivity estimation.

  2. Consistency of ablation results (W2): We carefully rechecked the experimental configurations and corrected the reporting inconsistencies across Tables 2, 4, and 9, ensuring that equivalent settings are reported consistently.

  3. Statistical reliability (W3): We conducted four independent training runs and reported the mean and standard deviation of Avg@8 and Pass@8, providing additional evidence of the stability of JB-OPD across different seeds.

We have incorporated these additional experiments and corrections into the revised manuscript. We hope these efforts help clarify the concerns raised in the original review.

We fully respect your decision and would be grateful if you could share any remaining concerns that have not been adequately addressed. We would be more than happy to provide further clarification or additional evidence.

Thank you again for your time and thoughtful consideration.

Reviewer_a7RdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

On-Policy Distillation methods usually apply uniform supervision across token positions. This paper identifies two characteristic regimes for student generations: Junctions, where local variations can strongly alter subsequent generation and Basins, where local variations induce limited changes to subsequent generation. The paper further proposes a light-weight gradient-based future sensitivity estimator to identify low and high sensitivity regions in the student trajectory. Their proposed method, Junction-Basin OPD, applies mode-seeking/mode-covering supervision at Junction/Basin-like tokens. Experiments on six mathematical reasoning benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  • The brute-force way of estimating token-level future sensitivity score requires L forward passes for an L token length sequence. Section 3.2 introduces a lightweight estimator for identifying Junction/Basin token positions in a single backward pass. This makes the approach practically viable.

  • Experiments span three Qwen and two Llama family models of varying sizes on six mathematical reasoning benchmarks and show improvements in pass@8 against compared baselines.

Critical weaknesses
  • The future-sensitivity score C_t measures how the trajectory objective changes to an infinitesimal perturbation in the direction of the highest-probability alternative token. The paper does not establish that this score correlates with an actual token swap continued by regeneration of the sequence from that position. Missing ablation studies to support the importance of ζ_t including: i) shuffle the ζ_t score, ii) assign random ζ_t score, iii) compare against OPD that uses Jeffreys Divergence (0.5RKL + 0.5FKL) instead of just RKL.

  • Avg@8 gains over baselines are marginal. Missing standard deviation makes it difficult to assess if gains are within noise.

Questions for authors
  1. Can the authors compare against relevant and missing baselines [1, 2]?
  2. Qwen3 models recommend using the following sampling parameters: Temperature=0.6, TopP=0.95, TopK=20, and MinP=0. Can the authors report Avg@8 using this config?
  3. For table 1 JB-OPD row, can the authors also report the raw values (total problems passed, denominator) for Pass@8?




References



[1] Xu et. al., TIP: Token Importance in On-Policy Distillation, 2026


[2] Zhang et. al., Reinforcement-aware Knowledge Distillation for LLM Reasoning, 2026

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

(W1) Validation of Future Sensitivity and Alternative Weighting Strategies.

We thank the reviewer for this valuable suggestion. To validate our gradient-based future-sensitivity estimation, we additionally compare it with explicit token swapping. As shown in Table 1, our method achieves comparable performance while avoiding the substantial computational overhead of repeated forward evaluations.

Table 1: Comparison between explicit token swapping and gradient-based future-sensitivity estimation.

Sensitivity EstimationMATH500 Avg@8Pass@8AMC23 Avg@8Pass@8AIME24 Avg@8Pass@8AIME25 Avg@8Pass@8
Explicit Token Swapping70.3890.2043.5280.6311.4629.177.6022.50
Gradient-Based (Ours)70.2590.4043.7580.0011.6730.007.5023.33

We further examine the correlation between gradient-based sensitivity scores and the actual effects of token swapping. As shown in Table 2, the two measures exhibit a strong positive rank correlation (average Spearman's ρ≈0.75\rho \approx 0.75), suggesting that our gradient-based estimator effectively captures the relative influence of token substitutions on subsequent generation.

Table 2: Spearman correlation between gradient-based sensitivity and explicit token swapping.

BenchmarkSpearman's ρ\rho ↑
MATH5000.78
AMC230.76
AIME240.72
AIME250.74
Average0.75

Furthermore, we introduce additional weighting baselines, including fixed KL mixtures, random weighting, shuffled Junction posteriors, and global weighting. In particular, the fixed mixture with ζ=0.5\zeta = 0.5 corresponds to Jeffreys divergence. As shown in Table 3, JB-OPD achieves the strongest overall performance, demonstrating that its improvements stem from sensitivity-aware token-specific KL routing rather than simply combining RKL and FKL or assigning heterogeneous weights.

Table 3: Comparison of alternative KL weighting strategies.

Weighting StrategyMATH500 Avg@8Pass@8AMC23 Avg@8Pass@8AIME24 Avg@8Pass@8AIME25 Avg@8Pass@8
Jeffreys (ζ=0.5\zeta = 0.5)67.7085.5039.3872.509.0623.335.8317.50
Random Weighting66.9583.9037.9767.507.7119.174.9013.33
Shuffled67.3084.8038.4469.388.3321.675.6216.67
**JB-OPD **70.2590.4043.7580.0011.6730.007.5023.33

Overall, these results provide complementary evidence for the proposed method: explicit token swapping validates the practical effectiveness of our efficient sensitivity approximation, the rank-correlation analysis supports its ability to identify influential token positions, and the weighting ablations highlight the importance of sensitivity-aware Junction--Basin routing.

(W2) Multi-Seed Evaluation. We thank the reviewer for this suggestion. We additionally conduct four independent training runs and report the mean and standard deviation of Avg@8 and Pass@8 across different seeds. As shown in Table 1 of the revised manuscript, JB-OPD consistently achieves stronger overall performance than the competing methods. These results further support the robustness and reliability of our proposed method.

( Q1) Comparison with Additional Baselines.

We thank the reviewer for this suggestion. We additionally compare JB-OPD with the relevant methods [1, 2], as shown in Table 1 of the revised manuscript. Unlike these methods, which do not explicitly distinguish token positions based on their future influence, JB-OPD adaptively assigns different KL objectives to Junction-like and Basin-like positions. The results demonstrate that our method achieves superior overall performance, further highlighting the effectiveness of sensitivity-aware token-level distillation.

(Q2) Evaluation with Recommended Sampling Configuration.

We thank the reviewer for pointing this out. We additionally evaluate all methods using the recommended Qwen3 sampling configuration (Temperature = 0.6, TopP = 0.95, TopK = 20, and MinP = 0). As reported in Table 13 of the revised manuscript, JB-OPD consistently achieves strong Avg@8 performance and maintains its advantage over competing methods, further demonstrating the robustness of our approach under the recommended sampling settings.

(Q3) Raw Pass@8 Statistics.

We thank the reviewer for this suggestion. We have additionally included the raw Pass@8 statistics for JB-OPD, reporting the total number of successfully passed problems for each benchmark. These values are now provided in Table 11 of the revised manuscript.

Thank you authors for your response and the revised manuscript. The additional ablations (fixed, random and shuffled weighting), the TIP and RLAD baselines, and the results with the recommended sampling settings are appreciated. However, some of my concerns remain.

  1. Multiple seed results. In the revised Table 1, the means for all methods present in the original submission, including JB-OPD, are identical to the single-run numbers reported there, apart from minor changes. Similarly, the Table 11 totals for these methods correspond to four times the original single-run counts. Only the newly added TIP and RLAD rows show the variation expected from four independent runs. Can the authors clarify which methods were re-run with multiple seeds? I would ask the authors to share the per-run Avg@8 and Pass@8 values, and the per-run number of problems passed, for every method in Table 1.

  2. Validation of the future-sensitivity score. The Spearman correlation between C_t and explicit token swapping reported in the rebuttal does not seem to be included in the revised manuscript. Can the authors add it and describe how the swap effect is measured, how many positions and sequences were used, and whether the correlation holds within a sequence after controlling for token position? Since C_t sums gradients over all subsequent tokens, earlier positions may naturally score higher. For Appendix D.5, can the authors also report the compute cost of explicit token swapping and regeneration during training?

  3. Changed ablation results. The RKL/FKL row in Table 2 and the AMC23 Pass@8 value in Table 4 have changed from the original submission without explanation. Can the authors clarify the reason for these changes?

  4. Missing baseline details (hyp-params). The authors should provide implementation details and hyper-parameters for TIP and RLAD.

AuthorsOpenReview ↗

Rebuttal by Authors

W1 Multiple seed results..

We thank the reviewer for this comment. We clarify that all methods reported in Table 1 were evaluated across four independent runs with different random seeds. To improve transparency and reproducibility, we have added Table 16,17,18 to the revised manuscript, which provides the complete per-run results for all methods, student model scales, and benchmarks. Specifically, we report Avg@8, Pass@8, and the number of problems passed in each individual run.

W2 Validation of the future-sensitivity score.

We thank the reviewer for this suggestion. We have included the validation results and computational analysis in the revised manuscript. Swap-effect measurement. We replace each selected token with an alternative token while keeping the subsequent tokens unchanged, and measure the change in the Student's cumulative downstream log-probability. Sequences and positions. We evaluate 100, 40, 30, and 30 student-generated sequences from MATH500, AMC23, AIME24, and AIME25, respectively, sampling 100 token positions per sequence. Position-controlled correlation. The average Spearman correlation is 0.75 across the four benchmarks. After controlling for normalized token position within each sequence, the average correlation remains 0.70, indicating that the future-sensitivity score captures token-specific influence beyond positional effects. Computational cost. We have extended Appendix D.5 to compare the computational overhead of different sensitivity estimation strategies. Under the Qwen3-1.7B-Base + MATH configuration, JB-OPD requires only 1.4 s/step for future-sensitivity estimation, compared with approximately 185 s/step for explicit token swapping . Including the additional distillation terms, the corresponding total training times are 58.4, and 242.0 s/step, respectively.

W3 Changed Ablation Results

We sincerely apologize for the inconsistencies in the reported ablation results and thank the reviewer for carefully identifying these issues. These discrepancies were caused by errors in experimental result reporting, which have now been corrected in the revised manuscript.

W4 Missing Baseline Details

We thank the reviewer for this suggestion. We have added the implementation details and training hyperparameters for TIP and RLAD, including learning rate, optimizer, batch size, training epochs, and other relevant configurations, as reported in Table 5 of the revised manuscript.

Reviewer_KoVKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes token level supervision based on the formulated sensitivity of each position, categorizing positions as Junction like (more sensitive) or Basin like (less sensitive), and proposes a weighted loss function combining mode seeking and mode covering supervision. The weights are calculated from a future sensitivity score, which estimates how strongly a token position affects the subsequent student trajectory. The log-transformed scores are modeled with a two component GMM, whose posterior probabilities provide soft Junction Basin weights. The results suggest that JB-OPD is useful however, it is not quite clear whether the reported gains come from the future sensitivity modeling or mostly from combining RKL and FKL. The existing ablation is helpful, but I would like to see more experiments which I'll mention in the next sections.

Critical strengths

1-I believe the strongest point about the paper is that it is well-written, without unnecessary complicating the write-up and overall understandable and easy to follow. Also, the motivation behind the formulation is clear and well-stated. 2- The weighted RKL and KL combination has been used widely in the past, but this paper proposes an "interpretable" weighting mechanism that directly formulates the future sensitivity of token positions.

Critical weaknesses

1-The trajectory sensitive routing idea is interesting, and the results provide useful evidence that the method actually works. However, my main concern is that the experiments do not sufficiently prove the main contribution i.e. whether future sensitivity formulation and GMM modeling improve over a well tuned, weighted combination of the same distillation losses. More controlled experiments such as constant mixture baseline and shuffled posterior would directly test this, whereas the existing ablations only focus on routing and supervision. 2- The finetuning details need more clarification and limited reported training setups (e.g. full finetuning s PEFT methods like LoRA) further weaken confidence in the success of the JB-OPD.

Questions for authors
  1. Compare with a simple combination of FKL and RKL.

a) I'd like to see eq. (13) compared with a baseline that uses a combination weight shared across all token positions, either constant throughout training or adjusted according to a scheduler (e.g., by training stage), while keeping the basin supervision in eq. (12). Tune the constant weight or schedule on held-out validation data, and keep the rest of experimental configuration including teacher's top-K support, relational regularization, training settings the same as JB-OPD.

b) Table 2 compares uniform RKL and FKL, but not a tuned combination of the two, and it disables relational regularization. Uniform OPD in Table 3 also does not use the same combination of losses. The adaptive baselines in table 10 are useful, but a matched constant mixture baseline would more directly show the benefits of the proposed routing.

c)To measure the effectiveness of token level supervision, another useful control is to give every position the mean junction posterior for its trajectory. This keeps the overall mixing weight but removes token specific supervision.

  1. Check whether assigning the Junction posteriors to the right tokens matters.

Compute the Junction posteriors using eq. (9), then shuffle them across nonterminal positions within each trajectory before applying eq. (13). Keep the final token under standard OPD supervision, as in the paper. This keeps the same set of weights but breaks their link to each token's future sensitivity. It would help show whether JB-OPD benefits from identifying junction and basin positions, rather than simply giving different weights to different tokens. The hard vs. soft comparison in Table 3 does not answer this, since both settings use sensitivity based estimations.

  1. Compare performance and runtime with the simpler baselines.

The paper reports 1.4 seconds per step for future sensitivity estimation and 9.2 seconds for the additional distillation terms. These are different costs, so the full 22.2% overhead should not be attributed only to the estimator. Reporting the performance and runtime of the constant mixture baseline would make it easier to judge whether the estimator and GMM routing are worth the overhead cost.

  1. Clarify which student parameters are updated.

I could not find whether the experiments use full-parameter fine-tuning or LoRA. Please state this explicitly. If LoRA or other adapters are used, report their settings and target modules, and confirm that the setup is consistent across baselines.

  1. Repeat the key loss ablations on the Llama pair.

The Llama results in Table 7 are useful evidence that JB-OPD works beyond Qwen. However, the KL-direction ablations in Table 2 are limited to Qwen3-1.7B-Base. Do Junction-like positions still favor RKL, and Basin-like positions FKL, for the Llama pair? Repeating this ablation and the constant mixture comparison would help answer that question. This matters because the Qwen students are Base models, while the Llama student is instruction-tuned, and the teacher also changes. The current experiments do not separate model family architecture effects from initialization or teacher student compatibility.

  1. Examine how the junction/basin roles change during training.

a) Section 3.3 acknowledges that sensitivity scores can change across training stages. Since the scores depend on the current student, the inferred roles can change too. However, the paper does not show how these roles evolve, or whether the same RKL/FKL assignment remains appropriate throughout training. Please examine the scores at early, middle, and late checkpoints. Using a fixed set of prompt prefix pairs would help separate changes in the student from changes in its generated trajectories. A baseline whose mixing weight changes with training stage, but is shared across token positions, would also be useful.

  1. Report variation across training runs.

The key comparisons would be more convincing with multiple training seeds and uncertainty estimates. This would help establish whether small differences between routing strategies are reliable.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/2]

(W1 & Q1 Q2 ) Ablation of Adaptive KL Weighting Strategies and Token-Specific Junction--Basin Routing.

Thank you for this valuable suggestion. To isolate the contribution of token-level adaptive supervision, we introduce three additional baselines while keeping all other training settings identical to JB-OPD. The results are presented in Table 1. (a) Constant Mixture. We evaluate fixed RKL–FKL mixing coefficients of 0.3, 0.5, and 0.7, shared across all token positions. As shown in Table 1, these variants consistently underperform JB-OPD. This indicates that a fixed mixture cannot adequately capture the heterogeneous future sensitivities of individual tokens, particularly the distinction between trajectory-critical Junction-like positions and more tolerant Basin-like positions. (b) Scheduled Mixture. We further consider a training-stage-dependent schedule that adjusts the global mixing coefficient during optimization. Although this strategy accounts for changes across training stages, it still applies the same supervision mixture to all positions within each stage. Its inferior performance suggests that stage-level adaptation alone cannot replace token-specific routing, as Junction-like and Basin-like positions may coexist within the same reasoning trajectory. (c) Mean Junction Posterior. We replace each token's Junction posterior with the mean posterior of its trajectory, preserving the overall mixing balance while removing position-specific information. This baseline also underperforms JB-OPD, demonstrating that the improvement stems not merely from the average RKL–FKL proportion, but from aligning supervision with each token's future sensitivity.

Overall, these results demonstrate that the effectiveness of JB-OPD goes beyond combining RKL and FKL or adjusting their global proportions. Instead, its advantage lies in sensitivity-aware token-level routing, which emphasizes mode-seeking supervision at trajectory-critical Junction positions while retaining mode-covering, relation-preserving supervision at Basin positions.

Table 1: Comparison of different Junction–Basin routing and weighting strategies using Qwen3-1.7B-Base.

Routing StrategyMATH500 Avg@8Pass@8AMC23 Avg@8Pass@8AIME24 Avg@8Pass@8AIME25 Avg@8Pass@8
Fixed (0.30.3)67.4085.1038.6770.008.6522.505.5215.83
Fixed (0.50.5)67.7085.5039.3872.509.0623.335.8317.50
Fixed (0.70.7)67.1584.5038.2868.128.1220.005.1014.17
Random66.9583.9037.9767.507.7119.174.9013.33
Shuffle67.3084.8038.4469.388.3321.675.6216.67
Validation Schedule68.0586.0039.8473.759.2724.176.0418.33
Mean Posterior67.8585.8039.5373.129.1725.005.9417.50
Ours70.2590.4043.7580.0011.6730.007.5023.33

(Q3) Performance and Computational Efficiency Comparison. Thank you for this valuable comment. Compared with JB-OPD, constant mixture baselines directly assign a predefined RKL–FKL mixing coefficient to all token positions, without requiring future-sensitivity estimation or GMM-based routing. Therefore, their additional computational cost mainly comes from the distillation objectives themselves. However, as shown in Table 2, these constant mixture strategies consistently underperform JB-OPD. This is because a globally shared mixing coefficient overlooks the heterogeneous future sensitivities of individual tokens, treating trajectory-critical Junction positions and more tolerant Basin positions identically. In contrast, JB-OPD explicitly estimates token-level future sensitivity to adaptively allocate RKL and FKL supervision. These results suggest that the additional computational cost of sensitivity estimation provides meaningful performance benefits by enabling more precise, trajectory-aware distillation.

Table 2: Comparison of Constant RKL–FKL Mixture Strategies with Token-Adaptive JB-OPD.

Routing StrategyMATH500 Avg@8Pass@8AMC23 Avg@8Pass@8AIME24 Avg@8Pass@8AIME25 Avg@8Pass@8
Fixed (0.30.3)67.4085.1038.6770.008.6522.505.5215.83
Fixed (0.50.5)67.7085.5039.3872.509.0623.335.8317.50
Fixed (0.70.7)67.1584.5038.2868.128.1220.005.1014.17
Random66.9583.9037.9767.507.7119.174.9013.33
Ours70.2590.4043.7580.0011.6730.007.5023.33
AuthorsOpenReview ↗

Rebuttal by Authors [2/2]

( Q4) Clarification of Student Fine-Tuning Configuration. We thank the reviewer for raising this point and apologize for the lack of clarity. Following EOPD and other related works, we adopt full-parameter fine-tuning for all student models, without using LoRA or other parameter-efficient adapters, to ensure a fair and consistent comparison across all methods. We will explicitly state this in the revised manuscript's implementation details / experimental setup section.

( Q5) Generalization of KL-Direction Preferences to Llama. We thank the reviewer for this valuable suggestion. We extend the KL-direction ablation to Llama-3.2-3B-Instruct with Llama-3.1-8B-Instruct as the teacher, following the same settings as Table 2. The results are presented in Table 3. Consistent with the Qwen results, applying RKL to Junction-like positions and FKL to Basin-like positions achieves the strongest performance across all four benchmarks. In contrast, using the same KL direction for both regions is less effective, while reversing the assignment leads to further degradation. These findings suggest that high-sensitivity Junctions benefit from mode-seeking supervision, whereas low-sensitivity Basins favor mode-covering supervision that preserves plausible alternatives. The consistent trends across Qwen and Llama further support the generalizability of our role-specific distillation strategy. The corresponding revisions have been incorporated into the revised manuscript and are highlighted in red for clarity.

**Table 3: Effect of different distillation strategies for Junction-like and Basin-like positions using Llama-3.2-3B-Instruct as the student and Llama-3.1-8B-Instruct as the teacher. Relational regularization is disabled to isolate the effect of KL direction. **

JunctionBasinMATH500 Avg@8Pass@8AMC23 Avg@8Pass@8AIME24 Avg@8Pass@8AIME25 Avg@8Pass@8
RKLRKL45.0068.0020.8651.252.6011.671.778.33
FKLFKL44.6367.6020.1649.382.9212.501.567.50
FKLRKL44.1966.5019.5348.132.4010.001.466.67
RKLFKL45.8169.0022.5853.753.4415.002.5011.67

( Q6) Evolution of Junction--Basin Roles During Training.

We thank the reviewer for this suggestion. We clarify that JB-OPD employs continuous posterior-based weighting rather than hard Junction--Basin assignments. We analyze early (10%), middle (50%), and late (90%) checkpoints using the same fixed prompt--trajectory pairs. As shown in Table 4, the mean Junction posterior remains relatively stable across training stages, while the mean absolute posterior change increases from 0.118 to 0.164. These observations suggest that the overall KL mixing proportion remains stable, whereas token-specific supervision weights evolve as the student is updated, supporting our adaptive routing design.

Table 4: Evolution of Junction--Basin Posteriors During Training.

Training StageMean Junction PosteriorMean Absolute Posterior Change vs. Early
Early (10%)0.412—
Middle (50%)0.4260.118
Late (90%)0.4380.164

( Q7) Multi-Seed Evaluation. We thank the reviewer for this suggestion. We conduct four independent training runs and report the mean and standard deviation of Avg@8 and Pass@8 across different seeds. As shown in Table 1 of the revised manuscript, JB-OPD consistently achieves stronger overall performance than the competing methods. These results further support the robustness and reliability of our proposed method.

I appreciate the authors for their detailed rebuttal and the additional experiments. The new controls largely address my main concern by showing that the gains come from token-specific routing rather than simply from combining RKL and FKL, and the clarifications on fine-tuning and the Llama ablation are helpful. However, some concerns remain about the missing runtime comparison with simpler baselines and the lack of seed variation for the new ablations. Overall, I am raising my score from weak reject to weak accept.

We sincerely thank the reviewer for the positive assessment and for raising the score from weak reject to weak accept. We greatly appreciate the constructive feedback, which has helped us further improve the manuscript.

To address the remaining concerns, we have further expanded our analysis in two aspects.

First, regarding computational efficiency, we provide an additional comparison with the simpler constant-weighting baseline. Unlike JB-OPD, constant weighting does not require future-sensitivity estimation or GMM-based routing, while retaining the same distillation objectives. The training cost of constant weighting is 57.0 s/step (19.2% overhead over OPD), compared with 58.4 s/step (22.2% overhead) for JB-OPD.

Second, regarding the stability of the new ablations, we have expanded the presentation of the Junction–Basin routing and weighting strategies and the Llama-based KL-direction ablation. The revised tables retain the corresponding performance comparisons and consistently report the mean and standard deviation, providing a more comprehensive assessment of the stability and robustness of the proposed method. We sincerely appreciate the reviewer's thoughtful suggestions and recognition of our improvements.

Table 1: Comparison of different Junction–Basin routing and weighting strategies using Qwen3-1.7B-Base

Routing StrategyMATH500 Avg@8Pass@8AMC23 Avg@8Pass@8AIME24 Avg@8Pass@8AIME25 Avg@8Pass@8
Fixed (0.3)67.40 ± 0.985.10 ± 0.838.67 ± 1.070.00 ± 2.98.65 ± 1.022.50 ± 4.25.52 ± 1.815.83 ± 3.2
Fixed (0.5)67.70 ± 0.785.50 ± 0.939.38 ± 0.972.50 ± 2.99.06 ± 0.823.33 ± 3.85.83 ± 1.617.50 ± 3.2
Fixed (0.7)67.15 ± 1.084.50 ± 0.638.28 ± 1.168.12 ± 3.88.12 ± 1.120.00 ± 4.75.10 ± 2.014.17 ± 4.2
Random66.95 ± 1.183.90 ± 1.037.97 ± 1.267.50 ± 4.17.71 ± 1.319.17 ± 5.04.90 ± 2.213.33 ± 3.8
Shuffle67.30 ± 0.984.80 ± 0.838.44 ± 1.069.38 ± 3.18.33 ± 1.021.67 ± 4.35.62 ± 1.916.67 ± 3.8
Validation Schedule68.05 ± 0.886.00 ± 0.739.84 ± 0.973.75 ± 3.29.27 ± 0.924.17 ± 3.26.04 ± 1.518.33 ± 3.3
Mean Posterior67.85 ± 0.785.80 ± 0.939.53 ± 0.873.12 ± 3.19.17 ± 0.825.00 ± 4.35.94 ± 1.717.50 ± 3.2
Ours70.25 ± 0.590.40 ± 0.543.75 ± 0.680.00 ± 2.011.67 ± 0.330.00 ± 2.77.50 ± 0.523.33 ± 2.7

Table 3: Effect of different distillation strategies for Junction-like and Basin-like positions using Llama-3.2-3B-Instruct as the student and Llama-3.1-8B-Instruct as the teacher. Relational regularization is disabled to isolate the effect of KL direction.

JunctionBasinMATH500 Avg@8Pass@8AMC23 Avg@8Pass@8AIME24 Avg@8Pass@8AIME25 Avg@8Pass@8
RKLRKL45.00 ± 0.968.00 ± 0.820.86 ± 1.151.25 ± 3.22.60 ± 0.811.67 ± 3.31.77 ± 0.78.33 ± 3.3
FKLFKL44.63 ± 0.867.60 ± 0.920.16 ± 1.049.38 ± 3.82.92 ± 0.912.50 ± 3.21.56 ± 0.67.50 ± 3.2
FKLRKL44.19 ± 1.066.50 ± 1.119.53 ± 1.248.12 ± 3.82.40 ± 1.010.00 ± 3.81.46 ± 0.86.67 ± 3.8
RKLFKL45.81 ± 0.769.00 ± 0.622.58 ± 0.853.75 ± 3.23.44 ± 0.615.00 ± 3.32.50 ± 0.511.67 ± 1.9

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.