- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Junction–Basin On-Policy Distillation (JB-OPD), which adapts token-level supervision according to an estimated token’s influence on the subsequent student-generated trajectory. It uses a gradient-based directional sensitivity score and a two-component mixture model to assign a soft Junction–Basin weight. High-sensitivity positions receive more reverse-KL supervision; low-sensitivity positions receive more forward-KL supervision and a teacher-preference regularizer.
Critical strengths
The paper asks a worthwhile question: whether supervision should depend on a token’s downstream role rather than only its local predictive distribution. The proposed score can be computed without a separate rollout for every token. The experiments cover three Qwen student sizes, a Llama teacher–student pair, and several out-of-domain tasks.
Critical weaknesses
The central interpretation is not yet established. The score measures how the likelihood of a fixed, already-sampled continuation changes under a small movement in embedding space; it does not measure what the model would actually generate after replacing a token, or whether the final answer would change.
More seriously, apparently equivalent no-relational-regularization settings report different results: Table 2’s reverse-KL/forward-KL row differs from Table 4’s corresponding ablation, while AMC23 Pass@8 is 77.50 in Table 4 but 75.00 for λR = 0 in Table 9. These discrepancies require an explanation before the ablation evidence can be relied upon.
The paper also reports no training-seed variability or confidence intervals; this matters particularly for AIME results reported in increments of roughly 3.33 percentage points.
Questions for authors
see the weakness above
3 comments
-
I have read the rebuttal and i keep the decision of Weak rejection
Rebuttal by Authors
We sincerely thank the reviewer for taking the time to read our rebuttal and reconsider our submission. We respect your assessment and greatly appreciate your constructive feedback.
We would like to briefly summarize the additional efforts we have made to address your three main concerns:
-
Validation of future sensitivity (W1): We conducted explicit token-swapping experiments to directly evaluate the effects of token substitutions and compared them with our gradient-based estimator. The results show a strong positive Spearman rank correlation (0.75 on average), together with comparable downstream performance, providing additional empirical support for our sensitivity estimation.
-
Consistency of ablation results (W2): We carefully rechecked the experimental configurations and corrected the reporting inconsistencies across Tables 2, 4, and 9, ensuring that equivalent settings are reported consistently.
-
Statistical reliability (W3): We conducted four independent training runs and reported the mean and standard deviation of Avg@8 and Pass@8, providing additional evidence of the stability of JB-OPD across different seeds.
We have incorporated these additional experiments and corrections into the revised manuscript. We hope these efforts help clarify the concerns raised in the original review.
We fully respect your decision and would be grateful if you could share any remaining concerns that have not been adequately addressed. We would be more than happy to provide further clarification or additional evidence.
Thank you again for your time and thoughtful consideration.
Rebuttal by Authors
W1 Validation of Future Sensitivity Through Explicit Token Swapping.
We thank the reviewer for this insightful comment. To further validate the interpretation of our future-sensitivity score, we conduct an additional comparison with explicit token swapping. Specifically, we replace each selected token with its highest-probability alternative and assess its actual influence on future generation.
As shown in Table 1, our gradient-based approach achieves comparable performance to explicit token swapping while avoiding the substantial computational overhead. Furthermore, we measure the Spearman rank correlation between gradient-based sensitivity scores and the actual effects of token swapping. As reported in Table 2, the two measures exhibit a strong positive correlation (average ρ≈0.75), suggesting that our estimator effectively captures the relative influence of token substitutions, despite relying on a computationally efficient first-order approximation.
Table 1: Comparison between explicit token swapping and gradient-based future-sensitivity estimation.
Table 2: Spearman correlation between gradient-based sensitivity and explicit token swapping.
These findings provide complementary evidence that the proposed gradient-based score serves as an effective and efficient proxy for the relative impact of token substitutions. While it does not directly model the discrete regeneration process or guarantee changes in the final answer, its strong rank correlation with explicit token swapping and comparable downstream performance support its use for identifying influential positions in on-policy distillation.
W2 Clarification and Correction of Ablation Results.
We sincerely apologize for the inconsistencies in the reported ablation results and thank the reviewer for carefully identifying these issues. These discrepancies arose from errors in reporting the experimental results. We have carefully rechecked the experimental configurations and corrected the corresponding results in Tables 2, 4, and 9 to ensure consistency across equivalent settings, particularly regarding relational regularization. Importantly, these corrections do not affect our main conclusions regarding the effectiveness of Junction--Basin KL routing.
W3 Multi-Seed Evaluation and Statistical Reliability.
We thank the reviewer for emphasizing the importance of statistical reliability. We additionally conduct four independent training runs and report the mean and standard deviation of Avg@8 and Pass@8 across different seeds. As shown in Table 1 of the revised manuscript, JB-OPD maintains its overall performance advantage across multiple runs, demonstrating the robustness of our approach.