- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes C² OPSD, which combines PCMI Debiasing (PD) and Diversity-Aware Exploration (DAE) to mitigate correct-path coverage collapse in On-Policy Self-Distillation. PD corrects demonstration-induced preferences, while DAE encourages the preservation of diverse correct reasoning paths. The paper analyzes the debiasing effect under idealized conditions and evaluates the method on multiple science question-answering datasets. The method achieves better than the compared baselines.
Critical strengths
PD introduces a specific correction term to address PCMI-induced sharpening. Theorem 1 and Appendix A.2 derive the conditions for exact bias cancellation under idealized assumptions and discuss how standardization, clipping, and length normalization affect the correction strength, clarifying the scope of the theoretical results. The experiments cover two models and multiple science question-answering datasets, where the full method achieves good results. Component ablations also support the contributions of PD and DAE.
Critical weaknesses
- The debiasing derivation for PD is based on reverse KL, while Table 3 reports Generalized JSD as the divergence used in actual training. The paper does not fully clarify whether this difference affects the debiasing effect of PD.
- Figure 4 measures diversity across all valid responses without separating correct and incorrect ones. From this figure alone, it is unclear whether the increased diversity comes from more distinct correct reasoning paths.
Questions for authors
- What is the exact definition and parameterization of the Generalized JSD used in Table 3? Under this objective, do PCMI-induced sharpening and the correction effect of PD still follow the theoretical analysis?
- According to Eqs. (10)–(12), DAE provides no additional reward when a question has only two eligible correct responses. In practice, what proportion of training questions produce nonzero DAE advantages? Does this proportion vary substantially across datasets?
- If only correct responses are retained and the methods are compared on a per-question basis, do the diversity gains in Figure 4 still hold? Do the differences between these responses reflect distinct and valid reasoning paths?
5 comments
Responses to Reviewer wqf5 (2/2)
Q1: What is the exact definition and parameterization of the Generalized JSD used in Table 3? Under this objective, do PCMI-induced sharpening and the correction effect of PD still follow the theoretical analysis?
R3: In Table 3 (Table 5 in the revised manuscript), both SDPO and use weighted JSD with teacher weight . At each sampled response prefix, is the student’s next-token distribution and is the demonstration-conditioned EMA teacher’s distribution. We define
=(1-\alpha)\mathrm{KL}(P\|M)+\alpha\mathrm{KL}(Q\|M), \qquad M=(1-\alpha)P+\alpha Q.$$ The student and teacher weights are therefore 0.8 and 0.2 in both the mixture and the KL terms. Under generalized JSD, we theoretically validate that the correction effect of PD still holds and provide this proof in Appendix A.2.3 of the revised manuscript. For any $0<\alpha<1$, demonstration conditioning can still change the probability ratios between correct responses through a nonlinear function of PCMI. In the idealized sequence-level setting with a frozen base policy and a fixed demonstration distribution, the correction constructed for weighted JSD restores the base-policy probability ratios at $\gamma=1$. > Q2: What proportion of training questions produce non-zero DAE advantages? Does this proportion differ substantially across datasets? **R4:** We conducted a non-zero DAE advantage analysis on the 24 $C^2OPSD$ runs in Table 1. Specifically, we estimated the fraction of question groups with at least one positive DAE advantage using the logged proportion of groups with nonzero variance in their leave-one-out marginal contributions. The overall estimate was **64.0%** over steps 1–200 and **67.4%** over steps 11–200, after excluding the first 10 warm-up steps. As shown in the following table, Biology had the lowest rate for both models. Within each model, the gap between the highest and lowest dataset rates was about **8 percentage points** in both windows. | Dataset | Qwen3-1.7B (w/ warm-up) | Qwen3-1.7B (w/o warm-up) | Qwen3-4B (w/ warm-up) | Qwen3-4B (w/o warm-up) | |---|---:|---:|---:|---:| | Biology | 55.92% | 58.86% | 61.94% | 65.20% | | Chemistry | 64.17% | 67.54% | 67.65% | 71.22% | | Materials | 61.77% | 65.02% | 67.31% | 70.86% | | Physics | 63.92% | 67.28% | 69.67% | 73.33% |Dear Reviewer wqf5,
Thank you again for your valuable comments. As the discussion period is nearing its end, we would like to check whether our responses have addressed your concerns. If you have any further concerns or questions, please let us know.
Thank you for your time.
The Authors
Responses to Reviewer wqf5 (1/2)
Thank you for your helpful comments. We added experiments and revised our manuscript, with the revised parts highlighted in Red font. Please refer to the following responses for more details.
R1: We added empirical results and extended our theoretical analysis to clarify this potential effect.
Empirical evidence. As shown in Table 3 of the revised manuscript, we compared C2OPSD with SDPO under reverse KL. We report the results for the Biology and Physics datasets below. Under reverse KL, C2OPSD performs better than SDPO in most cases. We use JSD in Table 1 because it is the default distillation divergence in SDPO without feedback.
Theoretical scope. We also provided the proof of exact cancellation of PCMI-induced sharpening under generalized JSD (Theorem 2, Appendix A.2.3 of the revised manuscript). In this theorem, we offer a theoretical guarantee for PCMI debiasing under JSD. Please refer to R7 to Reviewer Nk7o for more details.
R2: For Figure 4 (Figure 5 in the revised manuscript), the reason for including incorrect responses is to assess whether DAE broadens the range of reasoning paths explored, regardless of final-answer correctness.
We recompute the diversity metrics using only verified-correct responses, and the results demonstrate that increased diversity derives from more distinct correct reasoning paths. As shown in the table below, C2OPSD with DAE still shows higher mean scores than SDPO on four reported diversity metrics (including a new metric). We have added the detailed results to Figure 6 of the revised manuscript and updated the accompanying analysis.
Additionally, we have already utilized several diversity metrics (1 - self-BELU, Mean cosine distance, DCScore, and stsb-roberta-base) to avoid the impact of different expressions of the same solution.