acceptodds
Under review as a conference paper at ICLR 2027

Fair Credit Across Correct Modes: Mitigating Correct-Path Coverage Collapse in On-Policy Self-Distillation for Language Models

Abstract

On-policy self-distillation (OPSD) is a promising approach for improving reasoning in large language models through dense supervision from a context-augmented version of the model. However, OPSD with sampled demonstrations (OPSD-SD) can achieve strong pass@1 accuracy while showing limited gains from additional rollouts, a phenomenon we call correct-path coverage collapse. Beyond general mode-seeking tendencies, demonstration conditioning in OPSD-SD can exacerbate this collapse through pointwise conditional mutual information (PCMI)-induced sharpening, further favoring already-probable correct solutions. To this end, we propose \ours, an OPSD framework that combines PCMI Debiasing (PD) and Diversity-Aware Exploration (DAE). PD corrects demonstration-dependent preferences among equally correct responses. Under idealized conditions, we show that this correction exactly cancels the additional PCMI-induced sharpening between correct responses. DAE then assigns credit according to each correct path’s marginal contribution to diversity, encouraging underrepresented paths. Experiments across multiple datasets demonstrate that \ours mitigates correct-path coverage collapse while outperforming strong baselines on both Pass@1 and Pass@.

Primary area:
generative models
Keywords:
On-policy self-distillation, Collapse, Diversity, Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wqf5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes C² OPSD, which combines PCMI Debiasing (PD) and Diversity-Aware Exploration (DAE) to mitigate correct-path coverage collapse in On-Policy Self-Distillation. PD corrects demonstration-induced preferences, while DAE encourages the preservation of diverse correct reasoning paths. The paper analyzes the debiasing effect under idealized conditions and evaluates the method on multiple science question-answering datasets. The method achieves better than the compared baselines.

Critical strengths

PD introduces a specific correction term to address PCMI-induced sharpening. Theorem 1 and Appendix A.2 derive the conditions for exact bias cancellation under idealized assumptions and discuss how standardization, clipping, and length normalization affect the correction strength, clarifying the scope of the theoretical results. The experiments cover two models and multiple science question-answering datasets, where the full method achieves good results. Component ablations also support the contributions of PD and DAE.

Critical weaknesses
  • The debiasing derivation for PD is based on reverse KL, while Table 3 reports Generalized JSD as the divergence used in actual training. The paper does not fully clarify whether this difference affects the debiasing effect of PD.
  • Figure 4 measures diversity across all valid responses without separating correct and incorrect ones. From this figure alone, it is unclear whether the increased diversity comes from more distinct correct reasoning paths.
Questions for authors
  • What is the exact definition and parameterization of the Generalized JSD used in Table 3? Under this objective, do PCMI-induced sharpening and the correction effect of PD still follow the theoretical analysis?
  • According to Eqs. (10)–(12), DAE provides no additional reward when a question has only two eligible correct responses. In practice, what proportion of training questions produce nonzero DAE advantages? Does this proportion vary substantially across datasets?
  • If only correct responses are retained and the methods are compared on a per-question basis, do the diversity gains in Figure 4 still hold? Do the differences between these responses reflect distinct and valid reasoning paths?
5 comments
AuthorsOpenReview ↗

Responses to Reviewer wqf5 (1/2)

Thank you for your helpful comments. We added experiments and revised our manuscript, with the revised parts highlighted in Red font. Please refer to the following responses for more details.

W1: It remains unclear how using Generalized JSD in place of reverse KL affects PD’s debiasing effect.

R1: We added empirical results and extended our theoretical analysis to clarify this potential effect.

Empirical evidence. As shown in Table 3 of the revised manuscript, we compared C2OPSDC^2OPSD with SDPO under reverse KL. We report the results for the Biology and Physics datasets below. Under reverse KL, C2OPSDC^2OPSD performs better than SDPO in most cases. We use JSD in Table 1 because it is the default distillation divergence in SDPO without feedback.

ModelMethodBiology Pass@1 (%)Biology Pass@16 (%)Physics Pass@1 (%)Physics Pass@16 (%)
Qwen3-1.7BSDPO31.33 ± 2.8838.00 ± 3.4646.64 ± 1.8067.50 ± 2.50
Qwen3-1.7BC2OPSDC^2OPSD36.54 ± 2.1244.67 ± 4.1647.14 ± 2.0469.58 ± 3.15
Qwen3-4BSDPO42.96 ± 1.2550.00 ± 4.0053.96 ± 0.5967.92 ± 1.91
Qwen3-4BC2OPSDC^2OPSD40.00 ± 8.3150.00 ± 5.2956.20 ± 3.4373.75 ± 3.31

Theoretical scope. We also provided the proof of exact cancellation of PCMI-induced sharpening under generalized JSD (Theorem 2, Appendix A.2.3 of the revised manuscript). In this theorem, we offer a theoretical guarantee for PCMI debiasing under JSD. Please refer to R7 to Reviewer Nk7o for more details.

W2/Q3: Figure 4 pools correct and incorrect responses, leaving it unclear whether the diversity gains reflect more distinct correct reasoning paths.

R2: For Figure 4 (Figure 5 in the revised manuscript), the reason for including incorrect responses is to assess whether DAE broadens the range of reasoning paths explored, regardless of final-answer correctness.

We recompute the diversity metrics using only verified-correct responses, and the results demonstrate that increased diversity derives from more distinct correct reasoning paths. As shown in the table below, C2OPSDC^2OPSD with DAE still shows higher mean scores than SDPO on four reported diversity metrics (including a new metric). We have added the detailed results to Figure 6 of the revised manuscript and updated the accompanying analysis.

Model1 − Self-BLEUMean cosine distanceDCScoreMean pairwise STS dissimilarity
Initial0.5160.1681.2480.301
SDPO0.5350.0351.0270.090
C2OPSDC^2OPSD with DAE only0.6340.3571.3780.342

Additionally, we have already utilized several diversity metrics (1 - self-BELU, Mean cosine distance, DCScore, and stsb-roberta-base) to avoid the impact of different expressions of the same solution.

AuthorsOpenReview ↗

Responses to Reviewer wqf5 (2/2)

Q1: What is the exact definition and parameterization of the Generalized JSD used in Table 3? Under this objective, do PCMI-induced sharpening and the correction effect of PD still follow the theoretical analysis?

R3: In Table 3 (Table 5 in the revised manuscript), both SDPO and C2OPSDC^2OPSD use weighted JSD with teacher weight α=0.2\alpha=0.2. At each sampled response prefix, PP is the student’s next-token distribution and QQ is the demonstration-conditioned EMA teacher’s distribution. We define

=(1-\alpha)\mathrm{KL}(P\|M)+\alpha\mathrm{KL}(Q\|M), \qquad M=(1-\alpha)P+\alpha Q.$$ The student and teacher weights are therefore 0.8 and 0.2 in both the mixture and the KL terms. Under generalized JSD, we theoretically validate that the correction effect of PD still holds and provide this proof in Appendix A.2.3 of the revised manuscript. For any $0<\alpha<1$, demonstration conditioning can still change the probability ratios between correct responses through a nonlinear function of PCMI. In the idealized sequence-level setting with a frozen base policy and a fixed demonstration distribution, the correction constructed for weighted JSD restores the base-policy probability ratios at $\gamma=1$. > Q2: What proportion of training questions produce non-zero DAE advantages? Does this proportion differ substantially across datasets? **R4:** We conducted a non-zero DAE advantage analysis on the 24 $C^2OPSD$ runs in Table 1. Specifically, we estimated the fraction of question groups with at least one positive DAE advantage using the logged proportion of groups with nonzero variance in their leave-one-out marginal contributions. The overall estimate was **64.0%** over steps 1–200 and **67.4%** over steps 11–200, after excluding the first 10 warm-up steps. As shown in the following table, Biology had the lowest rate for both models. Within each model, the gap between the highest and lowest dataset rates was about **8 percentage points** in both windows. | Dataset | Qwen3-1.7B (w/ warm-up) | Qwen3-1.7B (w/o warm-up) | Qwen3-4B (w/ warm-up) | Qwen3-4B (w/o warm-up) | |---|---:|---:|---:|---:| | Biology | 55.92% | 58.86% | 61.94% | 65.20% | | Chemistry | 64.17% | 67.54% | 67.65% | 71.22% | | Materials | 61.77% | 65.02% | 67.31% | 70.86% | | Physics | 63.92% | 67.28% | 69.67% | 73.33% |

Dear Reviewer wqf5,

Thank you again for your valuable comments. As the discussion period is nearing its end, we would like to check whether our responses have addressed your concerns. If you have any further concerns or questions, please let us know.

Thank you for your time.

The Authors

Reviewer_Nk7oOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of correct-path coverage collapse in OPSD It analyzes the preference sharpening induced by demonstration conditioning from the perspective of PCMI and proposes C^2 OPSD, which improves the coverage of correct responses through PD and DAE. The authors evaluate the proposed method on two Qwen3 models across four science question-answering tasks. The results show that on certain tasks, the method can maintain single-sample accuracy while improving multi-sample success rates, with the most pronounced gains observed in the Biology task.

Critical strengths
  1. The objective is practical and meaningful, offering a new explanatory perspective on how to mitigate mode collapse in OPSD-style methods.
  2. The design steps of PD and DAE are well-motivated and reasonable, although DAE itself appears somewhat heuristic.
  3. The overall experiments and analyses are relatively comprehensive.
Critical weaknesses
  1. Equation (6) characterizes probability reweighting that depends on expected PCMI, but interpreting this as favoring common correct paths requires establishing a relationship between base-policy probability (or path frequency) and expected PCMI. The paper has not yet provided sufficient theoretical or empirical evidence for this relationship; it is recommended to tighten the relevant claims in the Introduction and provide supplementary verification.
  2. The evaluation of diversity carries a risk of circularity: both DCScore and mean cosine distance in Figure 4 share the same similarity formulation and semantic encoder as the DAE training signal. On the only independent metric, 1 - Self-BLEU, the performance is nearly identical to that of SDPO.
  3. The gains of the proposed method exhibit strong domain dependency. Biology is the only domain where C^2 OPSD outperforms both baselines across both model scales and both metrics. While there are partial improvements in other domains, the gains across models and metrics are inconsistent, and performance even degrades on Physics with the 4B model. Furthermore, in Table 2, the full method does not outperform the PD-only variant (C^2 OPSD w/o DAE) in Pass@16 on Physics. This limits how well the current evidence supports the generalizability of the method.
  4. The compared baselines are too limited, including only GRPO and SDPO. More recent diversity-promoting methods should be considered for comparison.
  5. The "(JSD)" label in Figure 6 is not explained.
Questions for authors
  1. Beyond scientific QA tasks, is the proposed method effective on general mathematical reasoning tasks?
  2. Under the Generalized JSD setting, can results and conclusions analogous to Equation (6) still be derived?
5 comments
AuthorsOpenReview ↗

Responses to Reviewer Nk7o (1/2)

Thank you for your constructive comments. We added experiments and revised our manuscript, with the revised parts highlighted in Red font. Please refer to the following responses for more details.

W1: The claim that the PCMI-based reweighting in Eq. (6) favors common correct paths lacks sufficient theoretical or empirical evidence linking expected PCMI to base-policy probability or path frequency.

R1: Our interpretation of Eq. (6) follows the empirical and theoretical results of [1] (Sec. 2, the penultimate paragraph, Sec. 3, Remark 1, and Figure 2). The details of these three pieces of evidence are as follows:

  • In Sec. 2, common solution paths are more likely to be represented when demonstrations are drawn from the policy’s own correct rollouts. If the teacher gives a larger relative likelihood boost to rollouts that resemble those demonstrations, more probable correct rollouts can have higher expected PCMI.
  • Remark 1 shows that Eq. (6) of [1], which is equivalent to our Eq. (6), then amplifies their existing probability advantage.
  • Figure 2 illustrates this mechanism in a controlled setting, where self-distillation concentrates on the initially favored mode despite both modes receiving the same reward.

Additionally, we reanalyzed the initial-model scores underlying Fig. 3 (Sec. 4.5) to examine whether demonstrations give a larger boost to responses the base model already prefers. For 441 pairs of similarly long correct responses from 34 Biology questions, we compared how adding the same independently sampled demonstration changed each response’s log probability. On average, the change was 15.2 natural-log units higher for the response that was more likely before the demonstration was added (95% confidence interval [7.4, 23.4], obtained by bootstrapping questions).

Reference:

[1] On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity

W2: The diversity evaluation risks circularity, and the only independent metric (1 − Self-BLEU) shows nearly identical results to SDPO.

R2: To address the concern about circularity, we added mean pairwise STS dissimilarity to the diversity evaluation. This metric uses a different similarity formulation and semantic encoder from those used in our existing embedding-based metrics. It directly scores the semantic similarity of each response pair and aggregates these scores into a dissimilarity measure for each question (Appendix A.5.2). As shown in Figures 5(d) and 6(d), C2OPSDC^2OPSD exceeds SDPO in mean STS dissimilarity by 0.284 across all responses and by 0.2526 among responses with verified correct answers.

We also recompute 1 − Self-BLEU using only verified-correct responses. As shown in Figure 6 of the revised manuscript, the mean 1 − Self-BLEU score of C2OPSDC^2OPSD is 0.0989 higher than that of SDPO.

W3: The method consistently outperforms both baselines only in Biology, limiting the evidence for its generalizability across domains. Furthermore, in Table 2, the full method does not outperform the PD-only variant (C^2 OPSD w/o DAE) in Pass@16 on Physics.

R3: Limited compute prevented us from completing the hyperparameter search before the original submission, leading to inconsistent performance across datasets. We updated the results under the optimal hyperparameters in Table 1 of the revised manuscript.

In the revised Table 1, C2OPSDC^2OPSD achieves the best result in 11 of 16 comparisons and outperforms SDPO in 15 of 16. On Physics with Qwen3-4B, C2OPSDC^2OPSD raised Avg@16 from 63.91% to 68.39% and Pass@16 from 84.17% to 88.75%. With the revised setting, C2OPSDC^2OPSD achieves the highest mean Avg@16 and Pass@16 among the three methods. We also report results on a mathematical reasoning benchmark below, where C2OPSDC^2OPSD improves both Pass@1 and Pass@kk over SDPO. More details are provided in R6. Overall, C2OPSDC^2OPSD presents consistent performance across various tasks and model sizes, supporting its generalization capabilities.

For Table 2, the full method achieves a higher Avg@16 than PD alone (57.21% vs. 55.55%), while its Pass@16 is slightly lower (80.83% vs. 81.88%). DAE can reward a correct rollout for its contribution to diversity only when the training group contains multiple correct rollouts. One possible explanation is that some reasoning patterns rewarded for being underrepresented within individual training groups also recur across Physics questions. Reinforcing these patterns may make the model rely on them more often at test time. On questions that require a different approach and that PD alone solves only occasionally, this shift could reduce the chance of sampling at least one correct solution within 16 attempts. We also added this explanation to the Effect of PD & DAE in Sec. 4.3.

AuthorsOpenReview ↗

Responses to Reviewer Nk7o (2/2)

W4: The compared baselines are too limited.

R4: We added GRPO with Entropy Advantage [1] as a baseline on Qwen3-1.7B. Following [1], we set κ=2\kappa=2 and α=0.4\alpha=0.4 and used the same training settings as standard GRPO for all other hyperparameters. As shown in the following table, C2OPSDC^2OPSD outperforms GRPO with Entropy Advantage at every evaluated kk in Biology, Chemistry, and Material, with Pass@16 gains of 20.7, 25.6, and 3.9 percentage points, respectively. Given the time and compute available during the rebuttal period, we completed this comparison on Qwen3-1.7B. We will report the Qwen3-4B results once the comparison is complete.

DomainMethodPass@1Pass@2Pass@4Pass@8Pass@16
BiologyGRPO w/ Entropy Adv.43.2553.3860.3465.9170.00
BiologyC2OPSDC^2OPSD51.0466.1878.0986.3490.67
ChemistryGRPO w/ Entropy Adv.55.1857.6259.7561.5562.86
ChemistryC2OPSDC^2OPSD73.6078.3982.4385.8888.41
MaterialGRPO w/ Entropy Adv.47.9461.5671.8178.4781.91
MaterialC2OPSDC^2OPSD66.7872.7977.5381.8185.82

The baseline uses token-level entropy to encourage exploration, and its advantage-shaping mechanism is agnostic to the PCMI-induced sharpening that can arise in OPSD with sampled demonstrations. The probability debiasing component of C2OPSDC^2OPSD is designed specifically to counteract this sharpening.

Reference:

[1] Reasoning with Exploration: An Entropy Perspective

W5: The "(JSD)" label in Figure 6 is not explained.

R5: This is a typo, and we have removed “JSD” from Figure 8 of the revised manuscript (Figure 6 in the original version).

Q1: Is the proposed method effective on general mathematical reasoning tasks?

R6: We compared C2OPSDC^2OPSD with SDPO using the same 800-problem subset of the MATH training split and then evaluated the resulting checkpoints on MATH-500 (500 problems), GSM8K (1,319 problems), and AMC 2023 (40 problems). The following table summarizes the results on mathematical reasoning tasks. In the table, C2OPSDC^2OPSD outperforms SDPO in terms of Pass@1 and Pass@16 across several datasets, demonstrating the effectiveness of C2OPSDC^2OPSD on mathematical reasoning tasks. We also added detailed experimental results and their discussion in Appendix A.4.4. Please refer to Appendix A.4.4 for more details.

BenchmarkMethodAvg@16Pass@2Pass@4Pass@8Pass@16
MATH-500SDPO63.8571.7677.8382.3185.40
C2OPSDC^2OPSD63.8471.7578.1983.4387.60
Δ\Delta-0.01-0.01+0.36+1.12+2.20
GSM8KSDPO76.5883.4788.2391.4193.63
C2OPSDC^2OPSD77.9684.5088.7691.7293.71
Δ\Delta+1.38+1.03+0.53+0.31+0.08
AMC23SDPO37.1947.6056.1562.3567.50
C2OPSDC^2OPSD37.8147.9057.2066.0477.50
Δ\Delta+0.62+0.30+1.05+3.69+10.00

Q2: Under the Generalized JSD setting, can results and conclusions analogous to Equation (6) still be derived?

R7: Yes. We extend our theoretical analysis to the Generalized JSD setting in Appendix A.2.3 of the revised manuscript. Specifically, under the frozen sequence-level setting, let α∈(0,1)\alpha\in(0,1) be the teacher mixture weight and uy⋆=πγ⋆(y∣x)/π0(y∣x)u_y^\star=\pi_\gamma^\star(y\mid x)/\pi_0(y\mid x). The score is

\log\\!\left((1-\alpha)u+\alpha\exp\bigl(i(y;y^{\mathrm{corr}}\mid x)\bigr)\right).$$ With $\gamma=0$, subtracting the stationarity conditions for two correct responses gives the Generalized JSD analogue of Eq. (6): $$\mathcal S_{ab}(\pi_\gamma^\star) =\frac{1-\alpha}{\beta+1-\alpha} \bigl[G_{y_a}(u_{y_a}^\star)-G_{y_b}(u_{y_b}^\star)\bigr].$$ Demonstration conditioning can still amplify an existing probability imbalance between equally correct responses. A sufficient condition is that the response favored by the base policy has at least as high a PCMI for almost every demonstration, with strict inequality on a set of positive probability. The relation is implicit because $G_y$ is evaluated at the optimal policy's probability ratio. The nonlinear transformation inside the expectation means that expected PCMI alone cannot determine the direction of the shift. For any fixed $\alpha\in(0,1)$, the JSD-specific correction $h_x^{\mathrm{JS}}$ in Appendix A.2.3 exactly restores the base-policy probability ratios among correct responses at $\gamma=1$.

Dear Reviewer Nk7o,

We hope this message finds you well, and thank you for your review. As the discussion period draws to a close, we wanted to follow up on our response above. If you have any remaining concerns or questions, we would be happy to address them.

Thank you for your time.

The Authors

Reviewer_mrr2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses correct-path coverage collapse in on-policy self-distillation (OPSD). The authors propose C² OPSD, combining PCMI Debiasing (PD) to mitigate demonstration-induced bias and Diversity-Aware Exploration (DAE) to encourage diverse correct reasoning paths. Experiments on Qwen3-1.7B and Qwen3-4B show improvements in Pass@1 and Pass@16.

Critical strengths
  1. The paper identifies a meaningful limitation of OPSD and provides a theoretically motivated solution.
  2. The proposed components are intuitive, and experiments demonstrate promising improvements over GRPO and SDPO.
Critical weaknesses
  1. Theory-practice gap. The theoretical guarantee assumes an idealized reverse-KL objective, while the implementation uses Generalized JSD, EMA teachers, and standardized/clipped advantages. The practical effectiveness of PCMI debiasing is not sufficiently validated.
  2. Insufficient baselines. The experiments only compare against GRPO and SDPO, without representative entropy-preserving or diversity-aware optimization methods.
  3. Limited evidence of correct-path diversity. The diversity evaluation includes incorrect responses, making it difficult to establish whether the method genuinely improves coverage of distinct correct reasoning paths.
Questions for authors
  1. Can the authors directly measure whether PD reduces probability imbalance among correct reasoning paths?
  2. Can the authors evaluate diversity using only verified-correct responses and compare against stronger diversity-preserving baselines?
4 comments
AuthorsOpenReview ↗

Responses to Reviewer mrr2 (1/2)

Thank you for your constructive feedback. We added experiments and revised our manuscript, with the revised parts highlighted in Red font. Please refer to the following responses for more details.

W1: Theory–practice gap and insufficient empirical validation of PCMI debiasing.

R1: We added empirical results and extended our theoretical analysis to address this gap.

Empirical evidence. As shown in Table 3 of the revised manuscript, we compared C2OPSDC^2OPSD with SDPO under reverse KL. We report the results for the Biology and Physics datasets below. Under reverse KL, C2OPSDC^2OPSD performs better than SDPO in most cases. We use JSD in Table 1 because it is the default distillation divergence in SDPO without feedback.

ModelMethodBiology Pass@1 (%)Biology Pass@16 (%)Physics Pass@1 (%)Physics Pass@16 (%)
Qwen3-1.7BSDPO31.33 ± 2.8838.00 ± 3.4646.64 ± 1.8067.50 ± 2.50
Qwen3-1.7BC2OPSDC^2OPSD36.54 ± 2.1244.67 ± 4.1647.14 ± 2.0469.58 ± 3.15
Qwen3-4BSDPO42.96 ± 1.2550.00 ± 4.0053.96 ± 0.5967.92 ± 1.91
Qwen3-4BC2OPSDC^2OPSD40.00 ± 8.3150.00 ± 5.2956.20 ± 3.4373.75 ± 3.31

Theoretical scope. We also provided the proof of exact cancellation of PCMI-induced sharpening under generalized JSD (Theorem 2, Appendix A.2.3 of the revised manuscript). In this theorem, we offer a theoretical guarantee for PCMI debiasing under JSD. Please refer to R7 to Reviewer Nk7o for more details.

W2/Q2: The baseline comparisons are limited to GRPO and SDPO.

R2: We added GRPO with Entropy Advantage [1] as a baseline on Qwen3-1.7B. Following [1], we set κ=2\kappa=2 and α=0.4\alpha=0.4 and used the same training settings as standard GRPO for all other hyperparameters. As shown in the following table, C2OPSDC^2OPSD outperforms GRPO with Entropy Advantage at every evaluated kk in Biology, Chemistry, and Material, with Pass@16 gains of 20.7, 25.6, and 3.9 percentage points, respectively. Given the time and compute available during the rebuttal period, we completed this comparison on Qwen3-1.7B. We will report the Qwen3-4B results once the comparison is complete.

DatasetMethodPass@1Pass@2Pass@4Pass@8Pass@16
BiologyGRPO w/ Entropy Adv.43.2553.3860.3465.9170.00
BiologyC2OPSDC^2OPSD51.0466.1878.0986.3490.67
ChemistryGRPO w/ Entropy Adv.55.1857.6259.7561.5562.86
ChemistryC2OPSDC^2OPSD73.6078.3982.4385.8888.41
MaterialGRPO w/ Entropy Adv.47.9461.5671.8178.4781.91
MaterialC2OPSDC^2OPSD66.7872.7977.5381.8185.82

The baseline uses token-level entropy to encourage exploration, and its advantage-shaping mechanism is agnostic to the PCMI-induced sharpening that can arise in OPSD with sampled demonstrations. The probability debiasing component of C2OPSDC^2OPSD is designed specifically to counteract this sharpening.

Reference:

[1] Reasoning with Exploration: An Entropy Perspective

AuthorsOpenReview ↗

Responses to Reviewer mrr2 (2/2)

W3/Q2: Limited evidence of correct-path diversity.

R3: We are unsure whether “diversity evaluation” refers to DAE during training or to Figure 4 (Figure 5 in the revised manuscript). During training, DAE computes diversity only over responses with verifier-correct final answers. Please refer to Lines 245 and 253 for more details.

For Figure 4 (Figure 5 in the revised manuscript), the reason for including incorrect responses is to assess whether DAE broadens the range of reasoning paths explored, regardless of final-answer correctness. We recompute the diversity metrics using only verified-correct responses. As shown in the table below, C2OPSDC^2OPSD with DAE still shows higher mean scores than SDPO on four reported diversity metrics (including a new metric). We have added the detailed results to Figure 6 of the revised manuscript and updated the accompanying analysis.

Model1 − Self-BLEUMean cosine distanceDCScoreMean pairwise STS dissimilarity
Initial0.5160.1681.2480.301
SDPO0.5350.0351.0270.090
C2OPSDC^2OPSD with DAE only0.6340.3571.3780.342

Q1: Can the authors directly measure whether PD reduces probability imbalance among correct reasoning paths?

R4: We added an experiment to verify that PD reduces probability imbalance among correct reasoning paths, as shown in Figure 4 of the revised manuscript. We directly measured this on 15 questions from the Physics test set. For each question, we used an LLM to construct two correct reference derivations that differed in a key reasoning step. We used teacher forcing to score the same two derivations under the original prompt with the SDPO and C2OPSDC^2OPSD w/ PD. In Figure 4, we observe that C2OPSDC^2OPSD w/ PD decreases the probability imbalance between the two answers on 11 of 15 questions.

Dear Reviewer mrr2,

Thank you again for your review. As the discussion period is nearing its end, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know.

Thank you for your time.

The Authors

Dear Reviewer mrr2,

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to follow up briefly on our rebuttal. If you have a chance to look over the additional experiments and theoretical analysis, we would appreciate hearing whether they address your concerns.

For quick reference, we have summarized the main additions and where to find them.

  • W1 — PCMI debiasing. We added a reverse-KL comparison with SDPO (Table 3) and extended our theoretical analysis to generalized JSD (Theorem 2, Appendix A.2.3).
  • W2/Q2 — Baselines. We added GRPO with Entropy Advantage as a baseline on Biology, Chemistry, and Materials (comparison table in R2 of our rebuttal).
  • W3/Q2 — Diversity. We evaluated diversity among responses with verified correct final answers and added STS dissimilarity as a metric (Figure 6).
  • Q1 — Probability imbalance. We compared the log-probability gaps between fixed correct derivations on the Physics test set under SDPO and the PD-only variant (Figure 4).

Thank you for your time. We would be happy to clarify any remaining points.

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Evidence

Dear PCs, SACs, ACs, and Reviewers,

We thank the Area Chair for their time and all reviewers for their constructive feedback. The discussion has helped us improve the manuscript. As the rebuttal period draws to a close, we summarize three main additions below.

  1. Diversity among correct responses. In Figure 6, we repeat the Biology analysis using only responses with verified correct final answers. The DAE-only variant achieves higher mean scores than SDPO on all four diversity metrics. These include a new STS measure that uses a different encoder and similarity formulation from those used by DAE.

  2. Generalized JSD analysis and reverse-KL comparison. Theorem 2 (Appendix A.2.3) shows that a JSD-specific correction restores the base-policy probability ratios among correct responses under idealized sequence-level assumptions. Table 3 adds an empirical comparison with SDPO under reverse KL.

  3. Additional baseline and math benchmarks. We added GRPO with Entropy Advantage as a baseline on three science datasets. C2OPSDC^2OPSD achieves higher Pass@16 on all three. We also compared C2OPSDC^2OPSD with SDPO on three math benchmarks. C2OPSDC^2OPSD improves Pass@16 on all three, with the largest gain of 10 percentage points on AMC23 (Table 4).

Reviewer mrr2 has not yet posted a follow-up to our rebuttal. Reviewers wqf5 and Nk7o indicated that the additional analyses and experiments addressed most of their concerns. The same evidence also addresses the concerns that Reviewer mrr2 raised in common with the other reviewers about the scope of the theory under generalized JSD, diversity among responses with verified correct final answers, and the breadth of baseline comparisons.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.