- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies on-policy distillation for reasoning, where a locally valid teacher step may still harm the outcome attainable by the student. The proposed method pairs the student step with a one-step teacher proposal at selected states, estimates both branches under the same frozen student via Monte Carlo continuations, and amortizes the comparison with a counterfactual value model that drives process feedback, preference learning, and gated distillation. Experiments on mathematical reasoning show consistent gains over standard on-policy distillation, with analyses of signal quality, refresh necessity, and computational cost.
Critical strengths
1.The paper introduces a counterfactual perspective for selective on-policy distillation, evaluating whether replacing the student step with a teacher proposal benefits the same student, rather than relying on observed-path proxies. 2.The central premise is directly validated, as evaluation shows that a substantial fraction of student-visited states favor the student action.
Critical weaknesses
- The criterion for selecting reasoning steps for counterfactual pairing, and the definition of a step, are not specified.
- The experimental evaluation is confined to mathematical reasoning benchmarks and a limited set of baselines.
Questions for authors
- What is the criterion for selecting the reasoning steps used for counterfactual pairing, and how sensitive are the results to this choice?
- How does the framework perform on domains beyond mathematical reasoning, particularly where automated verifiers are unavailable?
- How does the method compare with other selective on-policy distillation methods beyond the adaptive baseline included in the experiments?
- Could the value model, which shares its backbone with the student and is warm-started across rounds, develop a systematic preference toward the student's own actions?
- How are tied pairs, where both branches succeed or fail equally under Monte Carlo estimation, handled in the preference and gating objectives?
2 comments
Thank you for the detailed responses. The step-selection ablation and the comparisons with TIP and TA-OPD substantially address my concerns regarding sensitivity and baseline coverage. The HH-RLHF experiment also provides preliminary evidence of applicability beyond mathematical reasoning. I will raise my score.
We thank the reviewer for the thorough review, insightful suggestions, and time devoted to our work.
Response to Q1
As defined in Appendix B, we segment reasoning steps at newlines and sentence boundaries, merging segments shorter than 8 characters with adjacent ones except at trajectory ends. We uniformly select at most 4 steps from each trajectory to cover early, middle, and late stages of reasoning while controlling the cost of counterfactual continuation sampling. We will add this definition to the main methodology section.
We further compare uniform selection, random selection, and different maximum numbers of selected steps. When selecting at most 4 steps, uniform selection achieves an Avg@8 of 44.28% and a Pass@8 of 71.20%, while random selection achieves 43.65% and 70.40%, respectively. Increasing the maximum number of selected steps to 6 further improves Avg@8 to 45.08% and Pass@8 to 72.80%, but also incurs higher continuation sampling costs. Therefore, we use 4 uniformly selected steps by default to balance sampling cost and performance.
Response to Q2
CFOPD requires an evaluation signal but does not rely on deterministic rule-based verifiers. In our additional experiments on HH-RLHF Helpfulness, we use a frozen Llama-3.1-8B-Instruct evaluator to estimate counterfactual action values and an independent Prometheus 2 7B model for final evaluation, while keeping the remaining training procedure unchanged.
Under identical training settings, CFOPD achieves a score of 0.78, compared to 0.71 for GKD-OPD, providing preliminary evidence of CFOPD's applicability to open-ended tasks without deterministic verifiers.
Response to Q3
We introduce two selective on-policy distillation baselines, TIP and TA-OPD. TIP selects supervision based on uncertainty and teacher–student disagreement, while TA-OPD selects supervision based on teachability. All methods use the same teacher and student models, training data, and student optimization budget. On MATH-500, CFOPD achieves an Avg@8 of 44.28% and a Pass@8 of 71.20%, while TIP achieves 42.15% and 68.80%, and TA-OPD achieves 42.95% and 69.40%, respectively. Under the current setting, supervision based on counterfactual values defined relative to the student policy achieves higher Avg@8 and Pass@8.
Response to Q4
CF-PRM uses the student model backbone only at initialization and does not share parameters with the student model during subsequent training. Across training rounds, it is warm-started from its own checkpoint from the previous round and then trained independently. Equation (7) applies symmetric Monte Carlo supervision to teacher and student actions, without defining different objectives based on action identity.
Figure 4 reports an overall sign accuracy of 72.3% against high-budget reference estimates. The empirical analysis in Section 4.4 shows that CFOPD retains over 75% of teacher-preferred interventions while suppressing approximately 79% of student-preferred interventions. These results show that CF-PRM identifies counterfactual value differences for both teacher and student actions, rather than consistently favoring student actions, providing evidence of bidirectional action discrimination.
Response to Q5
Monte Carlo tied pairs are still used to train CF-PRM; their role in student optimization depends on the value difference predicted by CF-PRM.
When the Monte Carlo estimates for the teacher and student branches are equal ((\Delta_{\mathrm{MC}} = 0)), the action pair still contributes to the Value Loss and Effect Loss in Eq. (7). The Effect Loss encourages the predicted value difference to approach zero. However, since the ranking indicator is inactive for tied pairs, they do not impose a directional constraint in the Ranking Loss.
During student optimization, Eqs. (11) and (12) use the value difference predicted by CF-PRM rather than the original Monte Carlo difference. Therefore, a Monte Carlo tie does not necessarily force the subsequent training signal to be zero. If the predicted value difference is also exactly zero, the Preference Loss produces no directional preference, while the distillation weight is still determined by the soft gating rule and is not automatically set to zero. This avoids treating empirical ties arising from finite Monte Carlo continuations as evidence that the two actions have exactly equal true values. We will explicitly clarify this case in the methodology section of the revised manuscript.