acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Imitate: Counterfactual Value Modeling for On-Policy Distillation

12 views

Abstract

On-policy distillation (OPD) addresses a central mismatch in reasoning distillation by querying the teacher on states actually visited by the student. Yet state alignment does not guarantee useful guidance: in multi-step reasoning, a locally valid teacher step may still lead to a worse final outcome under the student's continuation. Selective variants modulate imitation using uncertainty, disagreement, or teachability, but these proxies do not reveal whether substituting the teacher action improves the current student's eventual outcome. We introduce Counterfactual On-Policy Distillation (CFOPD) to estimate this policy-relative intervention value. CFOPD compares teacher and student actions through matched rollouts under the same frozen student policy and amortizes these comparisons with a shared counterfactual value model. The learned values induce two complementary contrasts: action-level contrasts guide supervision between teacher and student alternatives and modulate teacher distillation, while temporal contrasts track progress along the realized student trajectory. Because both depend on the student's continuation ability, the counterfactual supervision is refreshed as the student evolves. Across two Qwen3 teacher–student settings, CFOPD improves Avg@8 over GKD-OPD on all six reasoning benchmarks, with MATH-500 gains from 50.65 to 55.97 and 32.18 to 44.28, respectively. High-budget evaluation further finds the student action preferable at 30–50% of evaluated states despite the teacher's higher overall accuracy, supporting the need for policy-relative action evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Counterfactual Learning, Process Reward Modeling, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_QptuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation for reasoning, where a locally valid teacher step may still harm the outcome attainable by the student. The proposed method pairs the student step with a one-step teacher proposal at selected states, estimates both branches under the same frozen student via Monte Carlo continuations, and amortizes the comparison with a counterfactual value model that drives process feedback, preference learning, and gated distillation. Experiments on mathematical reasoning show consistent gains over standard on-policy distillation, with analyses of signal quality, refresh necessity, and computational cost.

Critical strengths

1.The paper introduces a counterfactual perspective for selective on-policy distillation, evaluating whether replacing the student step with a teacher proposal benefits the same student, rather than relying on observed-path proxies. 2.The central premise is directly validated, as evaluation shows that a substantial fraction of student-visited states favor the student action.

Critical weaknesses
  1. The criterion for selecting reasoning steps for counterfactual pairing, and the definition of a step, are not specified.
  2. The experimental evaluation is confined to mathematical reasoning benchmarks and a limited set of baselines.
Questions for authors
  1. What is the criterion for selecting the reasoning steps used for counterfactual pairing, and how sensitive are the results to this choice?
  2. How does the framework perform on domains beyond mathematical reasoning, particularly where automated verifiers are unavailable?
  3. How does the method compare with other selective on-policy distillation methods beyond the adaptive baseline included in the experiments?
  4. Could the value model, which shares its backbone with the student and is warm-started across rounds, develop a systematic preference toward the student's own actions?
  5. How are tied pairs, where both branches succeed or fail equally under Monte Carlo estimation, handled in the preference and gating objectives?
2 comments

We thank the reviewer for the thorough review, insightful suggestions, and time devoted to our work.

Response to Q1

As defined in Appendix B, we segment reasoning steps at newlines and sentence boundaries, merging segments shorter than 8 characters with adjacent ones except at trajectory ends. We uniformly select at most 4 steps from each trajectory to cover early, middle, and late stages of reasoning while controlling the cost of counterfactual continuation sampling. We will add this definition to the main methodology section.

We further compare uniform selection, random selection, and different maximum numbers of selected steps. When selecting at most 4 steps, uniform selection achieves an Avg@8 of 44.28% and a Pass@8 of 71.20%, while random selection achieves 43.65% and 70.40%, respectively. Increasing the maximum number of selected steps to 6 further improves Avg@8 to 45.08% and Pass@8 to 72.80%, but also incurs higher continuation sampling costs. Therefore, we use 4 uniformly selected steps by default to balance sampling cost and performance.

Selection StrategyMaximum StepsAvg@8 (%)Pass@8 (%)
Uniform selection444.2871.20
Random selection443.6570.40
Uniform selection241.5568.20
Uniform selection645.0872.80

Response to Q2

CFOPD requires an evaluation signal but does not rely on deterministic rule-based verifiers. In our additional experiments on HH-RLHF Helpfulness, we use a frozen Llama-3.1-8B-Instruct evaluator to estimate counterfactual action values and an independent Prometheus 2 7B model for final evaluation, while keeping the remaining training procedure unchanged.

Under identical training settings, CFOPD achieves a score of 0.78, compared to 0.71 for GKD-OPD, providing preliminary evidence of CFOPD's applicability to open-ended tasks without deterministic verifiers.

MethodHH-RLHF Helpfulness Score
GKD-OPD0.71
CFOPD0.78

Response to Q3

We introduce two selective on-policy distillation baselines, TIP and TA-OPD. TIP selects supervision based on uncertainty and teacher–student disagreement, while TA-OPD selects supervision based on teachability. All methods use the same teacher and student models, training data, and student optimization budget. On MATH-500, CFOPD achieves an Avg@8 of 44.28% and a Pass@8 of 71.20%, while TIP achieves 42.15% and 68.80%, and TA-OPD achieves 42.95% and 69.40%, respectively. Under the current setting, supervision based on counterfactual values defined relative to the student policy achieves higher Avg@8 and Pass@8.

MethodMATH-500 Avg@8 (%)Pass@8 (%)
TIP42.1568.80
TA-OPD42.9569.40
CFOPD44.2871.20

Response to Q4

CF-PRM uses the student model backbone only at initialization and does not share parameters with the student model during subsequent training. Across training rounds, it is warm-started from its own checkpoint from the previous round and then trained independently. Equation (7) applies symmetric Monte Carlo supervision to teacher and student actions, without defining different objectives based on action identity.

Figure 4 reports an overall sign accuracy of 72.3% against high-budget reference estimates. The empirical analysis in Section 4.4 shows that CFOPD retains over 75% of teacher-preferred interventions while suppressing approximately 79% of student-preferred interventions. These results show that CF-PRM identifies counterfactual value differences for both teacher and student actions, rather than consistently favoring student actions, providing evidence of bidirectional action discrimination.

Response to Q5

Monte Carlo tied pairs are still used to train CF-PRM; their role in student optimization depends on the value difference predicted by CF-PRM.

When the Monte Carlo estimates for the teacher and student branches are equal ((\Delta_{\mathrm{MC}} = 0)), the action pair still contributes to the Value Loss and Effect Loss in Eq. (7). The Effect Loss encourages the predicted value difference to approach zero. However, since the ranking indicator is inactive for tied pairs, they do not impose a directional constraint in the Ranking Loss.

During student optimization, Eqs. (11) and (12) use the value difference predicted by CF-PRM rather than the original Monte Carlo difference. Therefore, a Monte Carlo tie does not necessarily force the subsequent training signal to be zero. If the predicted value difference is also exactly zero, the Preference Loss produces no directional preference, while the distillation weight is still determined by the soft gating rule and is not automatically set to zero. This avoids treating empirical ties arising from finite Monte Carlo continuations as evidence that the two actions have exactly equal true values. We will explicitly clarify this case in the methodology section of the revised manuscript.

Thank you for the detailed responses. The step-selection ablation and the comparisons with TIP and TA-OPD substantially address my concerns regarding sensitivity and baseline coverage. The HH-RLHF experiment also provides preliminary evidence of applicability beyond mathematical reasoning. I will raise my score.

Reviewer_1NMVOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Overall, my understanding is that the authors of the paper created a method to address the limitation of on-policy distillation. The authors find that a stronger teacher's reasoning steps may not always improve the student's final performance. To resolve this issue, the authors propose CFOPD (Counterfactual On-Policy Distillation), it basically first evaluate whether the teacher's step can improve the student's chance of solving a problem. This is done using a counterfactual value model to compare teacher and student actions and adaptively guide the imitation. Experiments from the paper shows on the mathematical reasoning benchmarks, CFOPD consistently better than standard on-policy distillation, which shows selectively learning from teacher using the proposed method is better.

Critical strengths

Overall I think the author finds a good motivation and a smart solution to address the issue.

  1. The author finds that a stronger teacher's reasoning steps may not always improve the student's performance. This is a surprising finding since it questions the assumption that the teacher is always right for guiding the learning of student model.

  2. CFOPD results from the paper show consistent improvement over standard OPD across multiple reasoning benchmarks. The ablation study and counterfactual analyses also show that selectively guiding the student's imitation of the teacher can better improve reasoning performance.

Critical weaknesses

As a reviewer who works on training reasoning model, I think this is a good paper but I still have some concerns on the results and scalability of the method.

First, the experiment results from the paper mainly focus on the mathematical reasoning benchmarks, so I wonder if CFOPD can perform similarly on other reasoning tasks such as coding and long horizon tool use.

Second, for the CFOPD method to work, it requires a counterfactual value model to compare the teacher and student model actions based on the downstream value. This extra evaluation need addtional computation during the training time. Meanwhile, the value function requires training too. This create extra need when compute resource is scarce.

Questions for authors

Major Question: (I understand the rebuttal time period is short. The answer here need not to be perfect)

  1. How does the CFOPD perform on non-math reasoning benchmarks? Can the author test and compare CFOPD on coding or tool use benchmarks? I would like to see if the benefit goes beyond math benchmarks.

Minor Question:

  1. I understand that some teacher steps maybe hard for the student to follow now but maybe useful after more training steps. So, will this selective imitation prevent the student from learning these more advanced reasoning steps? Is this a question of trade off?
2 comments

We would like to express our sincere gratitude to the reviewer for the thoughtful evaluation and valuable comments. We appreciate the opportunity to clarify and improve our work.

Response to W1 Q1

To further evaluate the generalization ability of CFOPD beyond mathematical reasoning tasks, we conducted additional experiments on the code reasoning benchmark CRUXEval-O, using the same Avg@8 and Pass@8 evaluation protocol as in the mathematical reasoning benchmarks to compare CFOPD with GKD-OPD.

The results show that CFOPD achieves 40.06% Avg@8 and 66.00% Pass@8 on CRUXEval-O, outperforming GKD-OPD's 31.42% and 59.63%, with improvements of 8.64 and 6.37 percentage points, respectively.

MethodCRUXEval-O Avg@8Pass@8
GKD-OPD31.42%59.63%
CFOPD40.06%66.00%

These results further demonstrate that the performance advantages of CFOPD are not limited to mathematical reasoning tasks. Its adaptive supervision mechanism based on counterfactual intervention advantages can also effectively improve the model's code reasoning ability, providing additional empirical evidence for the generalization potential of CFOPD across different reasoning tasks.

Response to W2

Table 7 reports that full CFOPD requires 6.5 hours of training, compared with 3.3 hours for GKD-OPD. The additional computational cost comes from counterfactual continuations and value-model training. Figure 6 further shows that CFOPD still achieves better reasoning performance under comparable total training budgets.

For resource-constrained settings, CFOPD-Implicit in Table 8 provides a lightweight alternative. This variant requires neither additional Monte Carlo continuations nor explicit value-model training, taking only 3.4 hours, which is comparable to GKD-OPD, while improving MATH-500 Avg@8 from 50.65% to 53.95%. When additional training compute is available, full CFOPD further improves Avg@8 to 55.97%. All additional computation is incurred only during training; at deployment, only the student model is retained, so there is no additional inference overhead.

Response to Q2

CFOPD's selective imitation does not mean permanently excluding advanced reasoning steps that the current student finds difficult to utilize.

Counterfactual values are defined relative to the current student policy, and the value model is periodically updated as the student's capabilities improve. Therefore, a teacher step that currently appears unhelpful may receive a higher supervision weight in subsequent training rounds. Figure 5 shows that periodically updating the value model outperforms keeping it frozen. Furthermore, CFOPD adjusts distillation weights only at selected reasoning steps, while standard teacher supervision is retained at other positions.

Periodic updates can mitigate this risk, but the current objective does not explicitly model the long-term learning value of teacher steps. We will clarify this limitation in the revised manuscript.

Hi authors, thank you for addressing my concerns. I really like your paper!! Please consider open source the code so the community can benefit from this!!

Reviewer_rhhPOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CFOPD, a counterfactual on-policy distillation framework that aims to determine when teacher supervision is beneficial to a student model. The authors argue that although existing OPD methods address the distribution mismatch between teacher and student trajectories, a stronger teacher's reasoning step may not always be beneficial to the current student, especially when the student cannot successfully continue from that step. To address this issue, CFOPD evaluates teacher and student actions at the same reasoning state by sampling multiple continuations from a frozen student policy. These counterfactual outcomes are used to train a value model, which subsequently provides process-level feedback, preference supervision, and adaptive teacher distillation weights. Experiments on mathematical reasoning benchmarks show consistent improvements over standard OPD methods.

Critical strengths

The authors identify a meaningful limitation of existing OPD methods: even when the teacher and student are evaluated under the same context, the teacher's action is not necessarily the most beneficial for the student to imitate. I find this motivation reasonable, particularly for multi-step reasoning tasks where the student's ability to continue from a teacher-provided step can be limited.

Rather than relying solely on heuristic signals such as entropy or teacher–student disagreement, the authors directly estimate the downstream utility of teacher interventions through counterfactual rollouts. The use of a shared value model to guide different supervision objectives is also interesting.

The authors evaluate their method across six mathematical reasoning benchmarks under two teacher–student configurations. Beyond the main results, they investigate the contributions of different training objectives, compare counterfactual guidance with heuristic supervision signals, analyze the reliability of their value model, and study the importance of refreshing the model during training. I particularly appreciate the experiment showing that the student's own actions can outperform the teacher's actions at a considerable fraction of evaluated states, which directly supports the paper's motivation.

Critical weaknesses

The central assumption is that if replacing a student action with a teacher action improves the probability of reaching the correct answer, then the student should receive stronger supervision from the teacher at that position. However, these are not necessarily equivalent. The former evaluates the immediate intervention under the current student policy, while the latter concerns how useful the supervision will be for improving the student's future behavior. I think the paper could better explain this connection, ideally with experiments examining whether the estimated intervention advantages actually correlate with the learning benefits of applying teacher supervision.

The method relies on Monte Carlo continuations to estimate the success probabilities of teacher and student actions. However, with only eight continuations per branch by default, the estimates can be noisy, particularly when the two actions have similar success probabilities. The learned value model achieves 72.3% sign accuracy against higher-budget estimates, which is reasonable but still leaves a considerable fraction of intervention comparisons incorrectly predicted. Moreover, the authors report that student actions are preferable at approximately 30–50% of evaluated states, but it is unclear how many of these differences are statistically meaningful rather than near-ties caused by sampling variance. I would like to see further analysis of the distribution and uncertainty of the estimated intervention advantages.

CFOPD requires multiple additional student continuations for each selected reasoning step, along with repeated training of an auxiliary value model. According to Table 7, its training takes 6.5 hours compared with 3.3 hours for GKD-OPD. Although the authors provide a compute–performance analysis, I think a more explicit comparison with a simpler OPD baseline trained using the same total computational resources and additional student-generated data would be useful. Interestingly, the PRM-free variant achieves 53.95% on MATH-500 in 3.4 hours, compared with 55.97% in 6.5 hours for the full CFOPD. This raises the question of whether the additional Monte Carlo supervision and value-model training are worth their substantial computational cost.

Questions for authors

see the weakness part above

2 comments

We sincerely thank the reviewer for the careful review and constructive feedback. We greatly appreciate the time and effort devoted to evaluating our work.

Response to W1

CFOPD uses the intervention value under the current student policy as a proxy for allocating teacher supervision. Intervention value measures the effect of replacing an action on the current task success probability, whereas learning benefit measures the improvement in the future student policy resulting from such supervision. These two quantities are not equivalent. We conducted an additional experiment to examine their correlation.

In this experiment, we grouped student-visited states according to their estimated intervention advantages and applied teacher supervision with the same fixed weight within each group. Under identical student initializations and training budgets, we compared the training outcomes of each group against a baseline without teacher supervision and measured performance improvements on a held-out test set. The estimated intervention advantage shows a positive correlation with the actual learning benefit, with a Spearman correlation coefficient of 0.63. When higher-budget Monte Carlo estimates are used as the reference intervention advantages, the correlation increases to 0.74.

Intervention SignalSpearman Correlation with Learning Benefit
CF-PRM Estimated Advantage0.63
High-Budget Monte Carlo Advantage0.74

This positive correlation supports using intervention advantage as an empirical proxy for allocating teacher supervision, but does not imply equivalence or a causal relationship between the two quantities. We will include this correlation analysis in the revised manuscript.

Response to W2

The 72.3% sign accuracy measures the agreement between CF-PRM predictions and high-budget Monte Carlo reference estimates, rather than the statistical significance of the difference for an individual action pair. When the intervention advantage is close to zero, finite-sample estimates become less stable.

Table 4 shows that increasing the number of continuations per branch from 8 to 16 improves sign accuracy from 72.34% to 75.20%, while downstream performance gains exhibit diminishing returns. In addition, Eq. (11) weights each action pair by the absolute value of its predicted intervention advantage. Consequently, pairs with smaller predicted differences receive lower weights and have less influence on student optimization.

We further analyzed the absolute values of the high-budget estimated intervention advantages for student-preferred intervention pairs. The results show that 36.57% of the pairs have an absolute estimated advantage of no more than 0.2, 43.34% fall between 0.2 and 0.4, and 20.09% exceed 0.4. Furthermore, a two-sided Fisher's exact test shows that 65.36% of student-preferred intervention pairs exhibit statistically significant differences (p < 0.05). These results indicate that the observed student advantage is not solely attributable to near-zero estimated differences or sampling noise.

Advantage RangeProportion (%)
0 to 0.236.57%
0.2 to 0.443.34%
> 0.420.09%

Response to W3

Regarding the reviewer's concern about fair comparisons under comparable computational budgets, Figure 6 in the appendix plots performance against total compute. As the training compute budget increases, the performance gains of standard OPD gradually diminish, whereas CFOPD achieves better reasoning performance at comparable total budgets. We gave GKD-OPD the same amount of additional student-generated data. On MATH-500, GKD-OPD achieves 52.23% Avg@8 and 76.80% Pass@8, compared with CFOPD's 55.97% and 81.40%. Thus, CFOPD's gains cannot be attributed solely to additional compute; counterfactual supervision helps use compute more effectively, yielding a more favorable cost–performance trade-off.

PRM-free variant uses Eq. (15)'s implicit value function, estimating values from the ground-truth answer's likelihood under a frozen student model, without Monte Carlo continuations or value-model training. Full CFOPD estimates downstream action success probabilities under the current student policy via Monte Carlo continuations, providing more direct counterfactual supervision. In Table 8, PRM-free variant takes 3.4 hours to reach 53.95% Avg@8 and 79.20% Pass@8 on MATH-500. Full CFOPD reaches 55.97% and 81.40% in 6.5 hours.

These results demonstrate that implicit value estimation retains most gains from counterfactual guidance at lower compute cost, while explicit Monte Carlo supervision and value-model training further improve reasoning performance. Thus, the methods suit different budgets: PRM-free variant is more suitable for resource-constrained settings, while full CFOPD suits those prioritizing higher reasoning performance and accommodating additional training costs.

Thx for the response!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.