acceptodds
Under review as a conference paper at ICLR 2027

CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation

Abstract

On-policy distillation (OPD) supervises a student language model on trajectories sampled from its current policy, but typically treats response positions uniformly. Selective OPD addresses this limitation by allocating supervision non-uniformly across response tokens according to their estimated training value. Existing criteria, however, focus primarily on optimization need, such as uncertainty or teacher–student disagreement, while task relevance—whether supervision is tied to the semantic content of the current input—remains less directly characterized. We introduce Counterfactual Relevance for On-Policy Distillation (CROP), which operationalizes task relevance through a paraphrase-calibrated counterfactual sensitivity margin. For each source prompt, CROP constructs a validated original–paraphrase–counterfactual triplet, holds the student rollout fixed, and ranks response positions by how strongly the teacher reacts to a task-relevant condition change relative to a meaning-preserving paraphrase. At a 20% per-response token budget, CROP achieves the best aggregate performance in two teacher–student settings, outperforming the strongest non-CROP baseline by 1.43 and 0.96 points, respectively. Across 5%–20% supervision budgets, CROP consistently outperforms the strongest matched hard selector, with the largest observed margin at the 5% budget. These results suggest that task relevance provides a complementary signal for selective on-policy distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, selective distillation, token selection, task relevance, large language model post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bUuyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CROP (Counterfactual Relevance for On-Policy Distillation), a token-selection method designed to improve the efficiency of on-policy distillation (OPD) by prioritizing tokens that are relevant to the semantics of the current task. CROP estimates task relevance through controlled prompt interventions. Specifically, CROP constructs an original prompt, a meaning-preserving paraphrase, and a counterfactual prompt that modifies one task-relevant condition. Given a fixed student-generated response, the method measures changes in the teacher's token distributions under these three prompts using Jensen–Shannon divergence. The difference between counterfactual sensitivity and paraphrase sensitivity serves as a token-level relevance score, which is used to select tokens for distillation under a fixed supervision budget. Experiments on mathematical reasoning benchmarks across two Qwen teacher–student configurations show that CROP achieves the highest aggregate performance among the evaluated methods at a 20% token supervision budget.

Critical strengths
  1. The paper identifies a meaningful distinction between optimization need and task relevance in selective OPD, offering an interesting perspective on token selection
  2. CROP outperforms several selective and dense OPD baselines across two teacher–student settings
  3. The paper includes matched token-budget controls, component ablations, and token-selection analyses that help characterize the proposed scoring mechanism
Critical weaknesses
  1. The empirical improvements are not significant enough. Although CROP achieves the highest aggregate performance in both settings, its improvements over the strongest baselines are relatively small (1.43 points in Setting A and 0.96 points in Setting B). CROP does not consistently outperform competing methods on every benchmark. For example, in Setting A, its AIME25 performance is below several baselines, while in Setting B, TIP and Dense CREDIT obtain stronger results on certain MATH-500 metrics

  2. The proposed relevance score may not necessarily reflect the actual training value of a token. The main assumption is that tokens with larger teacher-distribution shifts under counterfactual prompts are more useful for distillation. For example, in Figure 6, the counterfactual changes the coefficient from 3 to 5 in a mathematical expression. A token that simply repeats the original coefficient in the generated response could receive a high relevance score due to this change. But supervising such a token may only reinforce copying information from the input? Not necessarily for solving the math task? I think an analysis of such reward hacking problem should be discussed and evaluated.

  3. CROP requires teacher rescoring under the original, paraphrased, and counterfactual prompts for every student rollout. This introduces considerable overhead compared with simpler selection methods such as entropy-based selection. The paper does not report training time, teacher inference cost, or performance under matched computational budgets. This is important because CROP-student achieves 26.37 compared with CROP's 26.98 at the 20% budget (Table 2), so that the benefit of additional teacher-based scoring may be relatively small.

I think my main concern is that given the complexity and additional teacher inference required by CROP, the relatively small improvements over existing baselines seem insufficient to justify the additional computational cost.

Questions for authors

See the weaknesses.

1 comment

Thanks for the rebuttal. The multi seed results make the small gain over TA-OPD more convincing, and I appreciate the timing breakdown. I had expected teacher-based scoring to be noticeably slower than CROP-student, but the measured step times are nearly identical in your implementation.

My main concern is still whether CROP is worth the added cost over baselines (e.g. TA-OPD). In Setting A, CROP gains just under 1% on All-8 while taking 8.3% longer per step at the same 20% supervision fraction. CROP also requires generating and validating paraphrase and counterfactual prompts, then rescoring them during training. I appreciate that the prompt construction cost can be amortized across runs. Even so, I find the accuracy gain modest relative to the extra training time and complexity.

Reviewer_268sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CROP selects token positions for on-policy distillation using counterfactual sensitivity calibrated by paraphrase sensitivity. Experiments with two Qwen teacher–student configurations across four mathematical reasoning benchmarks show gains of 1.43 and 0.96 percentage points over the strongest compared baseline at a 20% supervision budget.

Critical strengths
  • The selection criterion is clearly defined and evaluated against informative controls, including reversed rankings, student-based scoring, and budgeted CREDIT.

  • Improvements across several supervision budgets support the usefulness of the selection rule beyond a single configuration.

  • The authors appropriately distinguish the heuristic score from causal importance and token supervision budgets from total computation.

Critical weaknesses
  1. The robustness of the performance gains across independent training runs remains unquantified.

Table 1 reports improvements of 1.43 and 0.96 percentage points over the strongest non-CROP baselines in Settings A and B, respectively. The consistent gains across two teacher–student configurations and multiple token budgets provide useful evidence that the benefit is not confined to a single experimental setting. However, Appendices C and E explicitly acknowledge that the evaluation does not estimate variability across independent training runs. Multiple decoding samples from a fixed checkpoint characterize evaluation-time sampling variability, but do not establish whether the advantage persists when training is repeated.

This distinction is particularly relevant to on-policy distillation, where stochastic student rollouts affect token selection, parameter updates, and subsequent training trajectories. Given the relatively modest margins over strong baselines, the reported results leave uncertainty about the reproducibility of the improvement. Repeating CROP and the principal competing baselines across several training seeds, with means and variability estimates, would substantially strengthen the empirical conclusion. The existing results are promising; the remaining concern is the stability of the comparative advantage.

  1. The selection score’s robustness to intervention construction and prefix compatibility is insufficiently characterized.

The paper provides meaningful validation of its scoring mechanism: Appendix A describes triplet generation, validation, and human inspection; the component ablations support combining counterfactual and paraphrase signals; Appendix B.6 examines selected-token categories; and Appendix D presents qualitative diagnostics. Section 3.3 also appropriately describes the score as a contrast-specific heuristic rather than ground-truth relevance or an unrestricted causal effect.

Nevertheless, these analyses do not establish how sensitive token selection and downstream gains are to the particular valid rewrites used. Different meaning-preserving paraphrases may induce different teacher responses and therefore change the subtracted calibration term. Furthermore, although holding the student prefix fixed appropriately controls for differences between independently sampled trajectories, an original-response prefix may become inconsistent with the counterfactual condition. Distributional shifts at later positions could then reflect both sensitivity to the changed condition and the teacher’s response to that inconsistency.

These possibilities do not invalidate the method, but leave its mechanism and robustness incompletely understood. Experiments using multiple valid rewrites of the same intervention, together with analyses distinguishing compatible from conflicting prefixes, would help determine whether the selected positions and training benefits remain stable. The current evidence supports the usefulness of the tested construction pipeline more strongly than robustness across alternative valid constructions.

Questions for authors

see above weakness

Reviewer_WeJQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a method (CROP) that chooses which response tokens receive teacher supervision in on-policy distillation.

For each training prompt, an LLM writes a paraphrase that preserves the meaning and a counterfactual that changes one condition.

The teacher re-scores the student's response under all three prompts. Tokens whose teacher prediction changes more under the counterfactual than under the paraphrase are kept, and only these tokens are used in the distillation loss.

The paper is well motivated, and the experiments are detailed.

Critical strengths

-Keeping the student response fixed and comparing a matched counterfactual against a paraphrase separates prompt sensitivity from differences between responses.

-The paper is well written and has useful controls (e.g. random selection, a reversed ranking, and teacher vs. student scoring).

Critical weaknesses
  • The contribution is incremental: CREDIT (Shen et al., 2026) already re-scores a fixed response under another prompt to select tokens. Concurrent work, Counterfactual Constraint-Conditioned OPD (2026), uses a similar counterfactual re-scoring idea; this is not counted as a major weakness, but the paper should discuss it.

  • Eq. (2)–(3): The two distributions use different leftover bins, so the divergence can be positive even when they are nearly the same. This mostly affects the paraphrase term, which is subtracted in Eq. (5).

  • Table 2: Each part of the score alone does no better than random, and the reversed ranking beats random.

  • Eq. (14): Keeping fewer tokens gives each one more weight, which acts like a larger step size. The gain may come from this rather than from token choice.

  • All results are single-seed with no error bars. The gains are small and come mostly from AIME, where a few extra correct answers move the score.

  • In Case C (Fig. 8), the answer changes from an integer to an irrational number, and the dice example (Appendix A.6) is not a standalone problem.

  • Extra cost of the additional teacher passes and the LLM-built data is not reported.

Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.