- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CROP (Counterfactual Relevance for On-Policy Distillation), a token-selection method designed to improve the efficiency of on-policy distillation (OPD) by prioritizing tokens that are relevant to the semantics of the current task. CROP estimates task relevance through controlled prompt interventions. Specifically, CROP constructs an original prompt, a meaning-preserving paraphrase, and a counterfactual prompt that modifies one task-relevant condition. Given a fixed student-generated response, the method measures changes in the teacher's token distributions under these three prompts using Jensen–Shannon divergence. The difference between counterfactual sensitivity and paraphrase sensitivity serves as a token-level relevance score, which is used to select tokens for distillation under a fixed supervision budget. Experiments on mathematical reasoning benchmarks across two Qwen teacher–student configurations show that CROP achieves the highest aggregate performance among the evaluated methods at a 20% token supervision budget.
Critical strengths
- The paper identifies a meaningful distinction between optimization need and task relevance in selective OPD, offering an interesting perspective on token selection
- CROP outperforms several selective and dense OPD baselines across two teacher–student settings
- The paper includes matched token-budget controls, component ablations, and token-selection analyses that help characterize the proposed scoring mechanism
Critical weaknesses
-
The empirical improvements are not significant enough. Although CROP achieves the highest aggregate performance in both settings, its improvements over the strongest baselines are relatively small (1.43 points in Setting A and 0.96 points in Setting B). CROP does not consistently outperform competing methods on every benchmark. For example, in Setting A, its AIME25 performance is below several baselines, while in Setting B, TIP and Dense CREDIT obtain stronger results on certain MATH-500 metrics
-
The proposed relevance score may not necessarily reflect the actual training value of a token. The main assumption is that tokens with larger teacher-distribution shifts under counterfactual prompts are more useful for distillation. For example, in Figure 6, the counterfactual changes the coefficient from 3 to 5 in a mathematical expression. A token that simply repeats the original coefficient in the generated response could receive a high relevance score due to this change. But supervising such a token may only reinforce copying information from the input? Not necessarily for solving the math task? I think an analysis of such reward hacking problem should be discussed and evaluated.
-
CROP requires teacher rescoring under the original, paraphrased, and counterfactual prompts for every student rollout. This introduces considerable overhead compared with simpler selection methods such as entropy-based selection. The paper does not report training time, teacher inference cost, or performance under matched computational budgets. This is important because CROP-student achieves 26.37 compared with CROP's 26.98 at the 20% budget (Table 2), so that the benefit of additional teacher-based scoring may be relatively small.
I think my main concern is that given the complexity and additional teacher inference required by CROP, the relatively small improvements over existing baselines seem insufficient to justify the additional computational cost.
Questions for authors
See the weaknesses.
Thanks for the rebuttal. The multi seed results make the small gain over TA-OPD more convincing, and I appreciate the timing breakdown. I had expected teacher-based scoring to be noticeably slower than CROP-student, but the measured step times are nearly identical in your implementation.
My main concern is still whether CROP is worth the added cost over baselines (e.g. TA-OPD). In Setting A, CROP gains just under 1% on All-8 while taking 8.3% longer per step at the same 20% supervision fraction. CROP also requires generating and validating paraphrase and counterfactual prompts, then rescoring them during training. I appreciate that the prompt construction cost can be amortized across runs. Even so, I find the accuracy gain modest relative to the extra training time and complexity.