acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Query Filtering for On-Policy Distillation

Abstract

On-policy distillation supervises student-generated responses with teacher predictions, but teacher–student disagreement alone does not reveal how strongly those predictions depend on the query. We propose Counterfactual Query Filtering (CQF) to allocate supervision using query sensitivity at fixed student prefixes. CQF compares frozen-teacher predictions under the original query and one fixed anchor per domain, retaining the most sensitive positions for original-query distillation without query-specific rewrites. Across eight mathematics and code benchmarks and four teacher–student configurations, retaining approximately 10% of positions improves the macro-average over full distillation by 0.76–2.58 percentage points in single-run comparisons, also outperforming count-, position-, and advantage-mass-matched random controls. At a shared 0.6B checkpoint, CQF updates align more closely with independent reference gradients and reduce reference loss more than count- and position-matched random selection at equal parameter displacement. Across methods, local diagnostic rankings differ from terminal accuracy rankings, distinguishing update utility from final performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models, token-level supervision allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nY8iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CQF, a selective OPD method that identifies informative tokens by comparing teacher predictions under the original query and a fixed domain anchor. CQF retains tokens with low cosine similarity for distillation. Experiments across several Qwen3 teacher-student configurations show improvements over several baselines.

Critical strengths
  1. The proposed method is simple and intuitive.
  2. The paper provides experiments across multiple benchmarks, together with several ablations and analyses.
Critical weaknesses
  1. Insufficient justification for the proposed selection criterion.

The method uses cosine similarity between teacher distributions to select supervised tokens, but the choice of this metric is not sufficiently justified. Controlled comparisons with alternative selection metrics, such as KL/JSD divergence or entropy-based selection, under the same objective are needed.

  1. Limited baseline comparisons.

The comparisons with existing selective OPD methods in Table 2 are conducted only on the 0.6B student, with relatively marginal improvements. More importantly, CQF uses clipped advantages and pre-selection token normalization, while the competing selective methods use unclipped advantages and selected-token normalization. These differences make the comparisons difficult to interpret.

The authors should extend the comparisons to 1.7B and 4B students, ideally including another model family beyond Qwen, and ensure consistent advantage clipping and loss normalization. Comparisons with strong OPD baselines such as ExOPD are also needed.

Questions for authors
  1. Could the authors compare cosine similarity with alternative selection criteria, such as KL/JSD divergence and entropy, under the same objective?
  2. Could the authors extend the current comparisons to more models under a fair setup (e.g., consistent clipping and normalization)?
  3. Could the authors include strong OPD baselines such as ExOPD?

The method is simple and clean. I am open to raising my score if the authors address these concerns with convincing results.

Reviewer_AsQ1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a method of on-policy distillation which takes into account the teacher’s sensitivity at a given token position to the query, in addition to considering the degree of disagreement between the teacher and the student. The method is based on the idea that using the teacher’s signal only on the tokens that are sensitive to the query should be advantageous compared to using all the tokens since it mitigates the noise arising from the disagreement in non-query-specific tokens. The query sensitivity is measured by the extent to which the teacher predicts similar tokens given the original query and an irrelevant fixed query (as measured by the cosine similarity between the corresponding sparse vectors). The experiments show improvements compared to existing selection methods.

Critical strengths

The algorithm is simple and avoids the need for query paraphrasing/re-writing for each original query, a requirement in existing methods such as CROP [1], reducing the computational resource requirement.

[1] Li, E., He, J., & Du, H. Crop: Task relevance via counterfactuals for selective on-policy distillation, 2026. URL https://arxiv. org/abs/2608.13387.

Critical weaknesses

One of the major concerns is the reasoning behind the functionality of the proposed method. The main insight behind the masking based on cosine similarity with query vs anchor generations is that it provides a proxy to query sensitivity. However, this is undermined by the fact that the student prefix is used in the completion (which can potentially encode information from the original query) and the inability to distinguish between the teacher actually trying to adhere to the query vs the teacher trying to complete the generation based on the student prefix. It raises a question on how this similarity translates to the importance of the supervision signal. On the other hand, unlike CROP which generates per-query reference queries, the proposed method uses a single anchor query which is supposed to be irrelevant to all the possible queries within the group. This way, the anchor does not encode any useful information about the current task.

Another concern is the statistical significance of the experiments, which are limited to a single model family and most importantly a single run for each setting. The clarity of the paper can also be largely improved, particularly in terms of explaining the motivations/insights behind the design choices, properly defining all the symbols used, and including illustrative examples.

Questions for authors
  1. Can the authors provide some representative examples for queries and their corresponding anchors?

  2. What metrics/properties determine the suitability of an anchor for making a better sensitivity estimation? I assume that simply being irrelevant to the current group of queries is not sufficient.

Reviewer_LckKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to improve on-policy distillation. The approach they use is to filter the tokens that the teacher provides supervision for down to a subset of about 10% of tokens, and then use the original distillation loss on those selected tokens.

To select the tokens, they want to see which teacher predictions depend on the question beyond the preceding answer tokens. To do that, they compare the teacher conditioned on the original question and a prefix of the answer with the teacher conditioned on another anchor question and the same prefix of the answer. If these two distributions are similar, they treat that position as less sensitive to the question replacement.

They evaluate several teacher–student distillations. Teachers range from 4B to 30B, and students range from 0.6B to 4B. They show improvements in average accuracy of about 0.8 to 2.6 percentage points across the four main configurations.

Critical strengths

1- The improvement in average accuracy is consistent across different student and teacher sizes, with evaluation across eight math and code benchmarks. 2- The analysis of whether the benefit comes from their selection rule or from selecting any random 10% of tokens is very interesting. They also compare against random selections matched for response position and total absolute advantage.

Critical weaknesses

The comparisons often use only a single training run per configuration. So I can't say for certain whether the improvements are due to random noise or whether their method consistently works better. The authors report three additional runs of their method under a different training schedule, but without corresponding repeated baseline runs.

The paper explores different retention rates and anchor choices for their method, but does not report comparable, separate hyperparameter tuning for the baselines. This leaves uncertainty about whether the baselines are competitive under their best settings. IThe paper acknowledges the absence of a lower-learning-rate full-distillation control, which would help assess whether changes in optimization scale explain some of the gains.

  1. I think training with matched wall-clock time on comparable hardware would make more sense for evaluating practical efficiency. The proposed method uses more training time in the reported comparison, so the authors should also give full distillation the same time budget, allowing additional training steps and rollouts.

  2. I think [1] should be cited and evaluated against. It combines selective token supervision on student-generated trajectories with teacher–student log-probability rewards, reward clipping, and entropy-based token selection. A comparison would help establish the benefit of query-sensitive selection over this earlier selective OPD approach.

[1]: Ko, J., Abdali, S., Kim, Y. J., Chen, T., & Cameron, P. (2026). Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137.

Questions for authors

Since CQF retains roughly 10% of token losses but divides by the original token count, does this effectively reduce the strength of the updates relative to full distillation? I understand that this is not exactly equivalent to reducing the learning rate under Adam, but could differences in optimization scale explain some of the gains? Did you test full distillation with a learning rate one-tenth of CQF’s, or independently tune the learning rates for CQF and the baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.