- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CQF, a selective OPD method that identifies informative tokens by comparing teacher predictions under the original query and a fixed domain anchor. CQF retains tokens with low cosine similarity for distillation. Experiments across several Qwen3 teacher-student configurations show improvements over several baselines.
Critical strengths
- The proposed method is simple and intuitive.
- The paper provides experiments across multiple benchmarks, together with several ablations and analyses.
Critical weaknesses
- Insufficient justification for the proposed selection criterion.
The method uses cosine similarity between teacher distributions to select supervised tokens, but the choice of this metric is not sufficiently justified. Controlled comparisons with alternative selection metrics, such as KL/JSD divergence or entropy-based selection, under the same objective are needed.
- Limited baseline comparisons.
The comparisons with existing selective OPD methods in Table 2 are conducted only on the 0.6B student, with relatively marginal improvements. More importantly, CQF uses clipped advantages and pre-selection token normalization, while the competing selective methods use unclipped advantages and selected-token normalization. These differences make the comparisons difficult to interpret.
The authors should extend the comparisons to 1.7B and 4B students, ideally including another model family beyond Qwen, and ensure consistent advantage clipping and loss normalization. Comparisons with strong OPD baselines such as ExOPD are also needed.
Questions for authors
- Could the authors compare cosine similarity with alternative selection criteria, such as KL/JSD divergence and entropy, under the same objective?
- Could the authors extend the current comparisons to more models under a fair setup (e.g., consistent clipping and normalization)?
- Could the authors include strong OPD baselines such as ExOPD?
The method is simple and clean. I am open to raising my score if the authors address these concerns with convincing results.