Counterfactual Query Filtering for On-Policy Distillation

On-policy distillation supervises student-generated responses with teacher predictions, but teacher–student disagreement alone does not reveal how strongly those predictions depend on the query. We propose Counterfactual Query Filtering (CQF) to allocate supervision using query sensitivity at fixed student prefixes. CQF compares frozen-teacher predictions under the original query and one fixed anchor per domain, retaining the most sensitive positions for original-query distillation without query-specific rewrites. Across eight mathematics and code benchmarks and four teacher–student configurations, retaining approximately 10% of positions improves the macro-average over full distillation by 0.76–2.58 percentage points in single-run comparisons, also outperforming count-, position-, and advantage-mass-matched random controls. At a shared 0.6B checkpoint, CQF updates align more closely with independent reference gradients and reduce reference loss more than count- and position-matched random selection at equal parameter displacement. Across methods, local diagnostic rankings differ from terminal accuracy rankings, distinguishing update utility from final performance.