Disagreement Is Not Guidance: Auditing and Selecting Supervision in Agentic On-Policy Distillation

On-policy distillation (OPD) uses teacher supervision on states visited by the student. Action- distribution disagreement alone gives little information about whether teacher guidance improves environmental utility. We propose Behavioral Necessity Audit (BNA), which evaluates student and teacher distributions on a fixed executable candidate set and shared continuation policy. BNA jointly measures action-distribution distance, executable outcome distance, and the conditional utility gain from shifting probability toward the teacher. On frozen ALFWorld and official full WebShop populations, outcome-aware transport costs are 93.16% and 93.40% lower than identity-based costs, while beneficial guidance occurs on 17.44% and 35.07% of states. We then propose Horizon-Split Necessity Gating (HSNG), which uses a short executable prefix to select teacher targets. H16 identifies terminally beneficial guidance with AUROCs of 0.8595 and 0.8693 on the original populations. On disjoint follow-up populations of 80 new states from 40 new task clusters per environment, the frozen rule reaches AUROCs of 0.80 and 0.79, with positive accepted terminal gains of $+0.10$ and $+0.07$. In a matched, single-seed ALFWorld micro-update, HSNG reduces mean off-selection KL drift by 90.60% while all six selected states move toward their teacher targets. These results ground supervision allocation in conditional environmental utility rather than surface action disagreement.