Distilling Actions from Black-Box Teachers: Action-Posterior On-Policy Distillation for Agentic Tasks

On-policy distillation (OPD) trains a language model with teacher supervision on states induced by the model itself. Standard OPD relies on the teacher's next-token logits, which provide a rich supervision signal when available. However, many of the strongest models are proprietary black boxes that expose only sampled text, preventing them from serving as OPD teachers. Token-level supervision is also poorly matched to interactive environments, where task outcomes are determined by executable actions: different responses may induce the same action, while similar responses may lead to different state transitions. We therefore argue that black-box distillation should recover the teacher's distribution over environment-defined decisions rather than imitate its surface text. We introduce Action-Posterior On-Policy Distillation (AP-OPD), a reward-free method that pushes repeated teacher samples through the environment's parser or tool schema, aggregates decision-equivalent responses, and estimates an action posterior at each student-visited state. Because these posteriors concentrate rapidly at most states but remain diffuse at a few decision points, a consensus-gated sampler stops when responses agree and allocates additional queries to ambiguous states. We derive the query-optimal oracle allocation for a prescribed estimation accuracy, whose plug-in form defines the sampler, and train the student model by matching the estimated posterior with action-level cross-entropy. Across ALFWorld, ScienceWorld, and ${\tau }^2$-bench, AP-OPD improves over single-sample on-policy imitation by $12.6\%$ relative on ALFWorld and over white-box OPD by $18.9\%$, while matching the ALFWorld success of ten-sample querying with $54\%$ fewer teacher tokens. Using the black-box DeepSeek-V4-Flash teacher further improves a Qwen3-4B student by $25.1\%$ on ALFWorld and $120.4\%$ on ScienceWorld relative to Qwen3-32B.