- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Agentic On-Policy Distillation (AOPD), a method for distilling language-model agents that supervises at the semantic action level rather than the token level. The core idea: at student-encountered states, sample actions from a black-box teacher and candidates from the student, then reward student candidates by their semantic agreement with teacher actions (matching tool name+canonicalized arguments, or message intent/content). The reward feeds a REINFORCE/PPO-style policy gradient over the student's own reasoning and action tokens.
Critical strengths
Black-box, cross-family teacher interface is a genuine practical contribution. AOPD requires only generation access to the teacher, no token log-probabilities, no tokenizer alignment, no shared model family.
The likelihood→score-function derivation (Section 2.1) is clean and correctly identifies that the objective is expected likelihood (not log-likelihood), with a concrete explanation of why the inverse-likelihood factor is absent.
Critical weaknesses
Evaluation is too narrow to support the central comparative claims. The only apples-to-apples distillation comparison (KD, OPD) is on τ²-bench with 178 training tasks. The headline 4-benchmark results (Table 1) contain no distillation baseline. None of the six recent related methods the paper itself surveys (GAD, ROPD, OVD, OmniOPD, MENTOR, ASL) are evaluated.
The reward signal's reliability is load-bearing and unmeasured. Two compounding, unquantified factors determine whether the semantic-agreement reward actually approximates the intended rT: (a) the teacher's action error rate on visited states (teachers are 65–71% on direct eval, so errors are common), and (b) the matcher's false-positive/negative rates. The paper acknowledges both qualitatively in the limitations section but measures neither.
Questions for authors
On a labeled subset of (student candidate, teacher sample) action pairs, what are the matcher's false-positive and false-negative rates against a human gold label?
Can you report KD and OPD on the full 4-benchmark table (Table 1) for many student–teacher configurations (e.g., Qwen3-4B + DeepSeek-V4-Flash)?
What is seed-to-seed variance for the main configurations and for the AOPD-vs-OPD τ²-bench comparison?
The privileged teacher receives "reference tool names and arguments, target states, and required messages from evaluation criteria." How is this different from giving the teacher the answer?