acceptodds
Withdrawn from ICLR 2027

Agentic On-Policy Distillation: Distilling Decisions, Not Tokens

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce Agentic On-Policy Distillation (AOPD), motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, AOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, AOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.4-6.6 percentage points across four out-of-distribution benchmarks. AOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On -bench, AOPD achieves competitive performance against offline distillation and token-level OPD. We further explore Self-AOPD with a same-model teacher and privileged context. Together, these results establish AOPD as a promising approach to agent distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, On-Policy Distillation, Agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ypPjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Agentic On-Policy Distillation (AOPD), a method for distilling language-model agents that supervises at the semantic action level rather than the token level. The core idea: at student-encountered states, sample actions from a black-box teacher and candidates from the student, then reward student candidates by their semantic agreement with teacher actions (matching tool name+canonicalized arguments, or message intent/content). The reward feeds a REINFORCE/PPO-style policy gradient over the student's own reasoning and action tokens.

Critical strengths

Black-box, cross-family teacher interface is a genuine practical contribution. AOPD requires only generation access to the teacher, no token log-probabilities, no tokenizer alignment, no shared model family.

The likelihood→score-function derivation (Section 2.1) is clean and correctly identifies that the objective is expected likelihood (not log-likelihood), with a concrete explanation of why the inverse-likelihood factor is absent.

Critical weaknesses

Evaluation is too narrow to support the central comparative claims. The only apples-to-apples distillation comparison (KD, OPD) is on τ²-bench with 178 training tasks. The headline 4-benchmark results (Table 1) contain no distillation baseline. None of the six recent related methods the paper itself surveys (GAD, ROPD, OVD, OmniOPD, MENTOR, ASL) are evaluated.

The reward signal's reliability is load-bearing and unmeasured. Two compounding, unquantified factors determine whether the semantic-agreement reward actually approximates the intended rT: (a) the teacher's action error rate on visited states (teachers are 65–71% on direct eval, so errors are common), and (b) the matcher's false-positive/negative rates. The paper acknowledges both qualitatively in the limitations section but measures neither.

Questions for authors

On a labeled subset of (student candidate, teacher sample) action pairs, what are the matcher's false-positive and false-negative rates against a human gold label?

Can you report KD and OPD on the full 4-benchmark table (Table 1) for many student–teacher configurations (e.g., Qwen3-4B + DeepSeek-V4-Flash)?

What is seed-to-seed variance for the main configurations and for the AOPD-vs-OPD τ²-bench comparison?

The privileged teacher receives "reference tool names and arguments, target states, and required messages from evaluation criteria." How is this different from giving the teacher the answer?

Reviewer_vwqROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts on-policy distillation to agentic tasks through semantic agreement between student and teacher actions. This is a sensible, practical formulation that enables distillation from black-box teachers without access to token probabilities. However, the mathematical development mainly uses standard marginalization and policy-gradient identities, and the methodological novelty is incremental.

Critical strengths
  • Practical teacher interface requiring neither token probabilities nor tokenizer alignment.
  • Meaningful transfer gains across four benchmark families.
Critical weaknesses
  • The contribution is primarily an adaptation of existing distillation ideas; the core mathematics is standard.
  • Comparative gains are modest: Tau single-trial success is 40.0% versus OPD’s 39.6%, with lower four-trial reliability
  • The experiments do not isolate the benefit of semantic matching over simpler action matching or teacher-response imitation at encountered states.
Questions for authors
  • Are the concrete practical teacher-access settings where AOPD is preferable to token-level OP?
  • Why is expected teacher-action likelihood the appropriate objective for agent distillation, given its preference for the teacher’s most probable action? What benefit motivates this choice over semantic-action log-likelihood?
Reviewer_sr4hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Agentic On-Policy Distillation (AOPD), which trains an agent using semantic agreement between student and teacher actions. Instead of matching token distributions through reverse KL, the method samples teacher actions and uses their agreement with student actions as rewards for policy optimization. The approach is practical since it does not require teacher logits.

Critical strengths

The method provides a straightforward way to distill agent decisions from black-box teachers. The experiments cover multiple agent benchmarks, and the cross-model-family results are encouraging.

Critical weaknesses

1. Insufficiently competitive baselines

My main concern is the choice of baselines. The paper compares AOPD against offline KD, which is essentially SFT on successful teacher trajectories, and OPD, which uses only sampled-token reverse KL. While these are reasonable starting points, they are insufficient to establish the advantage of the proposed method.

AOPD uses RL-style optimization with teacher-derived rewards. More meaningful comparisons would include:

  • OPD/GKD + RL: Agarwal et al. (2024) explicitly investigate combining on-policy knowledge distillation with reinforcement learning. A similar baseline combining reverse-KL distillation with outcome-based RL would be valuable.
  • SFT followed by RL: Vattikonda et al. (2025) study a two-stage pipeline combining teacher-trajectory imitation with subsequent on-policy RL for web agents. This would provide a stronger alternative to the KD baseline.

These comparisons should account for additional outcome supervision. Without them, it is difficult to determine whether AOPD offers advantages over established distillation and RL pipelines.

Furthermore, Self-AOPD uses privileged task information, including private user goals, reference actions, and target outcomes. Given its similarity to On-Policy Self-Distillation (Zhao et al., 2026), a direct comparison with token-level OPSD using the same privileged information would be informative. This would help establish whether action-level supervision offers an advantage over token-level privileged distillation.

2. Limited statistical evidence and evaluation

Each configuration is trained only once. The authors should report results across multiple independent training seeds, including the mean and standard deviation.

Additionally, the paper reports pass^1 and pass^4 on τ²-bench but does not report conventional pass@k. While pass^4 measures repeated-success reliability, pass@k would provide complementary information about the model's ability to find successful trajectories across multiple attempts.

3. Computational cost and fairness of comparisons

The collection procedures differ substantially between AOPD and the baselines. AOPD samples multiple teacher and student responses at each visited state and uses semantic matching, whereas OPD collects complete student trajectories and obtains teacher token log-probabilities. The current experiments do not provide a compute-matched comparison or detailed accounting of teacher-generation and matching costs. This makes it difficult to assess whether the improvements justify the additional computational overhead.

References

  1. Agarwal, R., et al. (2024). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.
    https://proceedings.iclr.cc/paper_files/paper/2024/hash/5be69a584901a26c521c2b51e40a4c20-Abstract-Conference.html

  2. Vattikonda, D., et al. (2025). How to Train Your LLM Web Agent: A Statistical Diagnosis. NeurIPS 2025.
    https://proceedings.neurips.cc/paper_files/paper/2025/hash/4d239452d41ded07139801d7c966c629-Abstract-Conference.html

  3. Zhao, S., et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.
    https://proceedings.mlr.press/v306/zhao26be.html

Questions for authors
  1. Can the authors include stronger baselines such as OPD/GKD combined with outcome-based RL (Agarwal et al., 2024) and SFT followed by RL (Vattikonda et al., 2025)? These would help assess the benefits of AOPD over existing distillation and RL pipelines.

  2. For Self-AOPD, can the authors compare against token-level OPSD (Zhao et al., 2026) using the same privileged information and training setup?

  3. Can the authors report results across multiple independent training seeds, including mean and standard deviation?

  4. Can the authors additionally report conventional pass@k on τ²-bench? This would complement the reported pass^k reliability metric.

  5. Can the authors provide a comparison of training costs, including teacher-generation calls, semantic-matching overhead, and overall compute? How does AOPD compare with OPD under matched computational budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.