acceptodds
Under review as a conference paper at ICLR 2027

OPAD: On-Policy Action Distillation for Large Language Model Agents

Abstract

On-policy distillation trains language model agents with teacher guidance at student-visited states, but token-level matching can entangle decision learning with imitation of the teacher's text. The environment responds only to executed actions, making them a natural unit of knowledge transfer. We introduce On-Policy Action Distillation (OPAD), which aligns teacher and student preferences over executable actions. We show that the student's success gap to a reference that keeps the student's reasoning but executes the teacher's actions is bounded by missing or mismatched actions at student-visited states. OPAD builds teacher targets from sampled responses, scores candidate actions after the student's reasoning, concentrates supervision at local peaks of action-span entropy, and weights it by trajectory outcomes and teacher support to balance success consolidation and failure correction. The action interface supports cross-tokenizer and black-box teachers without vocabulary alignment or teacher logits. On ALFWorld, WebShop, and ScienceWorld, OPAD outperforms token-level and black-box distillation baselines in nearly all comparisons across student sizes, model families, and teacher access settings, with several students surpassing their teachers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agent, On Policy Distillation, Uncertainty-Guided Supervision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7vyQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes On-Policy Action Distillation (OPAD), a framework for transferring decision-making capabilities from teacher LLM agents to student agents. Unlike existing on-policy distillation approaches that primarily operate at the token level, OPAD aligns teacher and student distributions over complete executable actions. The method constructs teacher action distributions from sampled responses, estimates student action preferences using normalized continuation likelihoods, selects supervision positions based on action-span entropy, and applies outcome-dependent weighting.

Experiments on ALFWorld, WebShop, and ScienceWorld demonstrate improvements over several existing baselines across different student sizes, model families, and teacher-access settings.

Critical strengths
  1. Strong empirical performance across multiple settings. The paper provides comprehensive experimental comparisons, demonstrating consistent improvements over several relevant baselines. The experiments cover different student sizes, cross-family knowledge transfer, and black-box teacher settings. The component ablations also provide evidence that the proposed design choices contribute to the improvements. These results constitute the strongest argument for acceptance.

  2. Practical flexibility in teacher–student compatibility. Using executable actions as a shared supervision interface is practically appealing, especially when teacher and student models use different tokenizers or when teacher token probabilities are unavailable. The cross-family and black-box experiments demonstrate the practical value of this approach.

Critical weaknesses
  1. Limited methodological novelty. The main contribution appears relatively incremental compared with existing on-policy distillation methods, particularly OPSD-related approaches and action-level imitation. The framework primarily combines established techniques, including action-level distribution matching, entropy-based supervision selection, and outcome-dependent weighting. While these choices are reasonable, the paper offers limited new methodological insights beyond adapting existing distillation principles to structured action spaces.

  2. Discrepancy between the optimized action distribution and actual agent behavior. The student action distribution in Eq. (2) is constructed using length-normalized continuation likelihoods over a finite candidate set, which does not necessarily reflect the actual executed-action distribution under autoregressive generation. Consequently, minimizing the proposed KL objective does not directly guarantee a reduction in the action-distribution mismatch considered in Theorem 1. This weakens the theoretical justification for the proposed optimization objective.

Questions for authors

OPAD-API simultaneously changes the teacher-target construction and introduces bidirectional KL. Could the authors isolate the contributions of these two modifications to the improvements reported in Table 2?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7vyQ (Part 1/2): W1

We thank you for recognizing OPAD’s empirical performance and practical flexibility across teacher–student settings. Your comments helped us clarify the methodological contribution and strengthen the connection between theory and implementation. We address your concerns below.

W1: Methodological novelty.

Our motivation is that token-level OPD can couple decision transfer with imitation of the teacher’s textual expression: different reasoning can produce the same executable action, while changing an action argument can alter its environmental consequences. Direct token-probability matching also introduces compatibility constraints across tokenizers and black-box teachers.

OPAD’s core contribution is to construct action-distribution supervision for autoregressive agents, with components designed around this supervision interface. OPSD-related agent methods such as SERL shape token-level credit assignment or distillation. Within action imitation, LLM-agent DAgger learns teacher-provided response labels, while classical policy distillation [1] operates on directly available action distributions. OPAD constructs comparable action preferences from text-generating models: teacher responses are aggregated by parsed executable action, and the student scores the same alternatives after its own reasoning. This transfers relative decision preferences into the student’s reasoning context without making teacher-written reasoning the imitation target, while supporting different tokenizers and generation-only teachers.

Selection and weighting are adapted to the requirements of this action-level supervision. Querying every interaction step is costly, and guidance value varies across decisions; action-span uncertainty guides query allocation, with local peaks reducing concentration within a persistently uncertain segment. Weighting combines trajectory outcomes with the same teacher distribution’s support for the executed action, distinguishing locally supported decisions from questionable ones within both successful and failed trajectories. These components therefore connect target construction, query allocation, and supervision strength through a common decision unit.

To further isolate the value of the distributional target, we added an Action Semantic Feedback ablation that retains the supervision positions, teacher sampling budgets, outcome weights, and other settings, replacing only distribution matching with scalar feedback on the executed action.

Results after 500 online updates are shown below as SR means ± std:

MethodSeenUnseenHard
Action Semantic Feedback82.57 ± 2.3584.03 ± 1.9317.52 ± 1.82
Full OPAD89.29 ± 0.3586.57 ± 0.7620.83 ± 1.69

Action Semantic Feedback performs well across all three splits, supporting the effectiveness of supervising executable actions. OPAD performs better under otherwise matched settings, indicating that matching preferences across candidate actions provides additional value beyond feedback on the chosen action alone. With selection and weighting held fixed, these gains support the action-distribution formulation itself as a substantive contribution.

We added the results and analysis to Section 4.6 and Figures 6 and 7, with the detailed ablation setup and supplementary analysis in Appendix C.3.

[1] Czarnecki W M, Pascanu R, Osindero S, et al. Distilling policy distillation. PMLR 2019.

AuthorsOpenReview ↗

Response to Reviewer 7vyQ (Part 2/2): W2 & Q1

W2: Connecting the optimized distribution to actual agent behavior.

Theorem 1 motivates aligning executed-action distributions, and Eq. (2) provides a differentiable candidate-scoring surrogate for this purpose. We added diagnostics to examine whether OPAD training improves teacher agreement in the student’s actual generated actions.

We compare the initial and step-500 checkpoints using identical histories, student reasoning prefixes, candidate sets, and teacher targets. Student continuations use temperature 0.4 and top-p 1.0. Below, pmeanp_{\mathrm{mean}} is OPAD’s scoring distribution, psump_{\mathrm{sum}} removes length normalization when rescoring the same checkpoint, κ^\widehat{\kappa} is the sampled executed-action distribution, and qq is the fixed empirical teacher target from 16 samples. TV includes invalid and outside-candidate probability mass; entries are means over diagnostic contexts, with lower values indicating closer distributions.

SplitCheckpointTV(pmean,κ^)\mathrm{TV}(p_{\mathrm{mean}},\widehat{\kappa})TV(psum,κ^)\mathrm{TV}(p_{\mathrm{sum}},\widehat{\kappa})TV(pmean,q)\mathrm{TV}(p_{\mathrm{mean}},q)TV(κ^,q)\mathrm{TV}(\widehat{\kappa},q)
SeenInit0.80630.18450.81960.3794
Seen5000.03580.03790.05560.0421
UnseenInit0.79790.18920.83680.5257
Unseen5000.04560.04880.07640.0443
HardInit0.82780.30980.86950.6631
Hard5000.26870.17660.25890.2108

Both scoring–teacher and execution–teacher discrepancies decrease across all three splits, demonstrating that the learned alignment extends to generated actions. At step 500, mean and summed scores both closely match execution on Seen and Unseen. Hard retains a larger scoring–execution gap, with summed scores closer to execution, while training with OPAD’s original length-normalized objective still substantially improves executed-action agreement with the teacher.

All valid sampled actions lie within the candidate sets. At step 500, candidates cover 98.53%, 98.68%, and 92.02% of sampled continuation mass on Seen, Unseen, and Hard; the remainder consists of invalid actions.

These results empirically support candidate scoring as an effective training surrogate for action-level alignment. We clarified the theory–implementation connection in Section 3.1 and Appendix D.4, and added the diagnostic setup, results, and analysis to Appendix C.8.

Q1: Separating teacher-target construction and bidirectional KL.

We added two ablations using the same API teacher and 500 online updates: replacing direct probability elicitation with 16 sampled teacher responses while retaining SymKL, and replacing SymKL with forward KL while retaining direct elicitation. The sampled-target ablation reports one training run; the forward-KL ablation and OPAD-API reference report means over five independent training runs. Costs are ALFWorld teacher API expenses per update.

MethodSeenUnseenHardCost ($/update)
Sampled target + SymKL91.4388.8141.320.933
Direct probabilities + forward KL82.8680.6032.230.087
OPAD-API: direct probabilities + SymKL83.2982.6934.210.091

Sampling achieves higher observed SR at 10.25x the API cost, supporting our motivation for direct elicitation: reducing teacher-query costs. With direct elicitation fixed, the SymKL reference achieves higher observed SR than the forward-KL variant on all three splits, supporting the choice to combine coverage of reasonable alternatives with concentration on teacher-preferred actions. Forward KL also exceeds the reported black-box baseline means on all three splits, showing that the observed advantages persist without the reverse-KL term.

We added these ablations and their analysis to Appendix C.7, with their motivation and conclusions summarized in Section 4.4.

We appreciate your careful assessment and hope the added comparisons and diagnostics clarify OPAD’s contribution and its connection to actual agent behavior. We welcome further discussion and would be happy to address any remaining questions.

AuthorsOpenReview ↗

Follow-up to Reviewer 7vyQ

Dear Reviewer 7vyQ,

Thank you again for your careful assessment. We clarified OPAD’s methodological distinction from prior distillation and imitation approaches and added a controlled Action Semantic Feedback ablation. We also provided execution-distribution diagnostics addressing the theory–implementation connection and separate ablations of OPAD-API’s two modifications.

As the discussion period draws to a close, we would particularly value your feedback on whether this additional evidence addresses your concerns about novelty and the practical optimization objective. We welcome further discussion and would be happy to answer any remaining questions.

Best regards,

The authors of Submission 27476

Reviewer_AdC8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OPAD (On-Policy Action Distillation), an on-policy distillation method for LLM agents. Unlike conventional token-level OPD, OPAD constructs probability distributions over executable actions through teacher sampling and student candidate scoring, aligning them using KL divergence. Furthermore, the paper introduces entropy-guided supervision selection and outcome-aware weighting to improve distillation efficiency and decision-making capability. Experiments on ALFWorld, WebShop, and ScienceWorld demonstrate the effectiveness of the proposed method across different student sizes, model families, and black-box teacher settings.

Critical strengths
  1. For interactive agents, shifting the distillation target from token-level distributions to executable action distributions is practically meaningful and naturally supports cross-tokenizer and black-box distillation.
  2. The method achieves consistent performance gains across multiple agent benchmarks, student sizes, and teacher settings.
  3. The paper provides extensive analyses covering action-level supervision, teacher distribution construction, supervision selection, outcome weighting, and different distillation objectives.
Critical weaknesses
  1. The technical novelty relative to existing logit-free and agent-oriented OPD methods could be better established. OPAD shares similarities with OmniOPD in on-policy rollouts, teacher sampling, and entropy-guided supervision selection. In particular, OmniOPD already introduces a peak-entropy scheduler, while OPAD adopts a related selection principle with action-span aggregation and peak prominence filtering. SERL also explores action-span selective distillation and feedback-based reweighting in multi-turn agents. Although explicit action-level distribution matching represents a meaningful distinction, the existing experiments do not fully establish its advantages over simpler action-level semantic feedback methods.

  2. The fairness and comparability of black-box OPD baselines are not sufficiently established. Methods such as GAD and ROPD were originally developed for text generation or reasoning tasks, where the supervision granularity differs from multi-turn agent decision-making. The paper provides limited details on how their feedback mechanisms and credit assignment are adapted to the agent setting. Furthermore, OPAD-API introduces both direct teacher probability elicitation over candidate actions and a bidirectional KL objective, whereas other black-box baselines primarily rely on generative or evaluative feedback. These differences complicate the attribution of its performance gains to action-level distribution matching alone.

  3. There is a gap between the theoretical analysis and the practical optimization objective. Theorem 1 bounds the performance gap using the discrepancy between the student's actual executed-action distribution and the teacher's action distribution. However, OPAD optimizes a candidate-scoring distribution constructed through length-normalized action log-likelihoods and softmax normalization (Eq. 2), which is not equivalent to the executed-action distribution under autoregressive decoding. The restricted candidate set and length normalization can further alter the relative action probabilities. In addition, Appendix A.2 introduces protocol initialization and a gated action-start support loss to encourage valid action generation, but these components lack independent ablations. Consequently, the theorem motivates action-level supervision but does not directly establish that minimizing the proposed surrogate reduces the executed-action discrepancy appearing in the performance bound.

Questions for authors
  1. Beyond the existing token-level and top-1 preference ablations, could the authors compare OPAD against a step-level action-semantic feedback baseline under matched supervision positions and teacher query budgets? This would help isolate the benefit of explicit distribution matching from action-level semantic supervision more generally.

  2. How are GAD and ROPD adapted to multi-turn agents, particularly regarding feedback granularity and credit assignment? Additionally, could the authors separately analyze the contributions of direct teacher probability elicitation and bidirectional KL in OPAD-API to ensure a clearer comparison with existing black-box methods?

  3. How closely does the candidate-scoring distribution in Eq. 2 approximate the student's actual executed-action distribution? Could the authors analyze the effects of candidate coverage and length normalization, and provide an independent ablation of the gated action-start support loss (λsup=0\lambda_{\mathrm{sup}}=0)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer AdC8 (Part 1/2): W1-W2 & Q1-Q2

We thank you for recognizing the practical value of action-level distillation and our empirical results. Your comments helped us strengthen the comparisons and analysis. We address them one by one below.

W1. Distinction from OmniOPD and SERL.

OPAD’s central contribution is to distill preferences over complete executable actions. Sampled teacher responses are parsed and counted into an action distribution, which the student matches by scoring alternative actions after its own reasoning.

OmniOPD was developed for mathematical reasoning and code generation, with supervision targeting text chunks rather than executable agent actions. Both methods use teacher sampling and entropy-guided selection, but OmniOPD selects token positions within generated text. OPAD aggregates entropy within action spans and selects peaks across interaction steps, aligning supervision with complete environment decisions. Its teacher samples provide a distribution over alternative actions, rather than semantic feedback on a sampled text chunk.

SERL addresses credit assignment through hindsight-based GRPO reweighting and auxiliary action-token distillation. Restricting distillation to action spans answers which tokens receive supervision; using complete actions as the distillation unit answers what the teacher and student should align over. OPAD makes the relative probabilities of alternative executable actions the matching target.

We clarified the OmniOPD comparison in Section 2 and the selection of supervision across interaction steps in Section 3.3.

W1 & Q1. Distribution matching versus scalar action-semantic feedback.

Thank you for suggesting this useful variant. We conducted an additional controlled comparison with Action Semantic Feedback. At each selected step, it uses the same 16 teacher samples to score the student’s chosen action and weights its action-likelihood loss by this scalar feedback. The supervision-selection rule, outcome-based budgets and weights, protocol initialization, action-start support, and optimization settings remain unchanged.

Results after 500 online updates are shown below as SR means ± std:

MethodSeenUnseenHard
Action Semantic Feedback82.57 ± 2.3584.03 ± 1.9317.52 ± 1.82
Full OPAD89.29 ± 0.3586.57 ± 0.7620.83 ± 1.69

Action Semantic Feedback performs well across all three splits, supporting the effectiveness of supervising executable actions. OPAD performs better under otherwise matched settings, indicating that matching preferences across candidate actions provides additional value beyond feedback on the chosen action alone.

We added the results and analysis to Section 4.6 and Figures 6 and 7, with the detailed ablation setup and supplementary analysis in Appendix C.3.

W2 & Q2: Black-box baseline adaptation and OPAD-API design choices.

All methods share the teacher, student, training data, protocol initialization, action-start support, and evaluation settings. We clarified their agent adaptations in Appendix B.4: each student-visited history serves as the input, and feedback evaluates the complete next-step response. GAD uses a learned local discriminator; ROPD uses shared teacher-induced rubrics and blind verification. Both normalize rewards within response groups at the same history for GRPO updates, assigning each advantage to the corresponding response tokens.

As explained in Section 4.4, direct probability elicitation reduces API queries, while symmetric KL combines coverage of reasonable alternatives with concentration on teacher-preferred actions. To separate these two changes, we added two ALFWorld ablations using the same API teacher and 500 updates. The sampled-target ablation reports one training run; the forward-KL ablation and OPAD-API reference report means over five independent training runs. All costs below are ALFWorld teacher API expenses per update.

MethodSeenUnseenHardCost ($/update)
Sampled target + symmetric KL91.4388.8141.320.933
Direct probabilities + forward KL82.8680.6032.230.087
OPAD-API: direct probabilities + symmetric KL83.2982.6934.210.091

Sampling achieves higher SR at 10.25x the API cost, supporting direct elicitation as a cost-efficient choice. With direct elicitation fixed, the symmetric KL reference achieves higher mean SR than the forward-KL variant on all three splits, supporting our loss choice. Meanwhile, forward KL still exceeds the reported black-box baseline means on all three splits, showing that the observed gains persist without the reverse-KL term. We added these results to Appendix C.7 and their motivation and conclusions to Section 4.4.

AuthorsOpenReview ↗

Response to Reviewer AdC8 (Part 2/2): W3 & Q3

W3 & Q3. Connecting candidate scoring to executed actions.

Theorem 1 motivates aligning executed-action distributions, and OPAD implements this idea using the differentiable candidate-scoring distribution in Eq. (2). We added diagnostics to examine whether training with this surrogate improves teacher agreement in actual action generation.

We compare the initial and step-500 checkpoints using identical histories, student reasoning prefixes, candidate sets, and teacher targets. Student continuations use temperature 0.4 and top-p 1.0. Below, pmeanp_{\mathrm{mean}} is OPAD’s scoring distribution, psump_{\mathrm{sum}} removes length normalization when rescoring the same checkpoint, κ^\widehat{\kappa} is the sampled executed-action distribution, and qq is the fixed empirical teacher target from 16 samples. TV includes invalid and outside-candidate probability mass; entries are means over diagnostic contexts, with lower values indicating closer distributions.

SplitCheckpointTV(pmean,κ^)\mathrm{TV}(p_{\mathrm{mean}},\widehat{\kappa})TV(psum,κ^)\mathrm{TV}(p_{\mathrm{sum}},\widehat{\kappa})TV(pmean,q)\mathrm{TV}(p_{\mathrm{mean}},q)TV(κ^,q)\mathrm{TV}(\widehat{\kappa},q)
SeenInit0.80630.18450.81960.3794
Seen5000.03580.03790.05560.0421
UnseenInit0.79790.18920.83680.5257
Unseen5000.04560.04880.07640.0443
HardInit0.82780.30980.86950.6631
Hard5000.26870.17660.25890.2108

OPAD training reduces both scoring–teacher and execution–teacher discrepancies across all three splits, demonstrating that the learned alignment extends to generated actions. At step 500, mean and summed scores both closely match execution on Seen and Unseen. Summed scores are closer on Hard, where the scoring–execution gap remains larger; nevertheless, training with OPAD’s length-normalized objective substantially improves executed-action agreement with the teacher target.

Regarding candidate coverage, all valid sampled actions lie within the candidate sets. At step 500, candidates cover 98.53%, 98.68%, and 92.02% of sampled continuation mass on Seen, Unseen, and Hard; the remaining mass consists of invalid actions.

These results empirically support candidate scoring as an effective training surrogate for action-level alignment. We clarified the theory–implementation connection in Section 3.1 and Appendix D.4, and added the diagnostic setup, metric definitions, results, and analysis to Appendix C.8.

W3 & Q3. Independent ablations of protocol adaptation.

To isolate the contributions of protocol initialization and action-start support, we added independent ablations in Appendix C.6. All rows below use 500 online updates and report SR mean ± std, with other settings held fixed.

MethodSeenUnseenHard
w/o protocol initialization85.71 ± 0.7283.28 ± 1.2316.69 ± 1.19
w/o action-start support87.14 ± 0.7885.52 ± 0.8619.17 ± 1.22
Full OPAD89.29 ± 0.3586.57 ± 0.7620.83 ± 1.69

Both components improve SR across all three splits.

Thank you again for your constructive feedback; we welcome any further questions or suggestions.

AuthorsOpenReview ↗

Follow-up to Reviewer AdC8

Dear Reviewer AdC8,

Thank you again for your constructive suggestions. Our rebuttal includes the Action Semantic Feedback comparison you suggested, separate ablations of OPAD-API’s target construction and loss, and diagnostics of candidate scoring versus actual action generation. We also added protocol-adaptation ablations and clarified the baseline adaptations and distinctions from prior work.

With the discussion period nearing its end, we would appreciate your assessment of these additions and any remaining concerns you would like us to address. Thank you for your time.

Best regards,

The authors of Submission 27476

Reviewer_YKyWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OPAD distills LLM agents at the level of whole executable actions instead of tokens. It supervises only a few student steps, chosen at local peaks of action entropy. A failed trajectory gets up to 4 such steps, and a successful one gets 1. The teacher target comes from counting the parsed actions in 16 teacher samples. The student distribution is a softmax over its length-normalized likelihoods of the candidate actions, computed after its own reasoning. The loss is a forward KL, weighted by the outcome and by teacher support. In the main setting, a Qwen2.5-3B student learns from a GiGPO-trained Qwen2.5-7B. On ALFWorld Seen/Unseen/Hard, OPAD reaches 89.29/86.57/20.83 success rate, and Guided-OPD 83.86/80.30/17.19. The paper also tests students from other model families and a black-box teacher (OPAD-API).

Critical strengths
  1. Because OPAD works on actions, it needs neither teacher logits nor a shared tokenizer. This helps across model families. On ALFWorld Seen (Fig. 4), OPAD reaches 79.43 with Llama-3.2-3B and 83.57 with Phi-4-mini. The best baselines that use teacher outputs reach about 73-76, and SimCT, CT-OPD and GOLD about 52-64.

  2. In Table 1, OPAD has the best success rate and the fewest rounds on every split. The baselines are GRPO, StepOPSD, SERL and four token-level OPD methods. OPAD also reaches 70% Seen success rate in 15.23 GPU-hours, against 41-54 for six baselines (Table 7).

Critical weaknesses

OPAD changes several things at once, so the action unit's own share of the gain is unclear. The protocol initialization and the action-start support loss appear only in App. A and are not ablated. The one direct test, token-level supervision at the same steps, loses 4.6 points on Seen. It has no error bars, and its loss is not specified. Removing the outcome-based budgets or weights costs about as much (−6.7 and −4.2). I still lean toward acceptance.

Questions for authors
  1. How were the baselines tuned, and did they share the protocol cold-start? Vanilla OPD at 51.71 on Seen looks low.

  2. App. B.1 says the 121 Hard tasks come from the ALFWorld training split. Were they in OPAD's training pool?

2 comments
AuthorsOpenReview ↗

Response to Reviewer YKyW

We thank you for your thoughtful assessment and for recognizing OPAD’s cross-family applicability and empirical gains. Your comments helped us clarify the contribution of action-level supervision and strengthen the evidence for the protocol-adaptation components. We address your comments one by one below.

W1. Contribution of action-level supervision.

The token-level ablation retains OPAD’s supervision selection, outcome-based budgets and weights, protocol initialization, and action-start support. At each selected history, it minimizes forward KL between teacher and student next-token distributions, conditioned on the same student-generated prefixes. The loss covers the complete response, including reasoning and action tokens, and is averaged over response tokens. This token-level objective follows GKD [1], with the KL direction and other training settings matched to OPAD to isolate the effect of the supervision unit.

Under these matched settings, action-level supervision achieves higher SR than token-level supervision on Seen, Unseen, and Hard. This supports matching complete executable decisions as the distillation target. The budget and weighting ablations demonstrate complementary contributions from supervision allocation. We clarified the token-level objective in Appendix C.3 and added standard deviations to Figures 6 and 7.

[1] Agarwal R, Vieillard N, Zhou Y, et al. On-policy distillation of language models: Learning from self-generated mistakes. ICLR 2024.

W2. Role and presentation of protocol adaptation.

To isolate the contributions of protocol initialization and action-start support to OPAD’s performance, we added the following ablations in Appendix C.6:

MethodOnline updatesSeenUnseenHard
Protocol initialization only014.57 ± 2.7513.28 ± 1.822.98 ± 2.02
w/o protocol initialization50085.71 ± 0.7283.28 ± 1.2316.69 ± 1.19
w/o protocol initialization70088.29 ± 1.1186.12 ± 1.5819.67 ± 1.74
w/o action-start support50087.14 ± 0.7885.52 ± 0.8619.17 ± 1.22
Full OPAD50089.29 ± 0.3586.57 ± 0.7620.83 ± 1.69

Initialization alone yields low SR, while OPAD retains strong performance when either protocol component is individually removed. Extending training without initialization to 700 updates narrows the gaps to full OPAD to 1.00/0.45/1.16 points, supporting initialization’s role in online learning efficiency. Action-start support provides additional gains of 2.15/1.05/1.66 points at 500 updates. The main text now briefly introduces both components and references Appendix A.2, which also links to these ablations.

Q1. Baseline tuning and Vanilla OPD’s performance.

All trained baselines share OPAD’s training pool, protocol initialization, and action-start support. We use official implementations or paper-recommended configurations and tune all methods, including OPAD, on the same validation set with comparable tuning budgets. These settings are now explicit in Appendix B.4.

A plausible explanation for Vanilla OPD’s lower SR is that it applies token-level supervision across full multi-turn rollouts without prioritizing uncertain decisions or adapting supervision to trajectory outcomes. Under a limited training budget, this can allocate substantial supervision to routine steps and textual differences that do not change the executed action. OPAD instead concentrates supervision on selected decisions and weights it using trajectory outcomes and teacher support.

Q2. Purpose of Hard and its inclusion in training.

Yes. All 121 Hard tasks are included in the training pool shared by OPAD and all baselines. Hard is a fixed difficulty stress test inherited from TCOD, which selected training tasks that its original teacher failed in all ten sampled attempts. Its purpose is to evaluate how effectively methods learn to solve these difficult tasks under the same access to training data. We keep the Hard task list fixed across methods; Unseen provides the complementary evaluation of held-out generalization. Appendix B.1 now explicitly clarifies the training-pool membership and evaluation purpose.

Thank you again for your constructive feedback; we welcome any further questions or suggestions.

AuthorsOpenReview ↗

Follow-up to Reviewer YKyW

Dear Reviewer YKyW,

Thank you again for your thoughtful review. Following your suggestions, we added independent protocol-adaptation ablations, included standard deviations in the component-ablation figures, and specified the token-level objective. We also clarified the shared baseline settings and Hard’s inclusion in the training pool.

As the discussion period draws to a close, we would appreciate your feedback on whether our responses and revisions address your concerns. We would be happy to clarify any remaining points.

Best regards,

The authors of Submission 27476

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their thoughtful feedback, which helped us clarify the presentation and strengthen the experimental evidence. We summarize the revisions below and indicate the corresponding reviewer comments for each substantive change.

  1. Clarified OPAD’s distinction from prior methods and its selection across interaction steps in Sections 2 and 3.3.

    In response to Reviewer AdC8 W1 and Reviewer 7vyQ W1.

  2. Introduced protocol adaptation in Section 3.4, referencing Appendix A.2, and added independent initialization and action-start support ablations in Appendix C.6.

    In response to Reviewer YKyW W2 and Reviewer AdC8 W3 & Q3.

  3. Added the Action Semantic Feedback ablation in Section 4.6 and Appendix C.3.

    In response to Reviewer AdC8 W1 & Q1 and Reviewer 7vyQ W1.

  4. Added standard deviations to Figures 6 and 7 and specified the existing token-level ablation’s objective in Appendix C.3.

    In response to Reviewer YKyW W1.

  5. Clarified shared baseline settings, tuning, and GAD/ROPD agent adaptations in Appendix B.4.

    In response to Reviewer YKyW Q1 and Reviewer AdC8 W2 & Q2.

  6. Clarified Hard’s inclusion in the shared training pool and its purpose as a difficulty stress test in Appendix B.1.

    In response to Reviewer YKyW Q2.

  7. Separately ablated OPAD-API’s target construction and KL objective in Appendix C.7, reporting SR, API costs, and run counts, with findings summarized in Section 4.4.

    In response to Reviewer AdC8 W2 & Q2 and Reviewer 7vyQ Q1.

  8. Clarified the theory–implementation connection in Section 3.1 and Appendix D.4, and added execution-distribution, length-normalization, and candidate-coverage diagnostics in Appendix C.8.

    In response to Reviewer AdC8 W3 & Q3 and Reviewer 7vyQ W2.

  9. Adjusted figure and table placement for readability.

For ease of reference, we highlight in blue the content added or revised in the manuscript for this rebuttal relative to the original submission, except for the graphics in Figures 6 and 7, which retain their original color schemes.

AuthorsOpenReview ↗

Thank You and Rebuttal Summary

We thank the reviewers and Area Chair for their time and constructive feedback. As the discussion draws to a close, we summarize OPAD’s contribution, together with the clarifications and additional evidence provided during the rebuttal.

Main Contributions

Our central contribution is to make complete executable actions the shared target of on-policy distillation. For an agent, different reasoning can lead to the same action, while a small change in an action argument can change its consequences. OPAD therefore teaches the student the teacher’s preferences among actions at the states it actually encounters, using its own reasoning as context. This also provides a common supervision interface across different tokenizers and black-box teachers. Selection and outcome-aware weighting build on this same action unit to allocate limited teacher guidance. We provide theoretical motivation for this formulation and demonstrate its practical benefits across agent benchmarks and model families.

We are pleased that the reviews recognize both this practical motivation and the experimental evidence supporting it:

Motivation and Practical Value

  • “Because OPAD works on actions, it needs neither teacher logits nor a shared tokenizer. This helps across model families.”
  • “For interactive agents, shifting the distillation target from token-level distributions to executable action distributions is practically meaningful and naturally supports cross-tokenizer and black-box distillation.”
  • “Using executable actions as a shared supervision interface is practically appealing, especially when teacher and student models use different tokenizers or when teacher token probabilities are unavailable.”
  • “The cross-family and black-box experiments demonstrate the practical value of this approach.”

Evaluation

  • “In Table 1, OPAD has the best success rate and the fewest rounds on every split.”
  • “The method achieves consistent performance gains across multiple agent benchmarks, student sizes, and teacher settings.”
  • “The paper provides extensive analyses”
  • “These results constitute the strongest argument for acceptance.”

During the rebuttal period, the reviewers raised valuable comments that helped us further clarify OPAD’s contribution and implementation. The most frequently mentioned concerns and our efforts to address them are summarized below.

Main Concerns Addressed

  1. Methodological novelty and the contribution of action-distribution matching

We clarified how OPAD differs from token-level OPSD and label-based action imitation: it constructs comparable soft preferences over complete actions from text-generating teachers and students. The new Action Semantic Feedback ablation supports the value of action-focused supervision, while full OPAD’s higher SR on all three ALFWorld splits demonstrates the additional benefit of matching preferences across alternatives.

  1. The connection between the theoretical bound and actual generated actions

Theorem 1 motivates action-level alignment as a target for distillation. To examine how OPAD’s candidate-scoring objective supports this goal in practice, we measured teacher agreement in the student’s generated actions before and after training. On fixed contexts across all three ALFWorld splits, both scoring–teacher and execution–teacher discrepancies decrease after training, providing empirical support for the surrogate’s effectiveness in aligning generated actions.

  1. Separating the two OPAD-API design choices

Separate ablations show that sampled teacher targets achieve higher observed SR at 10.25 times the API cost of standard OPAD-API, while SymKL achieves higher mean SR than forward KL with direct probability elicitation held fixed. These results support direct elicitation as a cost-efficient choice and SymKL for its stronger task performance.

Thank you again for your time and thoughtful feedback. We hope the clarifications and additional results offer a clearer and more complete picture of our work.

Best regards,
The authors of Submission 27476

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.