- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes On-Policy Action Distillation (OPAD), a framework for transferring decision-making capabilities from teacher LLM agents to student agents. Unlike existing on-policy distillation approaches that primarily operate at the token level, OPAD aligns teacher and student distributions over complete executable actions. The method constructs teacher action distributions from sampled responses, estimates student action preferences using normalized continuation likelihoods, selects supervision positions based on action-span entropy, and applies outcome-dependent weighting.
Experiments on ALFWorld, WebShop, and ScienceWorld demonstrate improvements over several existing baselines across different student sizes, model families, and teacher-access settings.
Critical strengths
-
Strong empirical performance across multiple settings. The paper provides comprehensive experimental comparisons, demonstrating consistent improvements over several relevant baselines. The experiments cover different student sizes, cross-family knowledge transfer, and black-box teacher settings. The component ablations also provide evidence that the proposed design choices contribute to the improvements. These results constitute the strongest argument for acceptance.
-
Practical flexibility in teacher–student compatibility. Using executable actions as a shared supervision interface is practically appealing, especially when teacher and student models use different tokenizers or when teacher token probabilities are unavailable. The cross-family and black-box experiments demonstrate the practical value of this approach.
Critical weaknesses
-
Limited methodological novelty. The main contribution appears relatively incremental compared with existing on-policy distillation methods, particularly OPSD-related approaches and action-level imitation. The framework primarily combines established techniques, including action-level distribution matching, entropy-based supervision selection, and outcome-dependent weighting. While these choices are reasonable, the paper offers limited new methodological insights beyond adapting existing distillation principles to structured action spaces.
-
Discrepancy between the optimized action distribution and actual agent behavior. The student action distribution in Eq. (2) is constructed using length-normalized continuation likelihoods over a finite candidate set, which does not necessarily reflect the actual executed-action distribution under autoregressive generation. Consequently, minimizing the proposed KL objective does not directly guarantee a reduction in the action-distribution mismatch considered in Theorem 1. This weakens the theoretical justification for the proposed optimization objective.
Questions for authors
OPAD-API simultaneously changes the teacher-target construction and introduces bidirectional KL. Could the authors isolate the contributions of these two modifications to the improvements reported in Table 2?
3 comments
Response to Reviewer 7vyQ (Part 2/2): W2 & Q1
W2: Connecting the optimized distribution to actual agent behavior.
Theorem 1 motivates aligning executed-action distributions, and Eq. (2) provides a differentiable candidate-scoring surrogate for this purpose. We added diagnostics to examine whether OPAD training improves teacher agreement in the student’s actual generated actions.
We compare the initial and step-500 checkpoints using identical histories, student reasoning prefixes, candidate sets, and teacher targets. Student continuations use temperature 0.4 and top-p 1.0. Below, is OPAD’s scoring distribution, removes length normalization when rescoring the same checkpoint, is the sampled executed-action distribution, and is the fixed empirical teacher target from 16 samples. TV includes invalid and outside-candidate probability mass; entries are means over diagnostic contexts, with lower values indicating closer distributions.
| Split | Checkpoint | ||||
|---|---|---|---|---|---|
| Seen | Init | 0.8063 | 0.1845 | 0.8196 | 0.3794 |
| Seen | 500 | 0.0358 | 0.0379 | 0.0556 | 0.0421 |
| Unseen | Init | 0.7979 | 0.1892 | 0.8368 | 0.5257 |
| Unseen | 500 | 0.0456 | 0.0488 | 0.0764 | 0.0443 |
| Hard | Init | 0.8278 | 0.3098 | 0.8695 | 0.6631 |
| Hard | 500 | 0.2687 | 0.1766 | 0.2589 | 0.2108 |
Both scoring–teacher and execution–teacher discrepancies decrease across all three splits, demonstrating that the learned alignment extends to generated actions. At step 500, mean and summed scores both closely match execution on Seen and Unseen. Hard retains a larger scoring–execution gap, with summed scores closer to execution, while training with OPAD’s original length-normalized objective still substantially improves executed-action agreement with the teacher.
All valid sampled actions lie within the candidate sets. At step 500, candidates cover 98.53%, 98.68%, and 92.02% of sampled continuation mass on Seen, Unseen, and Hard; the remainder consists of invalid actions.
These results empirically support candidate scoring as an effective training surrogate for action-level alignment. We clarified the theory–implementation connection in Section 3.1 and Appendix D.4, and added the diagnostic setup, results, and analysis to Appendix C.8.
Q1: Separating teacher-target construction and bidirectional KL.
We added two ablations using the same API teacher and 500 online updates: replacing direct probability elicitation with 16 sampled teacher responses while retaining SymKL, and replacing SymKL with forward KL while retaining direct elicitation. The sampled-target ablation reports one training run; the forward-KL ablation and OPAD-API reference report means over five independent training runs. Costs are ALFWorld teacher API expenses per update.
| Method | Seen | Unseen | Hard | Cost ($/update) |
|---|---|---|---|---|
| Sampled target + SymKL | 91.43 | 88.81 | 41.32 | 0.933 |
| Direct probabilities + forward KL | 82.86 | 80.60 | 32.23 | 0.087 |
| OPAD-API: direct probabilities + SymKL | 83.29 | 82.69 | 34.21 | 0.091 |
Sampling achieves higher observed SR at 10.25x the API cost, supporting our motivation for direct elicitation: reducing teacher-query costs. With direct elicitation fixed, the SymKL reference achieves higher observed SR than the forward-KL variant on all three splits, supporting the choice to combine coverage of reasonable alternatives with concentration on teacher-preferred actions. Forward KL also exceeds the reported black-box baseline means on all three splits, showing that the observed advantages persist without the reverse-KL term.
We added these ablations and their analysis to Appendix C.7, with their motivation and conclusions summarized in Section 4.4.
We appreciate your careful assessment and hope the added comparisons and diagnostics clarify OPAD’s contribution and its connection to actual agent behavior. We welcome further discussion and would be happy to address any remaining questions.
Follow-up to Reviewer 7vyQ
Dear Reviewer 7vyQ,
Thank you again for your careful assessment. We clarified OPAD’s methodological distinction from prior distillation and imitation approaches and added a controlled Action Semantic Feedback ablation. We also provided execution-distribution diagnostics addressing the theory–implementation connection and separate ablations of OPAD-API’s two modifications.
As the discussion period draws to a close, we would particularly value your feedback on whether this additional evidence addresses your concerns about novelty and the practical optimization objective. We welcome further discussion and would be happy to answer any remaining questions.
Best regards,
The authors of Submission 27476
Response to Reviewer 7vyQ (Part 1/2): W1
We thank you for recognizing OPAD’s empirical performance and practical flexibility across teacher–student settings. Your comments helped us clarify the methodological contribution and strengthen the connection between theory and implementation. We address your concerns below.
Our motivation is that token-level OPD can couple decision transfer with imitation of the teacher’s textual expression: different reasoning can produce the same executable action, while changing an action argument can alter its environmental consequences. Direct token-probability matching also introduces compatibility constraints across tokenizers and black-box teachers.
OPAD’s core contribution is to construct action-distribution supervision for autoregressive agents, with components designed around this supervision interface. OPSD-related agent methods such as SERL shape token-level credit assignment or distillation. Within action imitation, LLM-agent DAgger learns teacher-provided response labels, while classical policy distillation [1] operates on directly available action distributions. OPAD constructs comparable action preferences from text-generating models: teacher responses are aggregated by parsed executable action, and the student scores the same alternatives after its own reasoning. This transfers relative decision preferences into the student’s reasoning context without making teacher-written reasoning the imitation target, while supporting different tokenizers and generation-only teachers.
Selection and weighting are adapted to the requirements of this action-level supervision. Querying every interaction step is costly, and guidance value varies across decisions; action-span uncertainty guides query allocation, with local peaks reducing concentration within a persistently uncertain segment. Weighting combines trajectory outcomes with the same teacher distribution’s support for the executed action, distinguishing locally supported decisions from questionable ones within both successful and failed trajectories. These components therefore connect target construction, query allocation, and supervision strength through a common decision unit.
To further isolate the value of the distributional target, we added an Action Semantic Feedback ablation that retains the supervision positions, teacher sampling budgets, outcome weights, and other settings, replacing only distribution matching with scalar feedback on the executed action.
Results after 500 online updates are shown below as SR means ± std:
Action Semantic Feedback performs well across all three splits, supporting the effectiveness of supervising executable actions. OPAD performs better under otherwise matched settings, indicating that matching preferences across candidate actions provides additional value beyond feedback on the chosen action alone. With selection and weighting held fixed, these gains support the action-distribution formulation itself as a substantive contribution.
We added the results and analysis to Section 4.6 and Figures 6 and 7, with the detailed ablation setup and supplementary analysis in Appendix C.3.
[1] Czarnecki W M, Pascanu R, Osindero S, et al. Distilling policy distillation. PMLR 2019.