acceptodds
Under review as a conference paper at ICLR 2027

Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?

Abstract

Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both have proven effective, we argue that this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action. We introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule. We study this problem through self-distillation, where a frozen teacher has access to task guidance unavailable to the student, and conduct a three-stage controlled study across three models and four interactive environments. We find that student-induced histories paired with teacher-generated action prefixes generally provide the strongest standalone training signal, whereas following the student at both levels generally performs worst. Nevertheless, fully on-policy data becomes beneficial when incorporated into a mixture with other configurations. Moreover, mixtures with identical teacher/student proportions at each level can yield substantially different performance, showing that how the two levels are paired matters beyond the overall amount of on-policy data. Finally, front-loading fully teacher-generated contexts and delaying fully on-policy contexts improves over a static mixture. These findings provide practical principles for deciding where, how much, and when agent distillation should follow the student.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, knowledge distillation, self-distillation, post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xEvMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors investigate the effects of several combined feature of teacher-student distillation in LLMs: where do the history of interactions and selected actions come from: the teacher or the student? They mainly consider 4 settings: all from the teacher (TT), all from the student (SS) and both mixed settings (TS and TS). They also consider different "pairing" mechanisms and investigate the effect of a curriculum over all these settings. Besides, they use three different LLMs to perform the study. Among other things, the results show that distilling from student "trajectories" using teacher actions performs best, which is consistent with the good-old DAgger algorithm.

Critical strengths
  • The paper honestly investigates useful questions without trying to put forward the author's new model or solution. In that, it will be useful to people using LLM distillation, as it provides trustable hints on how to perform it efficiently.

  • I'm not aware of any paper investigating the question of using a curriculum over settings as the authors do, even in good-old imitation learning or distillation research using simpler policies.

Critical weaknesses
  • The paper is not sufficiently rooted in good-old imitation learning or distillation research using simpler policies. In particular, the DAgger algorithm (that the authors cite, Ross et al. 2011) already establishes their "ST" setting as performing best. In this simpler domain, the teacher has to be transposed into an expert policy, the history of interactions is called a trajectory, and the "prefix" (token prefix or action-prefix , see below) is just an action. Results seem to be similar among the different LLM policies, which suggests that the same conclusions may hold for much wider types of policies, including simpler ones like small MLPs.
  • From the simpler perspective, using a teacher + guidance is a way to have a (more or less) expert policy as teacher. The authors do not investigate the impact of different levels of expertise in the teacher, though it it a natural question from their framework
  • Being not expert in large LLM agents, I had difficulties understanding what were token prefixes and action-prefixes from reading the paper. Maybe providing a didactical example of the distillation framework in Appendix would help
  • Section 4.2 (Analyses) all relies in results that are in appendices, which makes the 9 pages paper far from self-contained. Reading the appendices is necessary to evaluate the paper, which is an unfair way to extend the page limits
Questions for authors

None so far

Minor issues, typos:

Liu and Lab, 2025: the firstname of Lab is Thinking Machines :)

9 comments

We thank the reviewer for the constructive comments and clarify the main points below.

Connection to DAgger

We agree that ST follows DAgger’s principle of obtaining expert supervision on learner-visited states. However, we want to clarify that our contribution is beyond the established principle in DAgger: (1) Our contribution is to examine how history-level supervision interacts with autoregressive action generation in interactive LLM agents. The teacher conditions on both the completed interaction history and the tokens already generated within the current action. These two contexts can come from different policies. For example, ST and SS both use student-induced histories, but query the teacher on teacher-generated and student-generated current-action prefixes, respectively. (2) We systematically compare these source choices under a common distillation objective, then study how to combine and schedule the four configurations. These research questions are not determined by the general motivation for expert supervision on learner-visited states. DAgger does not establish the relative effectiveness of these source choices or their combinations in our setting.

Level of teacher expertise

Our original setup uses a frozen copy of the student’s initial checkpoint with privileged guidance, controlling for model architecture and initialization and achieving a clean comparison. We use stronger teachers to represent better teacher expertise. We have now repeated the complete Stage 1 study with Qwen3-8B as the student and Qwen3-14B or Qwen3-32B as the teacher, keeping the guidance and training protocol unchanged.

The main findings persist: ST performs best overall, while SS remains weakest on every reported metric. For TT, ST, and TS, success rates increase with teacher size on all four benchmarks. SS shows less uniform gains: the 14B teacher slightly reduces its success rates on OfficeBench and WebShop. This may imply SS is less robust to the changes on the teacher expertise.

Table 1. Stage 1 results with a fixed Qwen3-8B student and different guided teachers. Bold and Italic indicate the best and the second best value, respectively, within each teacher group.

TeacherSettingToolQA SRpass@3OfficeBench SRpass@3ALFWorld SRpass@3WebShop SRpass@3Task Score
Qwen3-8B+ TT81.789.270.976.769.777.470.278.078.5
+ ST81.191.374.884.971.779.973.383.282.2
+ TS79.887.971.780.269.375.569.179.275.0
+ SS74.582.465.973.360.867.958.966.873.1
Qwen3-14B+ TT82.289.571.377.970.377.771.079.279.9
+ ST82.291.676.084.972.781.074.184.484.9
+ TS80.287.672.582.670.077.069.879.676.3
+ SS74.782.065.575.661.169.058.666.273.3
Qwen3-32B+ TT84.192.374.083.771.579.972.781.281.3
+ ST83.791.677.188.473.781.875.584.885.3
+ TS80.989.273.683.770.979.970.679.680.4
+ SS76.083.667.877.961.970.160.269.876.1

We also extend the Stage 2 pairing and composition comparisons to the Qwen3-32B teacher. Independent again outperforms Diagonal and Crossed on every benchmark, preserving the pairing result at matched marginal teacher/student proportions. The previously selected ST-more composition, with weights (TT,ST,TS,SS)=(0.125,0.5,0.25,0.125), further improves over Independent on all four benchmarks and achieves the highest success rates in Table 2. These results extend the main Stage 1 and Stage 2 findings beyond the original same-checkpoint self-distillation setting.

Table 2. Success rates with a fixed Qwen3-8B student under different mixtures and guided teachers.

TeacherMethodToolQAOfficeBenchALFWorldWebShop
Qwen3-8BStandalone ST81.174.871.773.3
Diagonal81.269.470.262.3
Crossed78.772.972.373.0
Independent81.676.774.175.6
Best composition81.978.376.376.8
Qwen3-32BStandalone ST83.777.073.775.5
Diagonal84.271.372.466.8
Crossed81.075.274.075.2
Independent85.078.376.277.1
Best composition85.479.877.978.5

** Clarification on token/action prefixes**

“Token prefix” and “current-action prefix” refer to the same object: the tokens already generated within the current action.

Let qq denote the task instruction and o0o_0 the initial environment observation. Before interaction step t, the history is ht=(q,o0,a1,o1,,at−1,ot−1)h_t=(q,o_0,a_1,o_1,,a_{t-1},o_{t-1}), where aia_i is the completed agent action at step ii, and oio_i is the environment observation following that action. The current action ata_t is a token sequence containing a reasoning segment followed by an operation. When predicting its jj-th token, xt,jx_{t,j}, the current-action prefix is: ut,j=(xt,1,…,xt,j−1).u_{t,j}=(x_{t,1},\ldots,x_{t,j-1}). Thus, hth_t contains completed earlier interactions, while ut,ju_{t,j} contains the unfinished current action.

The student and teacher predict next-token distributions

\quad\text{and}\quad \pi_T(\cdot\mid h_t,u_{t,j},g),$$ respectively, where $\theta$ denotes the student’s trainable parameters, $\pi_T$ is the frozen teacher, and $g$ is privileged task guidance. We select the history source $H\in�0�$, where $T$ denotes teacher and $S$ student. Their combinations give TT, ST, TS, and SS. The prefix is always generated conditional on the selected history. **Illustrative example: retrieving an apple from box B** Suppose the student has produced the following interaction: ```text Task instruction q: Retrieve an apple from box B. Initial observation o0: There is a box named B. Completed action a1: Thought: I will take the apple from box B. Operation: take("apple", "B") Environment observation o1: Box B is closed; the action failed. ``` At step 2, both ST and SS start from the same student-induced history, $h_2=(q,o_0,a_1,o_1)$. **ST: the teacher generates the current action $a_2$.** ```text Thought: The box is closed, so I need to open it first. Operation: open_box("B") ``` The successive prefixes of this teacher-generated action provide the contexts for next-token distillation. **SS: the student generates the current action $a_2$.** One partial prefix might be: ```text Thought: The box is closed. I will try taking the apple again. Operation: take( ``` At this position, the teacher provides its next-token distribution conditioned on the student-generated prefix. The partial prefix has not yet been executed in the environment. **In both cases, the teacher supplies the supervision and only the student is updated.** We include actual trajectory examples for the four configurations in the revised paper. ### ** Appendix placement** We would like to clarify that Section 4.2 already includes the necessary analysis on training exercises and the post-trained models' behaviour. We report reporting the diagnostic procedures, quantitative findings, and their interpretations. For example, it gives the unresolved-error rates for ST versus TT (82.7–85.0% versus 14.0–19.1%) and the decline in teacher-continuation appropriateness as more student-generated prefix is retained, including 83.8% to 60.2% for Qwen3-14B. These summaries explain how the training contexts affect supervision and the trained students’ behavior. To make the supporting evidence directly accessible, **we have moved tables and figures about training exercises analysis and error recovery analysis into the main text of the revised paper**. Detailed annotation procedures and supplementary results remain in the appendix.

Paper revision

We have revised the paper accordingly: (1) Corrected the citation typo identified by the reviewer. (2) Added the complete results using stronger teacher models in Appendix A.2.8. (3) Included trajectory examples for the TT/ST/TS/SS configurations in Appendix A.1.1. (4) Moved the tables and figures on training exercise analysis and error recovery analysis from the appendix to Section 4.2 (now presented as Tables 2 and 3 and Figure 2).

Reviewer_sYA3OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper decomposes on-policiness in LLM agent distillation into two levels: who generates the interaction history (H) and who generates the token prefix of the current action (A), each teacher (T) or student (S). The four cells map onto existing methods: TT is teacher-trajectory context distillation, ST is DAgger-style teacher relabelling of student states, TS is prefix replay (ReOPD), SS is on-policy self-distillation. All use the same token-level KL objective. The study is self-distillation: teacher and student start from the same checkpoint, the teacher is frozen and receives privileged task guidance, trained with rank-128 LoRA over 8 rounds. Stage 1 trains one student per cell on Qwen3-8B, Qwen3-14B and Ministral-3-14B across ToolQA, OfficeBench, ALFWorld and WebShop: ST has the best success rate in 10/12 settings and pass@3 in 12/12, SS is worst in all 12 (3.6-14.4 points below ST), robust to KL direction. Analyses show student histories contain unresolved errors in 83-85% of exercises vs. 14-19% for teacher histories, and teacher continuations become less appropriate as more student prefix is retained. Stage 2 compares mixtures at matched exercise budgets: at equal marginal on-policiness the independent mixture beats diagonal and crossed; retaining both TT and SS beats dropping either; a hybrid-heavy mixture improves over independent by 0.1-1.9 points; ST-more beats TS-more. Stage 3 schedules the independent mixture: front-loading TT speeds early learning, delaying SS raises final validation SR by 2.0-2.9 points, and their combination is best on validation.

Critical strengths
  1. The factorial design cleanly separates the two levels, and the Stage 1 result is both large and consistent: SS is worst in all twelve backbone-environment settings by 3.6-14.4 points, ST is best or within 0.6 points of best in all twelve, and the ordering survives switching forward to reverse KL. The accompanying analyses (unresolved-error rates in Table 6, teacher-continuation appropriateness in Table 8, the recovery evaluation in Fig. 7) give a coherent mechanism rather than just a ranking. This is the part of the paper I am confident in and the main reason I lean toward acceptance.

  2. The questions asked (where, how much, when) are the ones a practitioner actually faces, and the answers are stated as operational rules with matched exercise budgets, which makes the study directly usable.

Critical weaknesses
  1. Every configuration is trained once. The subscripts in Table 1 are standard errors over three evaluation trials of a single trained model, i.e. decoding variance, not training variance. This is tolerable for the Stage 1 ordering, whose gaps are large, but Stage 2 rests on single-run differences of 0.1-1.9 points (Table 9, described as improving "in every setting") and Stage 3 on 0.7-2.9 points measured on validation only, with no test numbers and with the weaker independent mixture rather than the paper's best. On OfficeBench the test set has 86 tasks, so one point is about one task, and its validation set has 9 tasks. The Stage 2 and Stage 3 principles are therefore plausible but not established. A related ambiguity: Stage 2 says mixtures "reuse the initial Stage 1 exercise pools"; if ST/TS/SS data are not refreshed with the mixture-trained student each round, the "on-policy" components of the mixtures are not on-policy for that student, and the Stage 3 timing claims lose their meaning. The per-round weight vectors in Stage 3 are also not given; SS-later trailing the static mixture early is hard to reconcile with Stage 1 unless the reallocated SS weight went to a weak configuration.

  2. The scope of the headline claim is wider than the evidence. The teacher is the same checkpoint plus a hint, so its only advantage over the student is the guidance, and on student-generated prefixes that advantage is diluted; "fully on-policy performs worst" may be a property of self-distillation with a weak teacher rather than of agent distillation, and it runs against OPD results obtained with strong external teachers. No strong-teacher control is run, yet the title and abstract speak of LLM agent distillation in general, and ST > TT is the DAgger result the paper itself cites. A second confound sits inside Stage 1: TT draws from three teacher trajectories per task cached once and never refreshed, while ST, TS and SS regenerate student-dependent data every round, so history source is entangled with data freshness and diversity.

Questions for authors
  1. How many training seeds underlie each number? Please add at least three seeds for the Stage 2 pairing comparison (Fig. 2) and the Stage 3 schedules (Fig. 4) on one backbone, and report Stage 3 on the test split.
  2. Please give the exact per-round weight vectors for TT-first, SS-later and the combined schedule, i.e. which configurations absorb the reallocated weight, and confirm whether mixture runs refresh ST/TS/SS data with the mixture-trained student each round.
  3. Does the Stage 1 ordering hold with a stronger external teacher, e.g. Qwen3-32B with guidance teaching Qwen3-8B on one benchmark? And does the TT result change if teacher trajectories are refreshed each round, or cached at eight rather than three?

I would move to clear acceptance if (1) confirms the Stage 2 and Stage 3 effects beyond seed noise and (3) shows the ordering is not a self-teacher artifact, or if the claims are rescoped to self-distillation.

9 comments

We thank the reviewer for the constructive comments and clarify the main points below.

Results across training seeds and on the test split

The original experiments used one training seed, 42. We have now added three training seeds (0, 1234, and 2024) giving four independent training runs per configuration for the comparisons below. The new tables report the mean and standard deviation across training seeds.

Table 1. Qwen3-14B pairing comparison across four training seeds. Subscripts denote standard deviations across training seeds. Bold indicates the highest mean performance among three matched-marginal mixtures.

MethodToolQA SRToolQA pass@3OfficeBench SROfficeBench pass@3ALFWorld SRALFWorld pass@3WebShop SRWebShop pass@3Task Score
Standalone ST85.40.385.4_{0.3}92.40.392.4_{0.3}77.70.277.7_{0.2}89.50.989.5_{0.9}74.40.474.4_{0.4}83.80.683.8_{0.6}79.30.579.3_{0.5}86.90.686.9_{0.6}85.20.285.2_{0.2}
Diagonal83.20.283.2_{0.2}88.40.288.4_{0.2}71.10.271.1_{0.2}78.20.678.2_{0.6}69.30.569.3_{0.5}78.20.378.2_{0.3}70.30.470.3_{0.4}78.20.478.2_{0.4}81.00.681.0_{0.6}
Crossed82.60.182.6_{0.1}90.00.290.0_{0.2}77.90.077.9_{0.0}85.80.685.8_{0.6}75.50.675.5_{0.6}82.70.482.7_{0.4}79.70.379.7_{0.3}86.30.586.3_{0.5}85.00.385.0_{0.3}
Independent86.80.3_{0.3}93.20.4_{0.4}80.00.2_{0.2}91.60.6_{0.6}77.90.3_{0.3}84.10.2_{0.2}80.70.2_{0.2}87.90.3_{0.3}85.50.4_{0.4}

Independent achieves the highest mean performance among the three matched-marginal mixtures on every reported metric, and improves success rate over standalone ST on all four benchmarks. The pairing result therefore persists across independently trained models.

For Stage 3, we repeated all four schedules with Qwen3-14B across the same four training seeds and evaluated every round on the full 500-task WebShop test set.

Table 2. WebShop test success rates across eight training rounds. Results are means across four training seeds, with standard deviations in subscripts. Bold and italic values indicate the highest and second-highest mean performance in each round.

AllocationRound 1Round 2Round 3Round 4Round 5Round 6Round 7Round 8
Static33.70.633.7_{0.6}42.60.342.6_{0.3}49.70.349.7_{0.3}58.10.458.1_{0.4}61.50.561.5_{0.5}71.10.271.1_{0.2}77.50.477.5_{0.4}80.70.280.7_{0.2}
TT-first40.10.1_{0.1}51.00.4_{0.4}62.90.2_{0.2}71.30.6_{0.6}75.40.3_{0.3}76.90.7_{0.7}79.80.2_{0.2}81.00.281.0_{0.2}
SS-later31.10.131.1_{0.1}38.70.338.7_{0.3}47.60.547.6_{0.5}52.30.252.3_{0.2}61.40.661.4_{0.6}72.50.472.5_{0.4}77.90.377.9_{0.3}81.80.2_{0.2}
TT-first+SS-later38.90.2_{0.2}47.80.1_{0.1}55.60.2_{0.2}62.30.2_{0.2}71.90.5_{0.5}79.40.5_{0.5}81.10.3_{0.3}82.70.1_{0.1}

The test results reproduce the distinct learning dynamics: TT-first leads during rounds 1–5, while SS-later initially progresses more slowly but exceeds the static mixture from round 6 onward. The combined schedule leads during rounds 6–8 and achieves the highest final mean SR. The round-8 comparison evaluates all schedules after the same full training budget.

Exact schedule weights

Weights are ordered as (TT, ST, TS, SS) and specify the sampling proportions of training exercises. The allocation is constant within each four-round phase. With equal exercise budgets per round, all four schedules have the same cumulative mixture weights, averaging to (0.25, 0.25, 0.25, 0.25) over eight rounds.

ScheduleRounds 1–4Rounds 5–8
Static(.25,.25,.25,.25).25,.25,.25,.25.25,.25,.25,.25
TT-first.40,.20,.20,.20.40,.20,.20,.20.10,.30,.30,.30.10,.30,.30,.30
SS-later.30,.30,.30,.10.30,.30,.30,.10.20,.20,.20,.40.20,.20,.20,.40
Combined.40,.25,.25,.10.40,.25,.25,.10.10,.25,.25,.40.10,.25,.25,.40
SS-later trailing the static mixture early is hard to reconcile with Stage 1 unless the reallocated SS weight went to a weak configuration.

Explanation: For SS-later, reducing the early SS allocation from 0.25 to 0.10 increases TT, ST, and TS equally from 0.25 to 0.30. Its early lag does not contradict the standalone ranking: Stage 1 compares configurations trained individually, whereas Stage 3 changes their contributions within mixed training. A weak standalone configuration can still contribute useful supervision within a mixture. The curves in Figure5 shows an early–late trade-off: this SS-later allocation slows initial progress but improves final performance, while combining it with TT-first provides stronger early and final performance.

Data refresh in Stages 2 and 3

Every mixture run refreshes its student-dependent data (ST/TS/SS) after each round using its own current student.

The phrase “reuse the initial Stage 1 exercise pools” refers only to the pools used before the first training update. Subsequent student-dependent data are refreshed separately for each run. Thus, the student-generated components remain on-policy with respect to the corresponding mixture-trained student at each collection round. We replace the ambiguous wording with this explicit description in the revised paper.

Results with stronger teachers

We have repeated the complete Stage 1 comparison with Qwen3-8B as the student and Qwen3-14B or Qwen3-32B as the teacher, keeping the guidance and training protocol unchanged.

Table 3. Stage 1 results with a fixed Qwen3-8B student and different guided teachers. Bold and italic indicate the best and second-best values within each teacher group.

TeacherSettingToolQA SRpass@3OfficeBench SRpass@3ALFWorld SRpass@3WebShop SRpass@3Task Score
Qwen3-8B+ TT81.789.270.976.769.777.470.278.078.5
+ ST81.191.374.884.971.779.973.383.282.2
+ TS79.887.971.780.269.375.569.179.275.0
+ SS74.582.465.973.360.867.958.966.873.1
Qwen3-14B+ TT82.289.571.377.970.377.771.079.279.9
+ ST82.291.676.084.872.781.074.184.484.9
+ TS80.287.672.582.670.077.069.879.676.3
+ SS74.782.065.575.661.169.058.666.273.3
Qwen3-32B+ TT84.192.374.083.771.579.972.781.281.3
+ ST83.791.677.088.473.781.875.584.885.3
+ TS80.989.273.683.770.979.970.679.680.4
+ SS76.083.667.877.461.970.160.269.876.1

The main findings persist: ST performs best overall, while SS remains weakest on every reported metric, including with the Qwen3-32B teacher. TT, ST, and TS achieve higher success rates as teacher size increases on all four benchmarks. SS shows non-monotonic changes with the 14B teacher, although the 32B teacher improves its success rates on all four benchmarks. The observed source preference therefore extends beyond the original same-checkpoint teacher.

We also extended the Stage 2 pairing and composition comparisons to the Qwen3-32B teacher; results appear in the reply to Reviewer xEvM and Appendix A.2.8, Table 11. Independent again outperforms Diagonal and Crossed on every benchmark, and the previously selected ST-more composition (0.125, 0.50, 0.25, 0.125) further improves over Independent on all four benchmarks.

We chose self-distillation for a controlled comparison: teacher and student share their architecture, model size, and initial checkpoint, while guidance provides the teacher with additional task information. The experiments with larger teachers show that our source and mixture findings also extend to the tested non-self-distillation settings. Our primary scope nevertheless remains self-distillation, where we conduct the complete three-stage investigation across three backbones and four environments. This scope is stated in Sections 1 and 2.1 and is now emphasized in the abstract and Introduction.

Teacher-trajectory refresh and cache size

We added two controls for the original three-trajectory TT cache: (1) refreshed TT: regenerate teacher trajectories between training rounds, evaluated on all three backbones. (2) Eight-trajectory cache: cache eight teacher trajectories per task before training, evaluated with Qwen3-14B. The training exercise budget is unchanged.

Table 4a. TT results under different trajectory-refresh and cache settings on ToolQA and OfficeBench.

ModelSettingToolQA SRToolQA pass@3OfficeBench SROfficeBench pass@3
Qwen3-8B+ TT81.789.270.976.7
+ TT (refreshed)81.988.970.976.7
Qwen3-14B+ TT84.991.375.284.9
+ TT (refreshed)85.091.674.884.9
+ TT (cached-at-8)84.691.374.883.7
Ministral-3+ TT82.490.173.381.4
+ TT (refreshed)82.789.872.981.4

Table 4b. TT results on ALFWorld and WebShop.

ModelSettingALFWorld SRALFWorld pass@3WebShop SRWebShop pass@3WebShop Task Score
Qwen3-8B+ TT69.777.470.278.078.5
+ TT (refreshed)69.377.770.077.878.1
Qwen3-14B+ TT72.180.775.681.282.7
+ TT (refreshed)72.380.375.280.882.4
+ TT (cached-at-8)72.181.075.581.282.6
Ministral-3+ TT71.079.972.679.679.9
+ TT (refreshed)71.379.972.379.879.5

Across these controls, TT success rates change by at most 0.4 percentage points, and the overall ST–TT comparison is unchanged. Increasing TT trajectory freshness or cache diversity therefore does not reverse the observed advantage of ST in the settings where ST originally outperformed TT.

Additional clarifications

1. Validation and test evaluation. Stage 2 evaluates all configurations on the full test splits. The original Stage 3 curves used validation sets to track progress after every round on ToolQA, ALFWorld, and WebShop. OfficeBench was not included in Stage 3, given its nine-task validation set. The new four-seed WebShop experiment adds full-test-set evaluation throughout training.

2. Connection to DAgger. ST shares DAgger’s principle of expert supervision on learner-visited states. Our contribution is the controlled study of two separately selectable context sources: the interaction history and the token prefix within the current action. We compare all four source combinations, test their pairing at identical marginal on-policiness, and examine their composition and timing. DAgger does not establish these source interactions or the relative effectiveness of their combinations in our setting, but these are the central questions our study addresses.

Paper revision

We have revised the manuscript accordingly, with changes highlighted in blue:

  1. Clarified the self-distillation scope in the Abstract and Introduction;

  2. Clarified the data refresh for Stages 2 and 3 in Sections 5 and 6;

  3. Specified the mixture weight vectors in Appendix A.1.6;

  4. Added complete results with stronger teachers in Appendix A.2.8;

  5. Added multi-seed results in Appendix A.2.9;

  6. Added analyses of TT trajectory refreshing and cache size in Appendix A.2.10.

In response to your stated conditions for clear acceptance, we have provided multi-seed results to validate the robustness of our findings, demonstrated their generalizability to stronger teachers, and explicitly clarified the self-distillation scope. We believe these additions address your key concerns and would greatly appreciate your reconsideration of the score.

Reviewer_GvrROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework that decomposes the on-policy / off-policy distinction in LLM agent distillation into two levels: the interaction history, consisting of the task instruction and preceding actions / observations, the generation of the current action; and the current-action prefix, i.e., the token sequence of the action currently being generated up to the present position. For each of these two levels, the authors choose either the Teacher or the Student, yielding four configurations (TT / TS / ST / SS), which they compare under a common distillation objective. They further conduct a systematic investigation that extends to mixture proportions and training schedules.

The main conclusions are as follows: ST, which pairs a Student-generated (on-policy) history with a Teacher-generated (off-policy) action prefix, is generally the strongest standalone configuration; not only the total amount of on-policy data but also the pairing of history and prefix sources itself matters; SS, although weak on its own, can become beneficial when included in a mixture; and a schedule that emphasizes TT in the early phase and SS in the later phase outperforms a static mixture. Based on these results, the authors argue that in agent distillation, what matters is not only "on-policy or off-policy," but also "at which level to follow the student," "how to combine the sources," and "when to use each configuration."

Critical strengths
  • The decomposition of the problem is clear. In particular, by treating the interaction history and the current action prefix as separate axes of on-policyness, the paper makes explicit a design space that is difficult to see under the conventional binary on/off-policy view.
  • The paper does not stop at a conceptual reorganization; it compares TT / TS / ST / SS under controlled conditions and further examines mixtures and schedules. The result that performance changes depending on the pairing, even at the same student/teacher proportions, provides experimental support for the value of thinking in terms of two levels.
  • Regarding the finding that ST is strong, the paper goes beyond a mere performance comparison and analyzes why: with student-induced histories, the Teacher has more opportunities to correct the Student's failures (corrective supervision), whereas when the Student generates part of the current action, the Teacher is constrained by that content and has difficulty generating appropriate corrective actions (teacher continuation).
Critical weaknesses
  • The relationship to existing work regarding where the paper's main novelty lies is somewhat unclear. In my understanding, the central contribution of this paper is not a new distillation algorithm per se, but rather the decomposition of existing distillation into the history level and the action-prefix level and its systematic characterization. Accordingly, I would like the authors to more clearly position, through comparison with the most closely related prior work, to what extent the paper's conceptual novelty lies in the two-level decomposition itself, i.e., in treating the history level and the prefix level independently.
  • The authors present their results as "practical principles," but at present the characterization of their dependence on conditions seems insufficient. Although there is an analysis of why ST is strong, it remains unclear under which tasks, teacher-student gaps, or stages of student learning this trend holds. Similarly, for mixtures and schedules, the paper does not reach the point of being able to predict under which conditions the optimal configuration changes. Therefore, at present, it seems more appropriate to interpret these findings as strong empirical rules obtained within the range of the models/environments examined, rather than as broadly generalizable principles.
Questions for authors
  • The analysis of ST's advantage is interesting, but do you have any additional insights into the conditions under which this mechanism holds? For example, could the trend change depending on the teacher-student gap, the student's initial performance, or trajectory length?
  • Stage 1 shows that ST is the strongest standalone configuration and that TT is also effective in the early phase of training. Therefore, a mixture combining only TT and ST seems like a natural baseline. Stage 2 examines the complementarity of the four configurations, but why is a TT+ST mixture not directly compared? Also, do you have additional results on whether your best mixture outperforms a simple combination of TT+ST?
  • Stage 2 compares mixture compositions, and Stage 3 compares schedules using the Independent mixture as the reference. However, the performance obtained by combining the best-performing mixture from Stage 2 (e.g., hybrid-heavy or ST-more) with the best-performing schedule from Stage 3 (TT-first + SS-later) is not reported. Would integrating these yield further improvement?
3 comments

We thank the reviewer for the constructive comments and clarify the main points below.

Clarification on the novelty and the comparison with related work

Our novelty lies in a unified framework and formulation for hierarchical agent distillation that reflects the structure of agent data: interaction histories across environment steps and token prefixes within each action. We explicitly distinguish history-level and action-prefix-level on-policiness, treating them as separate design choices whose pairing also affects training outcomes.

Several closely related works explore sampling choices connected to this formulation: [1] uses context distillation on guided teacher trajectories. [2] studies DAgger-style turn-level mixing. [3] replays teacher-generated histories, corresponding to a TS-style choice. [4] compares context distillation and OPD.

Our contribution goes beyond evaluating individual strategies by systematically characterizing the two levels and their interaction under a common distillation objective. We examine all four configurations, isolate pairing effects while controlling teacher/student proportions at each level, and investigate mixture composition and training schedules.

We have expanded Section 8 to clarify these connections.

[1] Alakuijala et al., “Memento No More: Coaching AI Agents to Master Multiple Tasks via Hints Internalization.”

[2] Li et al., "Revisiting DAgger in the Era of LLM-Agents."

[3] Liao et al., "Multi-Turn On-Policy Distillation with Prefix Replay."

[4] Chen et al., "Rethinking Continual Experience Internalization for Self-Evolving LLM Agents."

Scope of the findings

Our experiments cover four agentic environments, spanning tool use, reasoning, planning, code execution, and web interaction, across three model backbones. The consistent overall advantage of ST across these settings supports its value as a practical starting point. Our findings provide empirical guidance for choosing and evaluating distillation strategies. We have revised the Abstract, Introduction, Section 6, and Limitations to state this scope explicitly.

Conditions under which ST is advantageous

1. Teacher-student gap. We repeated the complete Stage 1 comparison with Qwen3-8B as the student and Qwen3-14B or Qwen3-32B as the stronger teachers, keeping the guidance and training protocol unchanged. ST remains the strongest configuration overall, while SS is weakest on every reported metric. Thus, ST’s advantage extends beyond the same-backbone setting and persists across the teacher capacities evaluated.

Table 1. Stage 1 results with different teacher backbones. The student is Qwen3-8B. Higher values are better. Bold and italic indicate the best and second-best distinct values within each teacher group.

TeacherSettingToolQA SRpass@3OfficeBench SRpass@3ALFWorld SRpass@3WebShop SRpass@3Task Score
Qwen3-8B+ TT81.789.270.976.769.777.470.278.078.5
+ ST81.191.374.884.971.779.973.383.282.2
+ TS79.887.971.780.269.375.569.179.275.0
+ SS74.582.465.973.360.867.958.966.873.1
Qwen3-14B+ TT82.289.571.377.970.377.771.079.279.9
+ ST82.291.676.084.972.781.074.184.484.9
+ TS80.287.672.582.670.077.069.879.676.3
+ SS74.782.065.575.661.169.058.666.273.3
Qwen3-32B+ TT84.192.374.083.771.579.972.781.281.3
+ ST83.791.677.188.473.781.875.584.885.3
+ TS80.989.273.683.770.979.970.679.680.4
+ SS76.083.667.877.961.970.160.269.876.1

2. Student initial performance and learning stage. Our three backbones have different initial performance across the four benchmarks (Table 1 of the paper). ST achieves the overall highest SR and the highest pass@3, supporting robustness across starting models with different capabilities.

3. Trajectory length. For each test task in ALFWorld and WebShop, we generated five reference trajectories using the guided Qwen3-14B teacher and retained tasks with at least one successful trajectory. We defined reference length as the mean number of actions across successful reference trajectories and split tasks into short and long groups using the benchmark-specific median: 9 actions for ALFWorld and 6 for WebShop.

Table 2. Success rate with Qwen3-14B across reference-length groups. ST−SS is measured in percentage points.

Benchmark / GroupTTSTTSSSST-SS
ALFWorld / short72.774.773.569.84.9
/ long71.574.271.865.78.5
WebShop / short75.780.876.370.110.7
/ long75.578.075.765.712.3

ST achieves the highest SR in every group, while SS performs worst. The ST–SS gap is larger in the long groups on both benchmarks. These results show that the observed ranking persists across the evaluated reference-length groups.

Together with the analysis in Section 4.2, these findings support ST as a strong starting point when the teacher can provide useful corrections to student mistakes. Student-generated histories expose those mistakes, while teacher-generated action prefixes allow corrective actions without inheriting potentially erroneous student prefixes.

Comparison with TT+ST

The original Stage 2 comparison used Diagonal, Crossed, and Independent mixtures because each fixes the student-generated share at 50% for histories and 50% for action prefixes. This controls marginal on-policiness and isolates the effect of pairing.

We have now added TT+ST baseline.

Table 3. Qwen3-14B results, including TT+ST. Bold indicates the best result among the mixtures shown.

MethodToolQA SRToolQA pass@3OfficeBench SROfficeBench pass@3ALFWorld SRALFWorld pass@3WebShop SRWebShop pass@3WebShop Task Score
Standalone ST85.292.677.589.574.583.979.486.885.0
Diagonal82.988.271.377.969.878.570.577.680.5
Crossed82.790.177.986.075.382.880.287.085.5
Independent86.692.979.891.977.983.980.988.085.9
TT+ST86.693.276.488.475.783.678.385.884.7

Independent matches TT+ST on ToolQA SR and outperforms all other settings. TT+ST achieves slightly higher ToolQA pass@3, consistent with TT’s competitiveness on that benchmark.

Combining ST-more with the Stage 3 schedule

We originally used Independent as a balanced reference to isolate scheduling effects while preserving each configuration’s total allocation. We have now applied the same scheduling rule to ST-more, preserving its overall composition of (0.125, 0.5, 0.25, 0.125) in (TT, ST, TS, SS) order.

Table 4. Success rate on the test set with Qwen3-14B.

ToolQAOfficeBenchALFWorldWebShop
Static (Independent)86.679.877.980.9
TT-first+SS-later (Independent)88.580.679.182.6
Static (ST-more)87.381.880.081.8
TT-first+SS-later (ST-more)88.882.681.183.0

Combining ST-more with TT-first + SS-later yields further gains on all four benchmarks.

Revisions to paper

We have revised the manuscript, with changes highlighted in blue:

  • Abstract, Introduction, Section 6, and Limitations: clarified the scope of the findings as practical empirical guidance.

  • Section 8 (Related Work): expanded comparisons with closely related approaches and clarified the conceptual and empirical contributions.

  • Appendix A.2.8: added the complete Stage 1 results with stronger teachers.

  • Appendix A.2.11: added configuration rankings across trajectory-length groups.

  • Appendix A.2.12: added the TT+ST mixture comparison.

  • Appendix A.2.13: added the combination of ST-more with TT-first + SS-later.

The authors have addressed my main questions, and the novelty and positioning of the paper are now much clearer. I appreciate the clarification that the contribution lies in a unified framework that separates history-level and action-prefix-level on-policiness and systematically studies their interaction.

I continue to view the paper as being above the acceptance threshold. However, I still have some reservation regarding its significance, as the proposed principles appear to provide useful empirical guidance rather than a broadly established mechanism leading to substantial direct performance improvements. I therefore plan to maintain my current rating.

Comments on the paper

1 comment

Dear AC and reviewers,

We thank all reviewers for their constructive feedback. We briefly summarize our main responses and revisions:

  • Novelty and positioning. We clarified our contribution beyond existing distillation strategies, emphasizing the hierarchical formulation and systematic study of source-level on-policiness, pairing, composition, and scheduling.

  • Additional clarifications. We clarified the data-refresh procedures, scheduling details, configuration examples, and conditions under which ST is advantageous.

  • Additional validation. In response to reviewers' suggestions, we conducted further experiments on stronger teachers, additional mixture and scheduling comparisons, multiple training seeds, and teacher-trajectory refresh/cache controls. These results further support our main findings.

We have incorporated these clarifications and results into the revised manuscript. We appreciate the reviewers' time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.