- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors investigate the effects of several combined feature of teacher-student distillation in LLMs: where do the history of interactions and selected actions come from: the teacher or the student? They mainly consider 4 settings: all from the teacher (TT), all from the student (SS) and both mixed settings (TS and TS). They also consider different "pairing" mechanisms and investigate the effect of a curriculum over all these settings. Besides, they use three different LLMs to perform the study. Among other things, the results show that distilling from student "trajectories" using teacher actions performs best, which is consistent with the good-old DAgger algorithm.
Critical strengths
-
The paper honestly investigates useful questions without trying to put forward the author's new model or solution. In that, it will be useful to people using LLM distillation, as it provides trustable hints on how to perform it efficiently.
-
I'm not aware of any paper investigating the question of using a curriculum over settings as the authors do, even in good-old imitation learning or distillation research using simpler policies.
Critical weaknesses
- The paper is not sufficiently rooted in good-old imitation learning or distillation research using simpler policies. In particular, the DAgger algorithm (that the authors cite, Ross et al. 2011) already establishes their "ST" setting as performing best. In this simpler domain, the teacher has to be transposed into an expert policy, the history of interactions is called a trajectory, and the "prefix" (token prefix or action-prefix , see below) is just an action. Results seem to be similar among the different LLM policies, which suggests that the same conclusions may hold for much wider types of policies, including simpler ones like small MLPs.
- From the simpler perspective, using a teacher + guidance is a way to have a (more or less) expert policy as teacher. The authors do not investigate the impact of different levels of expertise in the teacher, though it it a natural question from their framework
- Being not expert in large LLM agents, I had difficulties understanding what were token prefixes and action-prefixes from reading the paper. Maybe providing a didactical example of the distillation framework in Appendix would help
- Section 4.2 (Analyses) all relies in results that are in appendices, which makes the 9 pages paper far from self-contained. Reading the appendices is necessary to evaluate the paper, which is an unfair way to extend the page limits
Questions for authors
None so far
Minor issues, typos:
Liu and Lab, 2025: the firstname of Lab is Thinking Machines :)
9 comments
** Clarification on token/action prefixes**
“Token prefix” and “current-action prefix” refer to the same object: the tokens already generated within the current action.
Let denote the task instruction and the initial environment observation. Before interaction step t, the history is , where is the completed agent action at step , and is the environment observation following that action. The current action is a token sequence containing a reasoning segment followed by an operation. When predicting its -th token, , the current-action prefix is: Thus, contains completed earlier interactions, while contains the unfinished current action.
The student and teacher predict next-token distributions
\quad\text{and}\quad \pi_T(\cdot\mid h_t,u_{t,j},g),$$ respectively, where $\theta$ denotes the student’s trainable parameters, $\pi_T$ is the frozen teacher, and $g$ is privileged task guidance. We select the history source $H\in�0�$, where $T$ denotes teacher and $S$ student. Their combinations give TT, ST, TS, and SS. The prefix is always generated conditional on the selected history. **Illustrative example: retrieving an apple from box B** Suppose the student has produced the following interaction: ```text Task instruction q: Retrieve an apple from box B. Initial observation o0: There is a box named B. Completed action a1: Thought: I will take the apple from box B. Operation: take("apple", "B") Environment observation o1: Box B is closed; the action failed. ``` At step 2, both ST and SS start from the same student-induced history, $h_2=(q,o_0,a_1,o_1)$. **ST: the teacher generates the current action $a_2$.** ```text Thought: The box is closed, so I need to open it first. Operation: open_box("B") ``` The successive prefixes of this teacher-generated action provide the contexts for next-token distillation. **SS: the student generates the current action $a_2$.** One partial prefix might be: ```text Thought: The box is closed. I will try taking the apple again. Operation: take( ``` At this position, the teacher provides its next-token distribution conditioned on the student-generated prefix. The partial prefix has not yet been executed in the environment. **In both cases, the teacher supplies the supervision and only the student is updated.** We include actual trajectory examples for the four configurations in the revised paper. ### ** Appendix placement** We would like to clarify that Section 4.2 already includes the necessary analysis on training exercises and the post-trained models' behaviour. We report reporting the diagnostic procedures, quantitative findings, and their interpretations. For example, it gives the unresolved-error rates for ST versus TT (82.7–85.0% versus 14.0–19.1%) and the decline in teacher-continuation appropriateness as more student-generated prefix is retained, including 83.8% to 60.2% for Qwen3-14B. These summaries explain how the training contexts affect supervision and the trained students’ behavior. To make the supporting evidence directly accessible, **we have moved tables and figures about training exercises analysis and error recovery analysis into the main text of the revised paper**. Detailed annotation procedures and supplementary results remain in the appendix.Paper revision
We have revised the paper accordingly: (1) Corrected the citation typo identified by the reviewer. (2) Added the complete results using stronger teacher models in Appendix A.2.8. (3) Included trajectory examples for the TT/ST/TS/SS configurations in Appendix A.1.1. (4) Moved the tables and figures on training exercise analysis and error recovery analysis from the appendix to Section 4.2 (now presented as Tables 2 and 3 and Figure 2).
We thank the reviewer for the constructive comments and clarify the main points below.
Connection to DAgger
We agree that ST follows DAgger’s principle of obtaining expert supervision on learner-visited states. However, we want to clarify that our contribution is beyond the established principle in DAgger: (1) Our contribution is to examine how history-level supervision interacts with autoregressive action generation in interactive LLM agents. The teacher conditions on both the completed interaction history and the tokens already generated within the current action. These two contexts can come from different policies. For example, ST and SS both use student-induced histories, but query the teacher on teacher-generated and student-generated current-action prefixes, respectively. (2) We systematically compare these source choices under a common distillation objective, then study how to combine and schedule the four configurations. These research questions are not determined by the general motivation for expert supervision on learner-visited states. DAgger does not establish the relative effectiveness of these source choices or their combinations in our setting.
Level of teacher expertise
Our original setup uses a frozen copy of the student’s initial checkpoint with privileged guidance, controlling for model architecture and initialization and achieving a clean comparison. We use stronger teachers to represent better teacher expertise. We have now repeated the complete Stage 1 study with Qwen3-8B as the student and Qwen3-14B or Qwen3-32B as the teacher, keeping the guidance and training protocol unchanged.
The main findings persist: ST performs best overall, while SS remains weakest on every reported metric. For TT, ST, and TS, success rates increase with teacher size on all four benchmarks. SS shows less uniform gains: the 14B teacher slightly reduces its success rates on OfficeBench and WebShop. This may imply SS is less robust to the changes on the teacher expertise.
Table 1. Stage 1 results with a fixed Qwen3-8B student and different guided teachers. Bold and Italic indicate the best and the second best value, respectively, within each teacher group.
We also extend the Stage 2 pairing and composition comparisons to the Qwen3-32B teacher. Independent again outperforms Diagonal and Crossed on every benchmark, preserving the pairing result at matched marginal teacher/student proportions. The previously selected ST-more composition, with weights (TT,ST,TS,SS)=(0.125,0.5,0.25,0.125), further improves over Independent on all four benchmarks and achieves the highest success rates in Table 2. These results extend the main Stage 1 and Stage 2 findings beyond the original same-checkpoint self-distillation setting.
Table 2. Success rates with a fixed Qwen3-8B student under different mixtures and guided teachers.