- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes ReOPD, an offline on-policy distillation method for multi-turn agents. It reuses trajectories recorded by the teacher, lets the student generate the action for only one step on a teacher prefix, and has the teacher provide token-by-token supervision, with no environment interaction during training. The authors call the tradeoff in this setup the prefix trap, provide a two-sided error bound, and on that basis adopt a sampling strategy biased toward early steps. On mathematical reasoning with Python and on search tasks, ReOPD is comparable to or better than online OPD in accuracy and is faster per rollout.
Critical strengths
- The problem has practical relevance. For multi-turn agents, online OPD must interact with the environment at every update. ReOPD reuses teacher trajectories and removes this cost, and rollouts produced by the teacher during RL training can also be used directly.
- The method is simple and easy to integrate into existing pipelines, since it only samples steps on teacher prefixes and needs no extra models or environment calls.
- Within the stated setup, the experiments are reasonably well conducted. They cover multiple teacher and student scales, compare fairly with online OPD under the same teacher objective and update budget, and include ablations of the sampling strategy and the prefix source.
Critical weaknesses
- The positioning of the contribution does not hold. The Introduction and the related work do not establish the necessity of this work, and the related work in the main text is only four lines long. The prefix trap, presented as a core contribution, has already been identified by Wang et al. 2026, and the authors mention it only in one sentence in the appendix. The design based on reference policy roll-in is also closely related to learning-to-search work such as SEARN and LOLS, and the paper does not discuss either connection.
- The experimental setting does not match the motivation. The value of the method lies in settings where environment interaction is expensive, but the experiments use only local environments with very low interaction cost. On the search tasks the accuracy is also on par with OPD, and the practical value of the method has not been demonstrated.
Questions for authors
- Compared with the dual exposure bias proposed by Wang et al. 2026 and with the roll-in design based on a reference policy in learning-to-search, what is new in the problem definition or the method of this paper?
- We would like the authors to add one set of experiments, with a smaller scale being acceptable, in at least one environment where interaction is expensive, such as web, GUI, or software engineering, to show whether ReOPD can maintain accuracy comparable to online OPD and whether the overall cost still has an advantage after trajectory collection is taken into account.
- What evidence shows that the benefit comes from reliability-aware prefix selection rather than only from training early steps more, given that the theory section holds that the decay coefficient should be determined by the gap between the teacher and the student and that teacher reliability also decreases with the number of steps, while the experiments fix κ at 0.6 throughout and do not directly measure teacher reliability?
5 comments
Response Part 2
W2/Q2. Practical value, costly environments, and collection cost
Thank you for the suggestion. Here we add new results for AppWorld [1], a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs.
Setup. We train a teacher Qwen2.5-32B on the training set released together with the AppWorld Benchmark. For the teacher, we set GRPO n=8, max_turns=32, batch size of 16 prompts and max_response_len=32k. Later, we apply OPD or ReOPD to train a student Qwen2.5-14B. The rest of the training and evaluation settings stay the same as [2].
Results. As shown in the following table, both ReOPD and OPD are comparable to each other (ReOPD is slightly better overall), while ReOPD can save at most 32 times the environment interaction (max_turn=32). Overall, ReOPD is speedup of OPD for each update. When taking the trajectory collection into account (suppose we don't have the teacher's trajectories from the teacher's RL), the speedup is , still significantly faster than OPD while maintaining the performance.
| Method | Test-Normal (TGC) | Test-Normal (SGC) | Test-Challenge (TGC) | Test-Challenge (SGC) |
|---|---|---|---|---|
| Teacher | 64.9 | 44.6 | 38.1 | 23.7 |
| Student without training | 21.4 | 7.1 | 11.8 | 3.6 |
| OPD | 41.7 | 25.0 | 22.1 | 12.2 |
| ReOPD | 41.7 | 26.8 | 24.0 | 11.5 |
[1] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
[2] CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RL
Q3. "What evidence shows that the benefit comes from reliability-aware prefix selection rather than only from training early steps more"
Fig. 4 provides evidence beyond the position ablation. With the Qwen3-8B supervising teacher fixed, its own prefixes yield 53.7 average accuracy, compared with 52.9 from a 4B prefix generator and 49.3 from a stronger 32B generator. These results support the relevance of compatibility between the prefix source and supervising teacher.
Fig. 5 additionally measures decreasing student-to-teacher prefix likelihood ratios with depth, providing an occupancy-related diagnostic.
We further conduct a suggested experiment from Reviewer 1Y4n:
Setup. Fix the 8B teacher and initial 4B student. Collect one live trajectory per model on 100 random MATH500 problems. Pair nonterminal teacher/student prefixes from the same problem after 1 and 3 tool interactions, without filtering by correctness. Restore each Python state and sample two teacher continuations with identical decoding settings and additional token budgets (no tool): at most 800 full rollouts (100 questions × 2 depths × 2 prefix sources × 2 continuations), no retraining.
Metrics. Measure final-answer accuracy as a proxy for teacher target quality.
| Depth | Paired N | Accuracy: Teacher-prefix (%) | Accuracy: Student-prefix (%) | Gap (pp) |
|---|---|---|---|---|
| 1 | 200 | 86.0 | 82.0 | +4.0 |
| 3 | 200 | 88.0 | 80.0 | +8.0 |
Interpretation. The teacher performs better on its own prefixes, with the accuracy gap widening from 4 pp at depth 1 to 8 pp at depth 3. Since comparisons fix depth, step position alone cannot explain the gap. These findings support history-dependent teacher reliability.
Response Part 3
Q3. "the theory section holds that the decay coefficient should be determined by the gap between the teacher and the student and that teacher reliability also decreases with the number of steps, while the experiments fix κ at 0.6 throughout and do not directly measure teacher reliability?"
An explicit reformulation of L943:
=\mathbb E_{h}\mathbb E_{A\sim\pi_T(\cdot\mid h)} \left[\log\pi_T(A\mid h)-\log\pi_S(A\mid h)\right],$$ Equation (8), $\kappa\approx\exp(-\gamma\bar c)$, is checkpoint-dependent because the student changes during training. Initial-student calibration therefore yields an initial implied $\kappa$, not a fixed value prescribed throughout training. It also requires a specified $\gamma$; **the theory does not uniquely select it.** Our fixed $\kappa=0.6$ is a practical approximation: Fig. 3b reports 53.4, 53.0, and 53.4 for $\kappa=$ 0.2, 0.4, and 0.6, versus 50.9 for uniform sampling. This supports robustness, not quantitative optimality. **Since the student is changing during training, a golden $\kappa$ should be determined for every optimization step. However, it is too expensive since $\bar c$ requires the teacher to sample the action**, and this action is not used to optimize the student. Therefore, we suggest a modest training of $\kappa$. We didn't extensively tune $\kappa$ for each teacher-student pair and task. Instead, we only tune $\kappa$ for the math task and a teacher–student pair of 8B-4B, then set **a universal $\kappa=0.6$ for both math and search tasks and each teacher–student pair (4B-4B, 8B-4B, 30B-4B, 30B-8B) in the paper**. Figure 3 (right) also shows that **ReOPD achieves an average accuracy $\geq$ 50.9 for any $\kappa$, while OPD's accuracy is 51.0 in Table 1**. **Both show that ReOPD is not sensitive to $\kappa$.**
Response Part 1
Thank you very much for your thorough review and suggestions. Here we address your concerns:
Due to the page limit, we could only put a summary of the related work on the main page, and included a very detailed and complete (~1.5 pages) related work in Appendix A.
Thank you for the suggestion. Since we now have one extra page for the rebuttal, we have already put the detailed related work back on the main page in the updated version.
ReOPD's contribution is a concrete distillation mechanism that separates environment interaction from on-policy action supervision. Given a fixed pool of teacher action–observation trajectories, it trains a multi-turn student using fresh student actions and dense teacher targets, with zero new environment calls during student training. This addresses an operational question beyond identifying exposure bias: how can agent distillation retain action-level on-policy updates when live interaction is unavailable or costly?
Relative to Wang et al. (2026). Wang et al. identify dual exposure bias and address it by monitoring student reasoning, backtracking, generating teacher corrections, and training on stitched trajectories. ReOPD instead replays recorded tool histories and distills fresh student actions at selected turns. It requires neither corrective teacher suffix generation nor backtracking through the environment. The distinction concerns both data generation and the training objective:
Relative to SEARN and LOLS. The connection is broader than reference roll-in: these methods generate action-cost supervision through search and continuation evaluation. ReOPD needs neither a task-cost oracle nor continuation-based cost estimation for its student updates; teacher conditionals provide dense supervision directly. In particular, LOLS uses learned-policy roll-in, so it should not be characterized simply as reference-policy replay. Their roll-in designs do not by themselves specify ReOPD's combination of recorded tool observations, fresh student actions, and environment-free student updates.
Analytical and empirical contribution. Proposition 1 separates student-occupancy mismatch from history-dependent teacher-target error in the replay objective. Equations (5)–(6) then introduce a teacher-support surrogate and geometric occupancy bridge, motivating the practical depth weighting. This distinguishes the formal reliability term from its implementable surrogate. Empirically, Table 1 reports gains over SFT of 11.1 pp (4B/4B) and 8.7 pp (8B/4B) using the same teacher trajectories, showing that the procedure adds value beyond ordinary trajectory imitation. These gains establish practical utility; the methodological distinctions above establish how the procedure differs from the cited approaches.
We have updated the related work (in red) to discuss the connection.
Our main claim is that ReOPD matches or improves online OPD accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout. (L21-22). The main benefit of ReOPD is the mitigation of environment interaction (no need for any environment feedback) and speedup (>4 times) instead of the accuracy improvement.