acceptodds
Under review as a conference paper at ICLR 2027

Select Before You Act: Pre-Commitment Guidance for Agentic On-Policy Distillation

Abstract

On-policy distillation transfers the capabilities of language model agents through teacher supervision on student-generated interactions. In multi-turn tasks, however, student action errors can alter the environment and undermine the conditions for effective supervision at later turns. We introduce **PREACT**, a pre-commitment teacher-guided approach to multi-turn on-policy distillation. At each turn the student proposes several candidate responses, and a fixed teacher selects one for execution *before* its action reaches the environment Only the selected response is retained for distillation and appended to the interaction history with the resulting observation. Selecting a response before execution shapes not only the current supervision target but also the states in which subsequent learning occurs. **PREACT** thus extends teacher guidance from supervising individual responses to shaping the interactions that produce future training data, while keeping the student as the sole source of responses. Experiments on ALFWorld, ScienceWorld, and WebShop demonstrate that **PREACT** can improve student task performance across multiple model scales and under different initialization settings. These results highlight action commitment as an effective point of teacher guidance for training multi-turn agents. Our code is available at this [link](https://anonymous.4open.science/r/41099/).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
On-Policy Distillation, Large Language Model Agents, Multi-Turn Interaction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yWPWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PreAct, a teacher-guided approach to multi-turn agent distillation. At each turn, the student generates several candidates, and a fixed teacher selects the response with the lowest teacher perplexity for execution and distillation. Experiments on ALFWorld, ScienceWorld, and WebShop show improvements over vanilla OPD across several model configurations.

Critical strengths
  • The proposed approach is reasonable and easy to follow. Selecting responses before execution provides a simple way to influence both the current training sample and subsequent interaction histories.

  • PreAct improves over vanilla OPD across multiple environments and model configurations, including task-adapted initialization. The gains are retained when the student acts independently without teacher selection.

  • The ablations examine relevant design choices, including teacher versus student ranking, candidate budget, and supervision over selected versus all candidates. These provide useful support for the proposed recipe.

Critical weaknesses
  • The baseline comparisons are limited. Closely related methods such as Guided-OPD and SAGE-OPD are discussed but not evaluated, making it difficult to assess PreAct’s advantage over existing improvements to agentic OPD. Competitive SFT or RL student baselines would also help contextualize the results.

  • PreAct takes approximately 1.2–2.2 times the training time of vanilla OPD, but the main comparisons use matched update counts rather than matched training budgets. Random-Commit discards the additional candidates, so it does not establish whether using the extra budget for more OPD training would achieve similar gains.

  • The ablations do not fully separate the benefit of selecting better supervision from that of improving future interaction histories. Comparing PreAct with completed-trajectory filtering changes both selection timing and granularity. The conclusion’s claim that the comparison varies only timing therefore seems stronger than the evidence supports.

  • Teacher perplexity is a plausible ranking signal, but its connection to action quality remains insufficiently examined. Direct ranking-quality measurements or an action-only scoring ablation would help support this choice.

Questions for authors
  • How does PreAct compare with Guided-OPD or SAGE-OPD under comparable settings? Does its advantage over vanilla OPD remain when OPD receives the same total training budget?

  • What happens if training supervises the teacher-preferred candidate at each history, but execution uses a randomly selected candidate? This would help isolate the benefit of using teacher selection to shape subsequent histories.

Reviewer_u2EMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a method PREACT. The student may choose wrong action which will lead the task failed and the action is depend on preview action. In order to fix the problem in OPD. they design a two stage pipeline. First they ask the student to generate 4 candidates action and second they use the teacher model to calculate each token's probability and sum it to. However, this will increase the x1.21-2.22 more training time with respect to Vanilla OPD which only have 1 candidate. They choose the lowest PPL and let the student continue. This make sure all action is generated by the student, not the teacher. In the end, they use this token level distillation to supervise student and also use turn-level teacher PPL to guide trajectory collection. Finally they test this using Qwen models on ALFworld, Science World and WebShop. Also showing a impressive improvement compare to other methods

Critical strengths

This paper proposes a simple and compelling idea. They use a teacher to guide student choose the next action before action execution. A teacher guided pipeline for selecting student-generated actions. After the student has been trained through this method, they can apply these independently in real world. It has practical value for deployment. They ablation experiments support it will. In table 4, random commit suggesting that additional sampling alone is insufficient to explain the improvement. Student-PPL reflect the value of teacher's scoring signal. All-scored prove that supervising every candidate is not benficial. The comparison with trajectory-PPL supports selecting responses at each turn before execution, rather than filtering completed trajectories afterward. Table 5 further shows that increasing the candidate count does not improve performance. Beyond these ablations, Table 1 reports improvements across ALFWorld, ScienceWorld, and WebShop. Table 2 suggests that a smaller teacher also helps. Table 3 demonstrates additional improvement under task-adapted initialization with an SFT-trained student and an SFT + GRPO trained teacher. Together, these experiments provide convincing support for the practical effectiveness of the proposed strategy across the evaluated teacher sizes and initialization settings.

Critical weaknesses

The comparison does not show that PREACT's performance advantage under equal computational budgets. PREACT takes 1.21-2.22 times longer than vanilla OPD to train with the same number of updates (Table 10 Qwen3-8B). However, the paper does not report results where each method is given the same compute. Such as GPU hours. Random-Commit does not address this. It discards the extra candidates, so its training data follows the same distribution as vanilla OPD. As a result, there is no way to determine if PREACT's claimed advantage would continue if the other methods were able to use their extra computational budget to continue training.

Additionally, the paper cited Guided-OPD, ReOPD, and SAGE-OPD. TCOD-B2F but does not compare against them. It also evaluates only the F2B variant of TCOD and omits TCOD-B2F. Thus it is difficult to evaluate PREACT's contribution as compared to very similar methods.

Questions for authors
  1. How does PREACT compare to Guided-OPD, ReOPD, and SAGE-OPD that are mentioned in the introduction and related works but not evaluated in the experiment? Can you compare with at least one of these approaches or explain why they were excluded?

  2. Why was TCOD-B2F excluded from the evaluation? Could the author also compare TCOD-B2F and PREACT using the same teacher-student setup as well? It would help clarify PREACT’s contribution relative to this closely related approach.

  3. When PREACT is given the same total amount of GPU-hours for training as Vanilla OPD and TCOD-F2B, rather than just the same total number of updates, does PREACT still outperform Vanilla OPD and TCOD-F2B? Additionally, could you provide the training time of TCOD-F2B? The training time of TCOD-F2B is missing from Table 10.

Reviewer_Afz5OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PREACT, a method that, instead of sampling the most likely action from the student policy, selects the action with the lowest teacher perplexity at each state. This is demonstrably a better technique and provides significant improvements over vanilla on-policy distillation.

Critical strengths
  1. The paper is well presented, clear, and easy to follow. The technique proposed is well explained and shows clear empirical strengths.
  2. The ablation experiments demonstrate that the improved performance comes from the teacher selection rather than random selection or student selection
Critical weaknesses
  1. (not critical, but somewhat confusing) The presentation of results in Table 1 is a bit confusing; why report only Qwen3-32B distillation into Qwen3-1.7B, Qwen3-4B rather than averaging over all three distillation scenarios? Also, it is unclear why error bars are reported between the proposed method, PREACT, and vanilla OPD, but not for TCOD-F2B. It would be good to put a citation next to TCOD-F2B in the table since that is the first time that method appears in the text.
Questions for authors

Does this method scale well with the number of trajectories and the size of the model? How does the compute budget compare to that of standard OPD?

Comments on the paper

1 comment

Thanks for your response. Many of my concerns have been addressed. I will keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.