- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PreAct, a teacher-guided approach to multi-turn agent distillation. At each turn, the student generates several candidates, and a fixed teacher selects the response with the lowest teacher perplexity for execution and distillation. Experiments on ALFWorld, ScienceWorld, and WebShop show improvements over vanilla OPD across several model configurations.
Critical strengths
-
The proposed approach is reasonable and easy to follow. Selecting responses before execution provides a simple way to influence both the current training sample and subsequent interaction histories.
-
PreAct improves over vanilla OPD across multiple environments and model configurations, including task-adapted initialization. The gains are retained when the student acts independently without teacher selection.
-
The ablations examine relevant design choices, including teacher versus student ranking, candidate budget, and supervision over selected versus all candidates. These provide useful support for the proposed recipe.
Critical weaknesses
-
The baseline comparisons are limited. Closely related methods such as Guided-OPD and SAGE-OPD are discussed but not evaluated, making it difficult to assess PreAct’s advantage over existing improvements to agentic OPD. Competitive SFT or RL student baselines would also help contextualize the results.
-
PreAct takes approximately 1.2–2.2 times the training time of vanilla OPD, but the main comparisons use matched update counts rather than matched training budgets. Random-Commit discards the additional candidates, so it does not establish whether using the extra budget for more OPD training would achieve similar gains.
-
The ablations do not fully separate the benefit of selecting better supervision from that of improving future interaction histories. Comparing PreAct with completed-trajectory filtering changes both selection timing and granularity. The conclusion’s claim that the comparison varies only timing therefore seems stronger than the evidence supports.
-
Teacher perplexity is a plausible ranking signal, but its connection to action quality remains insufficiently examined. Direct ranking-quality measurements or an action-only scoring ablation would help support this choice.
Questions for authors
-
How does PreAct compare with Guided-OPD or SAGE-OPD under comparable settings? Does its advantage over vanilla OPD remain when OPD receives the same total training budget?
-
What happens if training supervises the teacher-preferred candidate at each history, but execution uses a randomly selected candidate? This would help isolate the benefit of using teacher selection to shape subsequent histories.
Thanks for your response. Many of my concerns have been addressed. I will keep my positive score.