- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Language-Mediated On-Policy Guidance (LM-OPG), a training framework for interactive agents in which a fixed teacher LLM writes action-conditioned observations, progress signals, and annealed hints directly into the student’s interaction transcript during on-policy rollouts. The key idea is to use language as a unified medium for both environment feedback and learning supervision, thereby letting teacher guidance shape the student’s next actions within the same trajectory without exposing teacher logits. Empirically, the authors analyze simulation fidelity on 5,868 replayed transitions and report downstream gains on BFCL Multi-Turn, Tau-Bench, and a GUI-agent setting transferring to WebArena; ablations dissect the role of hints, progress rewards, and observation fidelity.
Critical strengths
The paper addresses a practical problem: training interactive agents often requires task-specific environments that are expensive to implement and run. The use of a teacher LLM to simulate environments and generate action-conditioned feedback is a promising approach.
Critical weaknesses
-
The need for the proposed method is not clear enough. The paper's core premise (e.g., leveraging expensive teacher-guided RL to update policy weights for interactive agents) fails to demonstrate its necessity over modern, lightweight alternatives. Recent agent literature increasingly favors non-parametric, training-free paradigms such as Skill Evolution, dynamic memory refinement, and self-reflection, which achieve flexible tool and GUI adaptation without altering model parameters or running costly RL loops. Given the immense token and computational overhead of LM-OPG, the authors omit critical discussions and baseline comparisons against these training-free approaches. Without benchmarking against context-level evolution, it remains unconvincing whether heavy parametric fine-tuning offers genuine advantages over far lighter skill-level adaptation.
-
The efficiency and scalability claims are not convincing enough. The authors position the framework as a "Docker-free" solution that eliminates environment execution bottlenecks; however, this execution cost is merely shifted to an exorbitant online teacher LLM inference load of 11M–21M tokens per GRPO step. Across a complete RL training lifecycle, the total API expenditure becomes prohibitive, compounded by severe network latency. In contrast, deploying containerized sandboxes on local clusters provides a far more cost-effective and substantially faster wall-clock execution alternative. Ultimately, the paper merely substitutes infrastructure overhead with exorbitant API bills and network latency, failing to genuinely resolve the scaling bottleneck in agent training.
Questions for authors
-
Is the performance dependent on the teacher's capability? Results hinge on a strong teacher (DeepSeek-V3.2), but it is unclear how performance and fidelity degrade for weaker teachers.
-
See Weaknesses.