acceptodds
Withdrawn from ICLR 2027

Language as Computation: Simulated Feedback and On-Policy Guidance for Interactive Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study the point at which language becomes not only a medium for supervising computation, but a medium for carrying out interaction. For interactive agents, language can serve not only as the environment interface but also as the medium of the learning interface. Because environmental signals and learning feedback are both expressed as text, they can be written directly into the interaction history and participate in subsequent action generation, without requiring access to teacher logits. We instantiate this idea as a language-mediated on-policy learning framework in which a fixed teacher produces action-conditioned environmental feedback, progress signals, and gradually fading hints for training a student policy with GRPO. We evaluate both the fidelity of the simulated feedback and the transfer of the resulting policy to executable environments. Across 5,868 replayed transitions, DeepSeek-V3.2 achieves 75.3% normalized observation exact match, compared with 67.9% for Qwen3-32B, while simulation fidelity decreases with interaction depth. A Qwen3-8B student trained through these interactions reaches 44.13% on BFCL Multi-Turn and 46.88% on Tau-Bench, exceeding matched executable-environment RL by 2.25 and 2.07 points, respectively; without hints, simulated-feedback training remains competitive on BFCL. The same framework transfers to GUI agents on WebArena, matching executable-environment training in three categories and improving CMS.These results suggest that language can unify environmental signals and learning feedback within the policy's computational context, where both directly shape subsequent actions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Text Interface

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uES6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Language-Mediated On-Policy Guidance (LM-OPG), a training framework for interactive agents in which a fixed teacher LLM writes action-conditioned observations, progress signals, and annealed hints directly into the student’s interaction transcript during on-policy rollouts. The key idea is to use language as a unified medium for both environment feedback and learning supervision, thereby letting teacher guidance shape the student’s next actions within the same trajectory without exposing teacher logits. Empirically, the authors analyze simulation fidelity on 5,868 replayed transitions and report downstream gains on BFCL Multi-Turn, Tau-Bench, and a GUI-agent setting transferring to WebArena; ablations dissect the role of hints, progress rewards, and observation fidelity.

Critical strengths

The paper addresses a practical problem: training interactive agents often requires task-specific environments that are expensive to implement and run. The use of a teacher LLM to simulate environments and generate action-conditioned feedback is a promising approach.

Critical weaknesses
  1. The need for the proposed method is not clear enough. The paper's core premise (e.g., leveraging expensive teacher-guided RL to update policy weights for interactive agents) fails to demonstrate its necessity over modern, lightweight alternatives. Recent agent literature increasingly favors non-parametric, training-free paradigms such as Skill Evolution, dynamic memory refinement, and self-reflection, which achieve flexible tool and GUI adaptation without altering model parameters or running costly RL loops. Given the immense token and computational overhead of LM-OPG, the authors omit critical discussions and baseline comparisons against these training-free approaches. Without benchmarking against context-level evolution, it remains unconvincing whether heavy parametric fine-tuning offers genuine advantages over far lighter skill-level adaptation.

  2. The efficiency and scalability claims are not convincing enough. The authors position the framework as a "Docker-free" solution that eliminates environment execution bottlenecks; however, this execution cost is merely shifted to an exorbitant online teacher LLM inference load of 11M–21M tokens per GRPO step. Across a complete RL training lifecycle, the total API expenditure becomes prohibitive, compounded by severe network latency. In contrast, deploying containerized sandboxes on local clusters provides a far more cost-effective and substantially faster wall-clock execution alternative. Ultimately, the paper merely substitutes infrastructure overhead with exorbitant API bills and network latency, failing to genuinely resolve the scaling bottleneck in agent training.

Questions for authors
  1. Is the performance dependent on the teacher's capability? Results hinge on a strong teacher (DeepSeek-V3.2), but it is unclear how performance and fidelity degrade for weaker teachers.

  2. See Weaknesses.

Reviewer_qHPMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Language-Mediated On-Policy Guidance (LM-OPG) for training LLM agents with GRPO. For each student action, a teacher produces a simulated observation and a natural-language hint, both written into the student’s transcript, plus a private progress signal used to construct the RL reward. Hint strength and inclusion probability are annealed during training, so the student is deployed without hints against real environments. The authors argue that because environment feedback and teacher supervision are both text, teacher knowledge can directly shape the student’s ongoing trajectory, rather than entering only through a distillation loss that requires teacher logits.

A Qwen3-8B student trained on EnvScaler tasks reaches 44.13% on BFCL Multi-Turn, compared with 41.88% for matched real-environment RL and 40.88% for OPD. It reaches 46.44% on τ-bench, compared with 44.81% for real-environment RL. A smaller study trains on WebShop/ALFWorld and evaluates on WebArena.

Critical strengths
  1. The framing is clear, including the contrast with logit-based on-policy distillation.

  2. The design is simple and intuitive. It requires only black-box teacher access, and needs no shared tokenizer or teacher logits.

Critical weaknesses

My main critique is that the the authors do not sufficiently engage with closely related prior work. The central contribution, a stronger teacher injecting natural-language hints into the student’s context during on-policy RL rollouts and removing them before deployment, has substantial precedent that the paper does not cite or compare against. Here is are two works, but there are many others out there:

Guide-GRPO (Nath et al., 2025) is the closest. It adds hints to the model’s context for problems where all rollouts initially failed, and adjusts the importance sampling ratio of those trajectories so the policy is optimized for the hint-free context. Its hints are generated by a stronger model. This is essentially LM-OPG’s hint method, plus a correction for the train/deploy context mismatch that LM-OPG lacks.

Off-Context GRPO (2026) formalizes exactly this issue. It treats rollouts from a guidance-augmented prompt as “off-context” relative to the unguided target objective, and applies an importance-corrected objective to avoid the instability that uncorrected guided training causes. It also surveys a family of methods that inject privileged guidance into the prompt while the model samples its own rollouts.

What remains novel is the specific combination: one black-box teacher that simultaneously simulates the environment, judges progress, and writes per-turn hints in multi-turn tool use. That combination is a reasonable engineering contribution, but the paper should position it against the work above. It should also compare to the methods above and other related methods. Citing and comparing to prior work is a minimum prerequisite to getting any paper accepted for publication at any venue.

The reported differences are potentially within evaluation noise, and almost no uncertainty is reported. The paper reports a single evaluation run per benchmark, with no confidence intervals and no multiple training seeds. There are two exceptions. Figure 5 has error bars, but they are never explained, so I don’t know what they represent. The τ-bench baselines are averaged over 3 evaluation trials, but those numbers are copied from Song et al. (2026) rather than reproduced. The margins over the strongest baselines are small, which leads me to think the reported differences are within noise.

Separately, the τ-bench comparison is not valid as presented. The baselines are 3-trial means using a GPT-4.1 user simulator. The LM-OPG result is a single evaluation whose user simulator, decoding settings, and task/trial counts were not retained (per the Table 2 caption). Comparing methods across different, and partly unknown, evaluation protocols is not acceptable, and copying baseline numbers is only appropriate when the protocol is matched exactly. In its current form, this evaluation is insufficient to support the paper’s central claims.

Additionally, the hint annealing that the method depends on does not appear to have been carried out in the reported runs. The paper states that hints are “annealed to zero” before deployment, so that the student learns to act without them. However, Table 3 gives a tool-use inclusion horizon of K_p = 2000 updates, and Figure 2 shows 200 GRPO steps. Under Eq. 7, that implies p_k = 0.9 and hint strength of about 3/5 at step 200. Unless the evaluated tool-use checkpoint was trained well past step 2000, which is never stated, the student was trained with hints present on roughly 90% or more of turns and evaluated with none. That contradicts the method as described

Finally, the abstract makes claims that the body does not support. These include:

  1. It reports a τ-bench improvement over real-environment RL of 2.07 points. Table 2 and Section 4 give 1.63, and per Weakness 2 the comparison isn’t valid.
  2. It compares DeepSeek-V3.2 against Qwen3-32B (67.9% exact match) as a simulator, but no Qwen3-32B result appears anywhere in the paper.
  3. It states that training without hints “remains competitive on BFCL.” The no-hint variant scores 40.25%, below both real-environment RL (41.88%) and OPD (40.88%). Again though, there are no confidence intervals so this claim is hard to evaluate.
Questions for authors

How do you position your work relative to the extensive related prior work, and how does your method compare to those methods?

Can you evaluate your method across at least 3 seeds?

Reviewer_zSsJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new approach to learning task-completion with LLMs, Language-Mediated On-Policy Guidance (LM-OPG). Rather than deploying an LLM to a task environment, the task specification and requirements are given to a frontier LLM, which then simulates the environment interaction for a student LLM. The frontier LLM optionally also provides hints to guide the student towards a solution. LM-OPG is demonstrated on BFCL Multi-Turn and tau-bench, where it appears to perform better than RL on the real environment (without hints), compared to SFT on expert traces, or compared to on-policy distillation with a stronger model. This means that future task-specific training would not need to rely on environment setups, but could instead simply call large LLMs to simulate the task in question.

Critical strengths
  • LM-OPG is evaluated on two domains, showing that it works both in its training distribution (BFCL) and generalizes to new domains (tau-bench). The method is further tested on GUI interaction tasks, such as WebArena, where it appears to learn and perform on par with text-based environments.
  • LM-OPG enables learning without needing to set up dedicated training environments or sandboxes, as the entire loop is self-contained within a student-teacher loop. This setup could greatly reduce the complexity needed for training and deploying LLMs to new tasks, and would enable training on tasks that currently lack a dedicated environment or simulator.
Critical weaknesses
  • LM-OPG is not really a new idea. Providing hints to a student LLM during RL is a well-established practice at this point [1,2,3,4], and there is virtually no performance benefit to running LM-OPG over running RL + hints (Figure 5).
  • The paper has clear signals of being written by an LLM that was pointed to a code repo, and it suffers from serious readability problems. For example:
    • The results section alternates between reporting relative differences (+3% gain) and absolute scores (35%), requiring the reader to constantly jump back and forth from the text to the tables. This makes actually following the text quite confusing.
    • The text also makes claims that are directly contradicted by the figures, for example: "KL, entropy, length, and turns remain stable" (Line 262), while Figure 2 shows KL has grown 4x and is continuing to climb.
    • Figures alternate between using "OPG" and "LM-OPG"
    • Line 263: "The RL trained student exceeds the real-env-RL student" -- Which RL trained student? Presumably LM-OPG?
    • Table 2's caption includes the line "50.88 is not a multiple of 1/115" -- yet LM-OPG achieves 50.88 on a 115 task split, so how is this result possible? All of Table 2's caption is quite confusing and overly verbose.
    • The motivation and abstract are very difficult to understand and ultimately say very little about the actual method, instead making ungrounded claims about text as a "medium for interaction".
  • Numerous variables are undefined or defined too late. For example, kk and KK are both not defined in the main text, but are important to understanding equation 6. ntn_t is introduced on line 145 but not defined as a hint until line 157.
  • ht+1h_{t+1} (the transcript for the next state) is defined twice, once with a simulated observation \hat{ot+1o_{t+1}}, and once with a real observation ot+1o_{t+1}. No distinction is made between these two different definitions, and they clearly have very different implications for the paper. I believe LM-OPG is built on the assumption that all observations are simulated, but the actual method section is unclear on this.
  • Equation 6 imposes a max (0, 1-k/K) constraint, but if k is the learning episode out of K possible episodes, then surely the max is not necessary? K is not defined in the text, so it is hard to verify this, but this equation seems entirely unnecessary.
  • Appendix B claims that strength is an integer on the teacher's 1-5 scale, but the hint max/min strengths are set to 4 and 1, respectively (not 5 and 1).

[1] Jain, Vaibhav, and Gerrit Grossmann. "Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations." arXiv preprint arXiv:2510.08779 (2025).

[2] Wang, Zongsheng, et al. "VeriRole: Verifiable Role-Awareness through Hint-Guided Reinforcement Learning." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Cao, Zhiyu, et al. "Hint-Guided Diversified Policy Optimization for LLM Reasoning." arXiv preprint arXiv:2606.03021 (2026).

[4] Yu, Linhao, et al. "Knowrl: Boosting llm reasoning via reinforcement learning with minimal-sufficient knowledge guidance." arXiv preprint arXiv:2604.12627 (2026).

Questions for authors
  • What is cosine hint? This is in the results but is never introduced? The text says cosine-schedule, so I am guessing this is just a different decay rate (as opposed to linear). It is not clear why this should perform so much worse -- it is worse than no hints at all?
  • How does LM-OPG apply to GUI interactions? The paper focuses heavily on text as a means of expressing state or hints, yet GUI agents would likely require image input.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.