- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper first identifies possible failure from fixed simulators and environments in user-centric agentic RL, and proposes to overcome these issues by EAU, a meta-agent framework for environment–agent–user co-evolution. During training, EAU monitors trajectories, tool traces, rewards, and training statistics; diagnoses failures and uninformative interactions; and proposes targeted modifications to the user simulator, environment, reward design, and RL configuration. Such modification is made possible through an editable symbolic representation of user and environment dynamics, while the agent remains a neural LLM optimized with RL. Across three interactive-agent benchmarks and two backbone models, EAU consistently outperforms prior RL methods.
Critical strengths
To the best of my knowledge, this paper is the first to attack fixed simulators and environments in user-centric agentic RL. Prior user-centric methods improve the supervision signal presented to the policy but leave the simulator and environment untouched, and the closest prior effort co-trains the user simulator only (Yu et al., 2026; Sec. 2). By contrast, EAU identifies that the failure modes can lie in all three parts (RL training dynamics, user simulation, and the environment) and makes all of them modifiable online by a single meta-agent.
Critical weaknesses
-
Empirical comparison may be unfair as the cost of the meta-agent is not reported; it is implemented by Claude Opus-5 reading logs and editing code throughout training. Sec. 5.2 matches the per-run compute budget of InfoPO (Kong et al., 2026) but allows EAU up to five inter-run iterations "including the active restart of experiments", and Figure 6's own annotations (3050 vs. 1020 steps; "heights not comparable") indicate that total consumed steps vary substantially across settings. In addition, all baseline numbers except PPO are copied from InfoPO rather than re-run (Sec. 5.1), which means that the improvement compares policies trained under different total budgets.
-
Some key components of the proposed method appear insufficiently supported. For example, the symbolic user simulator, presented as a core contribution (Sec. 4.2, Conclusion), is only marginally positive in the only ablation isolating it: Hybrid 0.294 Pass / 0.213 Succ vs. LLM-only 0.284 / 0.200 on ColBench (Table 3). In addition, there are no ablation studies of the LLM-authored intermediate rewards (e.g., the first_tool 0.02 / first_uid 0.07 / first_oid 0.09 terms of Fig. 5), nor a hyperparameter sensitivity analysis.
Questions for authors
-
(Clarification) Does Table 1 compare systems matched on total training steps consumed (including EAU's restarts) or only on the per-run budget? Please report total environment steps and meta-agent API cost per setting, addressing the 3050 vs. 1020 steps discrepancy visible in Fig. 6.
-
(Clarification) Table 1's subscripts are standard deviations over 3 repeated evaluations; how stable is the final score across independent EAU run chains (fresh meta-agent decisions and user-simulator seeds)? Could a second run chain materially change the margin over InfoPO (Kong et al., 2026)?
-
(Clarification) What are the results with only the environment/reward channel active (symbolic user and agent-knob tuning disabled), and with only the user channel active? This is the ablation that would distinguish co-evolution from automated reward shaping plus debugging.
-
(Clarification) What exactly does the "bounded integrity gate" check before hot-swapping a modification (Sec. 4.1)? Table 6 mentions a 10-step verification window that the entropy-coefficient effect's lag exceeded — what fraction of proposals are rejected, and how much training damage occurs when a bad modification passes the gate (e.g., the reverted facts-per-reply 1 to 3 change)?
-
(Clarification) Were UserRL/InfoPO trained under your implementation on the identical pre-EAU training environments and CoSER simulator, or is the Table 1 comparison additionally confounded by simulator or reward differences relative to InfoPO?
-
(Suggestion) Since results depend on the meta-agent backbone (Table 5: 18.8 with Codex vs. 21.5 with Claude Opus-5 on -bench), an experiment with an open-source coding agent would help bound reproducibility.
2 comments
Response to Reviewer RfTU (2/2)
Q4: (Clarification) What exactly does the "bounded integrity gate" check before hot-swapping a modification (Sec. 4.1)? Table 6 mentions a 10-step verification window that the entropy-coefficient effect's lag exceeded — what fraction of proposals are rejected, and how much training damage occurs when a bad modification passes the gate (e.g., the reverted facts-per-reply 1 to 3 change)?
A: The “bounded integrity gate” is a set of heuristic values proposed by the meta agent itself in the very beginning of a training. Only a small fraction of proposals are rejected (). As we observed, a bad modification would cause huge damage to a training. But we believe that the meta-agent needs to pay this “tax” to learn that the modification is bad. And from a hindsight, it is helpful for the meta-agent to discover an efficient training recipe.
Q5: (Clarification) Were UserRL/InfoPO trained under your implementation on the identical pre-EAU training environments and CoSER simulator, or is the Table 1 comparison additionally confounded by simulator or reward differences relative to InfoPO?
A: In Table 1, UserRL and InfoPO are trained in the InfoPO’s setting, where they directly use the eval user sim as the training one. In the early experiments, we observe that this can lead to better performance when the training and the eval user simulators are consistent. But we instead use CoSER for cheaper deployment costs. For tau2bench, we train all the methods under the same setting.
Q6: (Suggestion) Since results depend on the meta-agent backbone (Table 5: 18.8 with Codex vs. 21.5 with Claude Opus-5 on -bench), an experiment with an open-source coding agent would help bound reproducibility.
A: Thank you for the suggestions. We have prepared results on the open-source coding. As shown in the table below, with GPT-6.1-Sol xhigh as the backbone and using open-source harness, EAU can still achieve competitive performance.
| Method | Airline | Retail | Telecom | Overall | |
|---|---|---|---|---|---|
| GPT+Pi | 16.25% | 24.38% | 13.75% | 18.50% | |
| GPT+mini-swe | 27.50% | 24.38% | 17.50% | 22.25% |
However, we also found that EAU does require strong backbone models to be effective. We ran an EAU agent using RecursiveSI/flint, an open-source 40BA2.5B model developed by RecusiveSI, dedicated for autonomous research. We used pi as the harness. We found that flint+pi makes consistent coding errors (such as execution errors) and cannot run.
Response to Reviewer RfTU (1/2)
A: It’s the per-run budget that we compare in Table 1. We use a slightly different audit frequency for different benchmarks, and the meta-agent API cost is around 0.66$/audit. The environment steps per run is the same. As for 3050 vs. 1020, it’s a number that we tune in our early experiment. Roughly we ask the meta-agent to audit the training every 20 steps
A: We conduct another experiment on ColBench using GPT-6.1-Sol (High, Codex). It achieves a similar score as Opus 5 (Medium, Claude Code) And more interestingly, they achieve this in a different way.
A: We agree that isolating each module is necessary. Below, we show the ablation of each module across 5 runs. For better comparison, we use the 0.2/0.7/0.9 intermediate reward and a pure symbolic user sim with warmup as the initial setting.
As shown in the table, by updating the user simulator and modifying agent RL hyper-params, both of the experiments achieve the highest performance in run 5. For reward ablation, instead, it’s hard to find a set of rewards that is much better than the current one. This also pictures a co-evolution of the three parts, i.e., user simulator makes sure the interactions are correct, reward lays a foundation for learning signals and RL hyper-params help achieve better performance by balancing exploration and exploitation.