acceptodds
Under review as a conference paper at ICLR 2027

EAU: Environment-Agent-User Co-Evolution for Training User-Centric Agents

Abstract

Training LLM agents to interact with users over multiple turns is challenging because the learning signal depends not only on the agent, but also on the user simulator and environment that generate its experience. Simulated users can be stochastic or inconsistent, while environments may contain unreliable tools, incomplete state coverage, and incorrect verifiers. Prior work mainly addresses the resulting instability through reward shaping or denser supervision, while treating these components as fixed. We instead introduce \method, a meta-agent framework for environment–agent–user co-evolution. During training, \method monitors trajectories, tool traces, rewards, and training statistics; diagnoses failures and uninformative interactions; and proposes targeted modifications to the user simulator, environment, reward design, and RL configuration. Validated changes are hot-swapped into the ongoing run, allowing the training setup to evolve together with the policy. These adaptations also persist across runs, so later training can build on previously discovered improvements rather than restarting from a fixed setup. We enable this process through an editable symbolic representation of user and environment dynamics, while the agent remains a neural LLM optimized with RL. Across three interactive-agent benchmarks and two backbone models, \method consistently outperforms prior RL methods, improving the overall score over the strongest baseline by 13.1–15.0% and increasing successful task completion on ColBench by up to 36.7%, showing that co-evolving the training system with the policy can substantially improve multi-turn RL.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
User Centric Agentic Reinforcement Learning, Meta-Agent, LLM Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RfTUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper first identifies possible failure from fixed simulators and environments in user-centric agentic RL, and proposes to overcome these issues by EAU, a meta-agent framework for environment–agent–user co-evolution. During training, EAU monitors trajectories, tool traces, rewards, and training statistics; diagnoses failures and uninformative interactions; and proposes targeted modifications to the user simulator, environment, reward design, and RL configuration. Such modification is made possible through an editable symbolic representation of user and environment dynamics, while the agent remains a neural LLM optimized with RL. Across three interactive-agent benchmarks and two backbone models, EAU consistently outperforms prior RL methods.

Critical strengths

To the best of my knowledge, this paper is the first to attack fixed simulators and environments in user-centric agentic RL. Prior user-centric methods improve the supervision signal presented to the policy but leave the simulator and environment untouched, and the closest prior effort co-trains the user simulator only (Yu et al., 2026; Sec. 2). By contrast, EAU identifies that the failure modes can lie in all three parts (RL training dynamics, user simulation, and the environment) and makes all of them modifiable online by a single meta-agent.

Critical weaknesses
  1. Empirical comparison may be unfair as the cost of the meta-agent is not reported; it is implemented by Claude Opus-5 reading logs and editing code throughout training. Sec. 5.2 matches the per-run compute budget of InfoPO (Kong et al., 2026) but allows EAU up to five inter-run iterations "including the active restart of experiments", and Figure 6's own annotations (3050 vs. 1020 steps; "heights not comparable") indicate that total consumed steps vary substantially across settings. In addition, all baseline numbers except PPO are copied from InfoPO rather than re-run (Sec. 5.1), which means that the improvement compares policies trained under different total budgets.

  2. Some key components of the proposed method appear insufficiently supported. For example, the symbolic user simulator, presented as a core contribution (Sec. 4.2, Conclusion), is only marginally positive in the only ablation isolating it: Hybrid 0.294 Pass / 0.213 Succ vs. LLM-only 0.284 / 0.200 on ColBench (Table 3). In addition, there are no ablation studies of the LLM-authored intermediate rewards (e.g., the first_tool 0.02 / first_uid 0.07 / first_oid 0.09 terms of Fig. 5), nor a hyperparameter sensitivity analysis.

Questions for authors
  1. (Clarification) Does Table 1 compare systems matched on total training steps consumed (including EAU's restarts) or only on the per-run budget? Please report total environment steps and meta-agent API cost per setting, addressing the 3050 vs. 1020 steps discrepancy visible in Fig. 6.

  2. (Clarification) Table 1's subscripts are standard deviations over 3 repeated evaluations; how stable is the final score across independent EAU run chains (fresh meta-agent decisions and user-simulator seeds)? Could a second run chain materially change the margin over InfoPO (Kong et al., 2026)?

  3. (Clarification) What are the results with only the environment/reward channel active (symbolic user and agent-knob tuning disabled), and with only the user channel active? This is the ablation that would distinguish co-evolution from automated reward shaping plus debugging.

  4. (Clarification) What exactly does the "bounded integrity gate" check before hot-swapping a modification (Sec. 4.1)? Table 6 mentions a 10-step verification window that the entropy-coefficient effect's lag exceeded — what fraction of proposals are rejected, and how much training damage occurs when a bad modification passes the gate (e.g., the reverted facts-per-reply 1 to 3 change)?

  5. (Clarification) Were UserRL/InfoPO trained under your implementation on the identical pre-EAU training environments and CoSER simulator, or is the Table 1 comparison additionally confounded by simulator or reward differences relative to InfoPO?

  6. (Suggestion) Since results depend on the meta-agent backbone (Table 5: 18.8 with Codex vs. 21.5 with Claude Opus-5 on τ2\tau^2-bench), an experiment with an open-source coding agent would help bound reproducibility.

2 comments
AuthorsOpenReview ↗

Response to Reviewer RfTU (1/2)

Q1: (Clarification) Does Table 1 compare systems matched on total training steps consumed (including EAU's restarts) or only on the per-run budget? Please report total environment steps and meta-agent API cost per setting, addressing the 3050 vs. 1020 steps discrepancy visible in Fig. 6.

A: It’s the per-run budget that we compare in Table 1. We use a slightly different audit frequency for different benchmarks, and the meta-agent API cost is around 0.66$/audit. The environment steps per run is the same. As for 3050 vs. 1020, it’s a number that we tune in our early experiment. Roughly we ask the meta-agent to audit the training every 20 steps

Q2: (Clarification) Table 1's subscripts are standard deviations over 3 repeated evaluations; how stable is the final score across independent EAU run chains (fresh meta-agent decisions and user-simulator seeds)? Could a second run chain materially change the margin over InfoPO (Kong et al., 2026)?

A: We conduct another experiment on ColBench using GPT-6.1-Sol (High, Codex). It achieves a similar score as Opus 5 (Medium, Claude Code) And more interestingly, they achieve this in a different way.

RunPass: mean ± stdSuccess: mean ± std
19.60 ± 0.297.00 ± 0.82
221.83 ± 0.7812.33 ± 0.47
344.18 ± 2.8438.51 ± 2.11
445.40 ± 1.3535.67 ± 1.25
568.63 ± 0.3160.33 ± 0.47

Q3: (Clarification) What are the results with only the environment/reward channel active (symbolic user and agent-knob tuning disabled), and with only the user channel active? This is the ablation that would distinguish co-evolution from automated reward shaping plus debugging.

A: We agree that isolating each module is necessary. Below, we show the ablation of each module across 5 runs. For better comparison, we use the 0.2/0.7/0.9 intermediate reward and a pure symbolic user sim with warmup as the initial setting.

AblationRoundRetailAirlineTelecomOverall
Initial-17.5010.009.3812.75
User-simR111.8827.5010.0014.25
User-simR223.1318.759.3816.75
User-simR321.2517.504.3813.75
User-simR420.6315.0011.2515.75
User-simR520.0020.0013.7517.50
Env/rewardR118.7517.509.3814.75
Env/rewardR216.8817.5010.0014.25
Env/rewardR316.2517.508.7513.50
Env/rewardR414.3822.5011.8815.00
Env/rewardR514.3820.007.5012.75
Agent/RLR112.5017.506.2511.00
Agent/RLR223.1315.0010.6316.50
Agent/RLR315.6321.257.5013.50
Agent/RLR413.7513.758.1311.50
Agent/RLR516.8822.5015.0017.25

As shown in the table, by updating the user simulator and modifying agent RL hyper-params, both of the experiments achieve the highest performance in run 5. For reward ablation, instead, it’s hard to find a set of rewards that is much better than the current one. This also pictures a co-evolution of the three parts, i.e., user simulator makes sure the interactions are correct, reward lays a foundation for learning signals and RL hyper-params help achieve better performance by balancing exploration and exploitation.

AuthorsOpenReview ↗

Response to Reviewer RfTU (2/2)

Q4: (Clarification) What exactly does the "bounded integrity gate" check before hot-swapping a modification (Sec. 4.1)? Table 6 mentions a 10-step verification window that the entropy-coefficient effect's lag exceeded — what fraction of proposals are rejected, and how much training damage occurs when a bad modification passes the gate (e.g., the reverted facts-per-reply 1 to 3 change)?

A: The “bounded integrity gate” is a set of heuristic values proposed by the meta agent itself in the very beginning of a training. Only a small fraction of proposals are rejected (). As we observed, a bad modification would cause huge damage to a training. But we believe that the meta-agent needs to pay this “tax” to learn that the modification is bad. And from a hindsight, it is helpful for the meta-agent to discover an efficient training recipe.

Q5: (Clarification) Were UserRL/InfoPO trained under your implementation on the identical pre-EAU training environments and CoSER simulator, or is the Table 1 comparison additionally confounded by simulator or reward differences relative to InfoPO?

A: In Table 1, UserRL and InfoPO are trained in the InfoPO’s setting, where they directly use the eval user sim as the training one. In the early experiments, we observe that this can lead to better performance when the training and the eval user simulators are consistent. But we instead use CoSER for cheaper deployment costs. For tau2bench, we train all the methods under the same setting.

Q6: (Suggestion) Since results depend on the meta-agent backbone (Table 5: 18.8 with Codex vs. 21.5 with Claude Opus-5 on -bench), an experiment with an open-source coding agent would help bound reproducibility.

A: Thank you for the suggestions. We have prepared results on the open-source coding. As shown in the table below, with GPT-6.1-Sol xhigh as the backbone and using open-source harness, EAU can still achieve competitive performance.

MethodAirlineRetailTelecomOverall
GPT+Pi16.25%24.38%13.75%18.50%
GPT+mini-swe27.50%24.38%17.50%22.25%

However, we also found that EAU does require strong backbone models to be effective. We ran an EAU agent using RecursiveSI/flint, an open-source 40BA2.5B model developed by RecusiveSI, dedicated for autonomous research. We used pi as the harness. We found that flint+pi makes consistent coding errors (such as execution errors) and cannot run.

Reviewer_KHdSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EAU uses a coding meta-agent to inspect interactive RL training and modify the components that generate learning experience: user simulation, environment and reward behavior, and optimization settings. It combines symbolic and LLM-based users and supports both changes during a run and experience carried between runs. Evaluation uses frozen benchmark settings, with the meta-agent described as receiving training feedback rather than evaluation scores. Across UserGym, ColBench, and tau2-bench, the system improves reported task performance. Ablations examine simulator choices, a generic meta-agent, different underlying training methods and meta-agent backbones, and an inter-run versus intra-plus-inter-run comparison.

Critical strengths
  1. The paper addresses a practical source of failure that policy optimization alone cannot fix. Appendix A documents concrete environment and simulator problems, including a deadlock and information disclosure that effectively supplies a reference solution. This makes the motivation for inspecting and repairing the training setup convincing. The record of harmful or ineffective interventions is also useful: it shows that co-evolution involves diagnosis and rollback, rather than assuming every modification helps.
  2. The gains are substantial and supported by more than one comparison. On ColBench with Qwen3-4B, success rises from 0.439 for InfoPO to 0.600, and improvements also appear with the second policy backbone and on other task suites. The generic-meta-agent comparison supports the value of the proposed scaffold. Figure 8 is particularly relevant because it compares fifth-run configurations given the same first four runs of experience, with and without intra-run intervention. These results give the paper a credible contribution beyond a qualitative account of automated debugging.
Critical weaknesses

The main uncertainty is the total budget and run-selection protocol behind the headline comparison. Section 5.2 states that compute is matched to InfoPO but also allows up to five inter-run iterations, including early restarts; Table 1 imports most baseline numbers from the prior paper. It is unclear whether the matching covers the cumulative training across those iterations or each individual run. The reported standard deviations concern repeated evaluations, so they do not establish the variability of the full adaptive training procedure. Figure 8 provides useful evidence for intra-run adaptation, but does not resolve the total-budget comparison with the strongest baseline. I would not require a meta-agent to be free of overhead; I do need to know the amount of additional search and how the reported run is chosen to assess the size of the advance fairly.

Questions for authors
  1. Does the matched budget include all restarted and completed runs? Please report cumulative policy-training steps or rollout tokens for the main comparisons, with meta-agent calls listed separately, and explain how the final reported checkpoint/run is selected without evaluation feedback.
  2. How many independent end-to-end EAU training sequences underlie the headline results, as distinct from the three repeated evaluations? If only one sequence was run, is there existing variation across runs or checkpoints that helps assess whether the largest gain depends on one intervention history?
1 comment
AuthorsOpenReview ↗

Response to Reviewer KHdS

W: The main uncertainty is the total budget and run-selection protocol behind the headline comparison. I do need to know the amount of additional search and how the reported run is chosen to assess the size of the advance fairly.

A: Thank you for pointing this out. We allow our meta-agent to run five independent experiments, each matching an InfoPO run. The basic idea is letting the meta-agent to warm up or learn with four runs and then exploit the knowledge or experience in the last run. Therefore the reported run is the last checkpoint of the last run.

Q1: Does the matched budget include all restarted and completed runs? Please report cumulative policy-training steps or rollout tokens for the main comparisons, with meta-agent calls listed separately, and explain how the final reported checkpoint/run is selected without evaluation feedback.

A: The final reported checkpoint is simply the last checkpoint of the final run, including the restarted ones.

Q2: How many independent end-to-end EAU training sequences underlie the headline results, as distinct from the three repeated evaluations? If only one sequence was run, is there existing variation across runs or checkpoints that helps assess whether the largest gain depends on one intervention history?

A: Only 1. To address your concern, we ran another ColBench experiment using GPT-6.1-Sol (High, Codex), and it yields similar results as Opus 5 (Medium, Claude Code). Specifically, we analyze the history and find that they achieve the strong performance with a different strategy of modifications.

RunPass: mean ± stdSuccess: mean ± std
19.60 ± 0.297.00 ± 0.82
221.83 ± 0.7812.33 ± 0.47
344.18 ± 2.8438.51 ± 2.11
445.40 ± 1.3535.67 ± 1.25
568.63 ± 0.3160.33 ± 0.47
Reviewer_qhU9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RL training of user-centric agents, which must uncover hidden information through conversation with a simulated user and tool calls. It argues that fixed simulated users and environments are a key bottleneck, because they can make rewards highly variable or misleading. It proposes EAU, a meta-agent (Claude Code with Claude Opus-5) that reads the training logs and trajectories during PPO training and edits the training setup. Besides fixing code, the meta-agent tunes knobs such as the rules of a new symbolic (rule-based) user, how often it replaces the LLM user, the LLM user's temperature and persona, intermediate reward rules, and the entropy and KL coefficients. Changes take effect in the running job, the meta-agent may restart runs (at most five runs per experiment, including restarts), and its findings are kept in an experience pool that carries over to later runs. EAU raises the overall UserGym and ColBench score over InfoPO from 0.545 to 0.627 with Qwen3-4B and from 0.544 to 0.615 with Qwen2.5-7B-Instruct; its τ²-bench average is 21.5, against 18.5 for PPO, the strongest baseline there. Ablations compare user types, a naive meta-agent, EAU on top of UserRL and InfoPO, and a Codex meta-agent.

Critical strengths
  1. Moving the intervention upstream to the user and the environment is well motivated, since reward shaping cannot fix a tool that silently fails, a verifier that wrongly rejects, or a user who reveals what the agent should ask for (Sec. 1). Gated knob changes and an experience pool across runs make this practical.

  2. The gains are large: with both policy models, EAU has the best overall score in Table 1, above InfoPO and the closed-source models, and the best score on the held-out Inten. gym, and it also improves UserRL and InfoPO on τ²-bench (Table 5). Table 6 reports harms and reverts openly, which makes the meta-agent's decisions easy to check.

Critical weaknesses
  1. The novelty seems to be limited. Sec. 2 separates EAU from earlier meta-agents because it acts within a run, and from co-evolution methods because it does not follow predefined update rules. However, population-based training [1], MIRA [2] and MUST [3] already change hyperparameters, rewards (with an LLM deciding the edits in MIRA) or the mix of rule-based and learned users during training. Since Sec. 4.1 presents EAU as a replacement for manual expert intervention, it is unclear what it adds beyond automating that tuning.

  2. Part of the gain seems to come from the strength of the closed meta-agent rather than from co-evolution. All main results use Claude Code with Claude Opus-5, although Sec. 5.4 calls EAU agnostic to the meta-agent backbone. The only other backbone, Codex with GPT-5.6-Sol, is tested on τ²-bench alone and reaches 18.8, barely above InfoPO (18.0) and PPO (18.5) without a meta-agent, against 21.0 and 21.5 with Claude Code (Tables 2 and 5). Because the meta-agent revises the user rules and can add reward rules, it may also pass its own task knowledge to the policy, and Table 4 shows the value of EAU's design only with the same backbone. On ColBench, of the 15 knob changes, the only one with a confirmed effect lowered Pass and was reverted (Table 6). Ablations with a smaller or open-source meta-agent, another harness and different policy sizes would show how much of the gain depends on the backbone. Since the meta-agent is a closed model and no code is linked, the results are also hard to replicate.

  3. Other methods built for user interaction are not compared, and compute is not reported. Apart from UserRL and InfoPO, the baselines are prompting methods or RL methods that do not target user interaction, while SWEET-RL [4], which introduced ColBench, and MUA-RL [5] and CM2 [6], which report τ²-bench results, are missing. Although Sec. 5.2 says the compute budget matches the InfoPO paper, the paper gives only training settings (Table 8), not GPU hours, total steps including restarts, or the meta-agent's API cost, and the efficiency claim of Secs. 4.5 and 5.4 is shown only per run (Figs. 7–8). A comparison at equal compute, for example against the best of five PPO or InfoPO runs, would clarify the benefit.

References

[1] Jaderberg et al., Population Based Training of Neural Networks, arXiv:1711.09846

[2] Zhang et al., MIRA: An LLM-Driven Dual-Loop Architecture for Metacognitive Reward Design, Systems 2025, doi:10.3390/systems13121124

[3] Liu et al., One Cannot Stand for Everyone! Leveraging Multiple User Simulators to train Task-oriented Dialogue Systems, ACL 2023, doi:10.18653/v1/2023.acl-long.1

[4] Zhou et al., SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks, arXiv:2503.15478

[5] Zhao et al., MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use, arXiv:2508.18669

[6] Zhang et al., CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Use, arXiv:2602.12268

Questions for authors
  1. Does Table 1 report the last or the best-scoring run of each setting? Fig. 7 tracks a best-score frontier over runs. Also, how is the meta-agent, which works in a filesystem with the source code, kept away from the evaluation data?
  2. How is the meta-agent's context managed over a long run? The ColBench runs in Table 6 have 624 steps each, and an experiment can include up to five runs. Does the meta-agent keep one session, or start a new one at each log check and rely on the experience pool?
2 comments
AuthorsOpenReview ↗

Response to Reviewer qhU9 (1/2)

W1: How does EAU differ from prior methods such as PBT [1], MIRA [2], and MUST [3], which also support modifying hyperparameters, rewards, or user simulators during training? What does EAU contribute beyond automating expert intervention?

A: Thank you for the question. We would like to clarify our key contributions and distinguish EAU from prior work.

We view automating expert intervention in RL training as an important contribution in itself. AutoML has demonstrated the value of automating traditionally human-driven decisions in machine learning. Inspired by this paradigm, we extend the AutoML idea to agentic, user-centric RL training, introducing EAU as a meta-agent that continuously supervises the training process. Importantly, EAU goes beyond conventional automated tuning in two ways: (1) allowing EAU to apply on-the-fly modifications and (2) taking control of all three components instead of simply tuning one of them.

EAU also differs from existing methods in the flexibility and generality of its interventions. While PBT, MIRA, and MUST also modify these components during training, they mostly follow human-designed algorithms. In contrast, we use a general-purpose meta-agent to autonomously diagnose training bottlenecks and propose targeted interventions. EAU can flexibly explore and combine different interventions instead of following human-designed heuristics along a particular axis. In addition, we emphasize that effective supervision of agentic RL training is a nontrivial capability. Our experiments show that simply equipping frontier models with state-of-the-art coding harnesses, such as Claude Code or Codex, does not yield comparable improvements. In contrast, EAU achieves stronger performance through our training supervision framework. This demonstrates that EAU contributes more than a straightforward application of existing coding agents to automated tuning.

Overall, our contribution goes beyond the ability to modify training configurations: we introduce a general-purpose meta-agent framework that flexibly composes interventions to supervise RL training and automate expert tuning.

W2: Ablations with a smaller or open-source meta-agent, another harness and different policy sizes would show how much of the gain depends on the backbone. Since the meta-agent is a closed model and no code is linked, the results are also hard to replicate.

A: Thank you for suggesting this experiment. We conduct the following experiments to address your concern.

In the first experiment, we run an EAU agent using RecursiveSI/flint, an open-source 40BA2.5B model developed by RecusiveSI, dedicated for autonomous research. We used pi as the harness. We found that flint+pi makes consistent coding errors (such as execution errors) and cannot run. The results suggest the EAU does require strong backbone models to be effective.

Later, we tried gpt-6.1-sol + pi and gpt-6.1-sol + mini-swe, and found that with open-source harness, a strong backbone model can still achieve competitive performance

MethodAirlineRetailTelecomOverall
GPT+Pi16.25%24.38%13.75%18.50%
GPT+mini-swe27.50%24.38%17.50%22.25%

To further address your concern, we ran another ColBench experiment using GPT-6.1-Sol (High, Codex), which yields similar results as Opus 5 (Medium, Claude Code) and we log the sessions and histories. We find that the approach to the strong result is different from the previous one found by Claude Code, which strengthens the reproducibility.

RunPass: mean ± stdSuccess: mean ± std
19.60 ± 0.297.00 ± 0.82
221.83 ± 0.7812.33 ± 0.47
344.18 ± 2.8438.51 ± 2.11
445.40 ± 1.3535.67 ± 1.25
568.63 ± 0.3160.33 ± 0.47
AuthorsOpenReview ↗

Response to Reviewer qhU9 (2/2)

W3.1: Other RL methods built for user interactions are not compared, such as SWEET-RL, MUA-RL, and CM2?

A3.1: We ran SWEET-RL on tau2bench, and it scores 7.5%, which is much lower than eau.

W3.2: The compute budget is not reported, such as total steps including restarts, or meta-agent’;s API cost.. Does EAU remain more efficient under an equal-compute comparison, e.g., against the best of five PPO or InfoPO runs?

A3.2: Thank you for the suggestion. In our experiments, we restrict the compute budget of each meta-agent run to match that of a single InfoPO run.To provide a more comprehensive comparison, we additionally run InfoPO with four different random seeds and report the best-of-five results. As shown in the table below, the

SeedRetailAirlineTelecomOverall
4224.4%10.0%15.6%18.00%
4320.00%11.25%8.13%13.50%
4413.13%26.25%11.25%15.00%
4511.88%23.75%10.00%13.50%
4614.38%18.75%13.13%14.75%

The API costs for meta agents is around 0.66 dollars per audit. We acknowledge that the meta-agent incurs additional API costs. However, we argue that these costs should be reported separately from RL training compute, as they rely on different computational resources and pricing models (e.g., coding agents are often available through subscription plans). We will incorporate these results and the additional cost analysis into the revised paper. Thank you for the valuable suggestion!

Q1.1: Does Table 1 report the last or the best-scoring run of each setting? Fig. 7 tracks a best-score frontier over runs.

A: Yes, all the results we report are from the last runs. Fig. 7 is a comparison of eau and the naive meta agent across runs.

Q1.2: Also, how is the meta-agent, which works in a filesystem with the source code, kept away from the evaluation data?

A: We use harbor to manage our experiments. In each run, eval data is isolated outside the sandbox so that the meta-agent can not see them.

Q2: How is the meta-agent's context managed over a long run? Does the meta-agent keep one session, or start a new one at each log check and rely on the experience pool?

A: A single session is kept across 5 runs given the large enough context window of the state-of-the-art coding agents. We acknowledge that in a long training run, the meta agent may consume much context when reading trajectories, programming, etc. Therefore, we only ask the meta agent to do so periodically, and design an experience pool (as a markdown file) to save the important information.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chair for handling our submission and the reviewers for their thoughtful reviews and constructive suggestions. We summarize the key strengths recognized by the reviewers and our additional experiments below for convenience.


Strengths identified by the reviewers.

  • An important problem overlooked by existing RL methods. KHdS recognizes that EAU "addresses a practical source of failure that policy optimization alone cannot fix." Similarly, qhU9 finds that "moving the intervention upstream to the user and the environment is well motivated," while RfTU highlights the novelty of addressing fixed simulators and environments in user-centric agentic RL.

  • Substantial and consistent empirical improvements. qhU9 notes that "the gains are large," while KHdS emphasizes that "the gains are substantial and supported by more than one comparison." The improvements span multiple benchmarks and policy backbones, with additional ablations supporting the effectiveness of our framework.

  • An effective framework beyond automated debugging. KHdS specifically recognizes that the generic-meta-agent comparison and intra-run adaptation experiments provide "a credible contribution beyond a qualitative account of automated debugging." RfTU also highlights EAU's ability to make all three components (environment, agent, and user) modifiable online through a unified meta-agent.

  • Transparent diagnosis and intervention. The reviewers appreciate our concrete case studies of simulator and environment failures. KHdS particularly highlights the documentation of harmful interventions and rollback, which demonstrates that EAU performs diagnosis and correction rather than assuming every modification is beneficial.

Concerns raised and how we addressed them.

We conducted several additional experiments and analyses to address the reviewers' concerns regarding reproducibility, compute fairness, component contributions, and novelty.

Concern (reviewer)What we didKey findings
Does EAU's improvement depend on a single intervention history? (KHdS, RfTU)New independent end-to-end ColBench experiment using GPT-6.1-Sol + Codex, compared with the original Opus-5 + Claude Code setup.The new run achieves 68.63% Pass and 60.33% Success, comparable to the original result. The two meta-agents achieve strong performance through different intervention histories.
Does EAU work with smaller or open-source coding agent (qhU9, RfTU)New experiments with open-source coding harnesses (Pi and mini-swe) and an open-source meta-agent (Flint).GPT-6.1-Sol + mini-swe achieves 22.25% on τ²-bench, demonstrating feasibility with an alternative harness. Flint + Pi encounters execution failures, revealing a limitation with weaker meta-agent backbones.
Is the comparison fair in terms of compute budget? (qhU9, KHdS, RfTU)Four additional InfoPO seeds, yielding five independent baseline runs; clarification of EAU's training budget and API overhead.Best-of-five InfoPO remains 18.0% on τ²-bench. We clarify that EAU matches the per-run, not necessarily cumulative, baseline training budget, and report approximately $0.66 API cost per meta-agent audit.
Which components of EAU contribute to the improvements? (RfTU)New ablations restricting interventions to User, Environment/Reward, or Agent/RL, respectively.User-only and Agent/RL-only adaptation reach 17.50% and 17.25%, respectively, from a 12.75% initial score. This also depicts a co-evolution of the three components.
Are relevant user-interaction RL baselines missing? (qhU9)Additional SWEET-RL evaluation on τ²-bench.SWEET-RL achieves 7.5%, below the reported EAU results.
How are training runs, checkpoints, and evaluation isolation handled? (qhU9, KHdS)Clarified the final-checkpoint selection protocol, context management, and separation between training and evaluation data.Results use the last checkpoint of the final run, rather than selecting the best evaluation score. Evaluation data remain inaccessible to the meta-agent through sandbox isolation.
What distinguishes EAU from prior adaptive training methods? (qhU9)Expanded discussion comparing EAU with PBT, MIRA, and MUST.EAU's contribution lies in general-purpose meta-agent and flexible cross-component intervention, rather than introducing individual hyperparameter or simulator modifications in isolation.

Overall, we believe these additional results strengthen the evidence for EAU's effectiveness, demonstrate that its gains are not limited to a single intervention history, and provide a more transparent account of its computational costs and limitations. We thank the reviewers for their valuable feedback, which has helped us substantially strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.