acceptodds
Under review as a conference paper at ICLR 2027

EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

Abstract

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision-making. Across BFCL-v4, τ²-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is available at [https://anonymous.4open.science/r/envace-anon-262A](https://anonymous.4open.science/r/envace-anon-262A).

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, LLM Agents, Tool Use.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4vFnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EnvACE, a reinforcement learning method for tool-using LLM agents in which a shared policy alternates between generating actions and simulating the resulting environment responses. Both roles are optimized using trajectory-level task rewards with role-wise GRPO, allowing training rollouts to proceed without external environment calls. At inference time, the model can rehearse interactions privately and summarize them into a memory that guides execution in the actual environment. The method is evaluated on four tool-use benchmarks, with additional analyses of parameter sharing, model scale, and test-time rehearsal.

Critical strengths
  1. The central design is simple and addresses a practical bottleneck in agent training. Rather than maintaining a separate simulator or using response prediction only as an auxiliary objective, EnvACE incorporates response generation directly into the policy’s rollout and optimization. Sharing parameters between acting and rehearsal provides a concrete mechanism for environment-modeling experience to influence decision making, without requiring an executable environment for each training interaction.
  2. The empirical results support the usefulness of this design beyond a single benchmark. At the 8B scale, EnvACE improves τ²-Bench performance from 31.2% with standard GRPO to 36.7%, and outperforms the separate-policy variant at 35.5% (Figure 3). The test-time comparison is also informative: with the acting policy held fixed, parallel rehearsal using EnvACE performs better than rehearsal using the base model on both benchmarks in Table 3. These results support the value of learned rehearsal rather than simply adding more inference steps.
Critical weaknesses
  1. The evidence for accurately internalizing environment dynamics is less direct than the evidence for improved task performance. The parameter-sharing ablation and case studies are suggestive, but the paper does not quantitatively evaluate response fidelity or consistency across state-changing interactions. Since generated responses receive the same task reward as actions, it remains unclear how the training signal distinguishes faithful simulation from responses that merely make the imagined task easier to complete. This does not invalidate the downstream gains, but it limits the interpretation of what the internalized world model has learned.
  2. The claimed scalability advantage is not quantified in computational terms. Avoiding executable environments is valuable, but EnvACE also generates environment responses and uses an LLM judge; test-time rehearsal adds simulation, evaluation, and summarization. Section 5.1 reports the training configuration, but not comparative token or wall-clock costs. The results therefore establish effectiveness more clearly than the overall resource advantage over conventional agent training.
Questions for authors
  1. What information or checks ground the training reward when tool responses are generated by the policy itself? Appendix D mentions ground-truth reference calls for test-time simulation; please clarify what reference information is available during training and whether you have measurements of response fidelity or state consistency that help distinguish accurate simulation from task-favorable fabrication.
  2. Can you report generated-token counts and wall-clock costs for EnvACE versus the standard GRPO baseline, including reward judging, together with the inference overhead of the two-attempt rehearsal setting? This would help assess the practical tradeoff behind the scalability claim.
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer 4vFn [Part 1/2]

We are grateful for your time and helpful suggestions on our work. All corresponding updates have been highlighted in blue in the revised manuscript.

W1 & Q1. What Information Grounds the Reward During Training, and Fidelity Evaluation

  • Training data: As described in §5.1, we train on the dataset introduced by CM2, where each task includes candidate tool schemas, a reference dialogue containing reference tool calls and their returns, and a checklist.
  • Rehearsal inputs and protocol: The ground-truth reference calls in Appendix D are taken from the CM2 training data: each training task comes with a reference dialogue, from which we extract the tool calls and their returns to serve as the reference for rehearsal on that task. During training, the inputs to rehearsal include the candidate tool whitelist, the extracted reference tool calls and returns above, the current conversation history, the current action, and the schema and arguments of the tool to be simulated. The protocol is the exact same one shown in Appendix D: rehearsal sequentially checks the call format, whether the tool is in the whitelist, and whether the arguments conform to the schema; if any check fails, it outputs a structured error and the trajectory terminates immediately; once passed, if the reference contains a call with the same name and similar arguments, it reuses that return, and otherwise generates a result consistent with the reference. Because all of these checks and reuses are carried out by the rehearsal role following the protocol, they are also optimized during RL.
  • Reward: We use a checklist-based LLM judge, Qwen3-30B-A3B, to assign trajectory-level rewards during training as detailed in §4.2 and §5.1. The checklists look beyond mere task completion: among the 115,145 checklist items in the training set, 18.6% directly inspect the generated tool returns, with at least one such item in 99.9% of tasks, and another 8,251 response checklist items require the assistant's response content to match the reference tool returns; the trajectory seen by the judge includes the tool returns generated by rehearsal. About 36% of the checklist items targeting tool returns require specific values to match the reference return, e.g., converted_value == 41.75 and unit == 'meters'. Consequently, rehearsal returns that deviate from the reference receive lower rewards.
  • Fidelity measurement: In revised Appendix E.3, we added an evaluation of response fidelity and state consistency against real environments. Using 6,649 tool calls obtained from running in the real τ²-Bench environments—which are unseen during training and cover airline, retail, and telecom across trajectories from EnvACE-8B, EnvScaler-8B, and Qwen3-8B—for each call, given the real interaction history and the action at that step, the model predicts the tool return under the rehearsal protocol and we compare it against the real return under greedy decoding; brackets below give 95% task-resampled bootstrap confidence intervals:
ModelSuccess/Error Acc.Error F1Field-level F1Read-after-Write Field F1
Qwen3-8B Base74.5 [72.7, 76.3]38.4 [33.0, 43.1]12.3 [11.5, 13.1]17.8 [15.1, 20.7]
EnvACE-8B85.5 [82.9, 87.7]64.4 [60.5, 68.0]16.4 [15.4, 17.6]25.3 [22.5, 28.4]

Read-after-Write Field F1 counts only read calls on objects that previously underwent a write operation. EnvACE outperforms the base model on every metric with non-overlapping confidence intervals, and breaking down the statistics by trajectory source yields consistent conclusions. Across the 1,748 genuinely failed calls, EnvACE predicts failure 49.1% of the time versus 29.6% for the base model; across the 4,901 genuinely successful calls, its false-failure rate is only 1.2% versus 8.8% for the base model. This demonstrates that EnvACE's rehearsal responses are closer to true environment transitions rather than biased toward task-favorable outcomes. In addition, the consistent advantage of EnvACE rehearsal over base rehearsal in Table 3, Figure 6, and Figure 9 indirectly indicates that the responses generated by EnvACE are more helpful for real execution.

AuthorsOpenReview ↗

Author Response to Reviewer 4vFn [Part 2/2]

W2 & Q2. Training and Inference Costs

We compiled the training costs in the table below and added them to revised Appendix E.4.

ScaleMethodACT tokens/stepREHEARSE tokens/stepMedian time/stepGPUsTraining GPU-hAllocated Peak Memory
8BStandard GRPO786K—492 s161,07633.7 GB
8BPer-role780K918K753 s161,85734.9 GB
8BEnvACE704K627K597 s161,27630.6 GB
1.7BStandard GRPO764K—429 s1696725.5 GB
1.7BPer-role595K884K450 s161,08034.4 GB
1.7BEnvACE654K775K533 s161,24127.0 GB

Tokens/step is the sum of input tokens and generated tokens across all model calls within one training step. The Qwen3-30B-A3B judge runs as an independent service shared across all three methods and is not tallied separately. Under the same 16-GPU configuration, EnvACE-8B uses ~19% more training GPU-hours than standard GRPO, 1,276 versus 1,076, in exchange for a 5.5-point improvement on τ²-Bench and without requiring an extra simulator service; compared with Per-role, EnvACE-8B reduces per-step training time by ~21% and GPU-hours by ~31%. Peak memory measures the peak allocated memory per GPU during training to compare shared parameters against separate policies: EnvACE trains only one set of parameters, whereas Per-role trains two sets on the same GPUs; peak memory is lower for EnvACE at both scales, at 30.6 versus 34.9 GB for 8B and 27.0 versus 34.4 GB for 1.7B.

As described in §5.1, the training setup uses 16 H20 GPUs for 470 steps. On the inference side, following Algorithm 1, the overhead of NN rehearsals consists of NN private simulations, NN self-evaluations, 1 summarization, plus 1 real execution. Empirically, for N=2N=2 parallel rehearsal, each task involves ~12 rehearsal calls and ~19K additional generated tokens, about 1.8× that of the final real execution, with wall-clock time ~2× that without TTS; among this, rehearsal response generation itself accounts for only ~10% of the additional generation, while the main overhead comes from repeated exploration in the simulated environment. None of this overhead requires interacting with external environments. As shown in Figures 6 and 9, a moderate budget already yields clear gains.

Dear Reviewer 4vFn,

Thank you again for your comments and for taking the time to engage with our work.

As the discussion period is coming to an end, we wanted to check whether our responses have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.

If you feel that your concerns have been adequately addressed, we would also appreciate it if you could take our responses into account when finalizing your evaluation.

Dear Reviewer 4vFn,

Thank you again for your comments and for taking the time to engage with our work.

As the discussion period will end within one day, we wanted to check whether our responses have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.

If you feel that your concerns have been adequately addressed, we would also appreciate it if you could take our responses into account when finalizing your evaluation.

Reviewer_MjNbOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains a shared policy to alternate between generating actions and simulating tool responses. The authors claim that this “world rehearsal” internalizes environment dynamics, improves downstream agent performance, and supports test-time rehearsal before real execution.

Critical strengths
  1. Evaluation spans four benchmarks covering function calling, stateful interactions, and financial tools.
  2. The method did report some improvements over standard GRPO (especially in 8B)
Critical weaknesses
  1. Limited contribution beyond existing works. "Simulating Environments with Reasoning Models for Agent Training" already trains agents using simulated tool outputs without executable environments. EnvACE’s remaining distinction is sharing parameters and jointly optimizing acting and response generation; the evidence for the added value of this combination is limited. This is a narrower contribution than introducing to me.
  2. The practical motivation is not adequately validated as that avoiding environment construction is potentially useful, but the paper provides no matched comparison of training cost, throughput, or task coverage against executable environments. 3.The central insight lacks direct experimental support. The evaluations primarily measure task performance, not tool-response accuracy, state consistency, or long-horizon prediction quality.
  3. The "world-model" framing exceeds what the objective establishes and is some-what overstated. Generated observations receive task-success rewards without an explicit constraint tying them to actual environment transitions. Reward-favorable but incorrect responses remain a plausible failure mode. Terms such as “internalizing environment dynamics” and “agent world model” imply capabilities that are neither enforced by the objective nor directly demonstrated.
Questions for authors
  1. I think Fig. 1 is overly flashy and lacks substantial practical content. I’m more curious about which AI generated it.
  2. Eq.7: With paired ACT–REHEARSE outputs, both role groups contain the same trajectory rewards with the same multiplicities, yielding identical baselines. How are terminal actions handled, and when does role-wise advantage estimation differ from using a shared baseline?
  3. Eqs. 7-8: The baseline weights each trajectory’s reward by its number of outputs for that role. Is this length weighting intentional? Please clarify how the loss is averaged across trajectories, outputs, and tokens.
8 comments
AuthorsOpenReview ↗

Author Response to Reviewer MjNb [Part 1/3]

We appreciate your detailed comments and insightful questions. In the revised PDF, we have marked all changes in blue for ease of reference.

W1. Contribution Beyond Simulated-Environment Training

We have already included Simulating Environments with Reasoning Models for Agent Training as the Simulator-8B baseline in our comparisons. The key distinction between the two is that Simulator-8B uses an external reasoning model to simulate environment feedback while the policy being trained merely consumes these responses; in EnvACE, environment responses are generated by the policy being trained itself and are jointly optimized via RL alongside acting. This leads to four testable differences:

  • Performance: Simulator-8B achieves an Overall score of 20.03 versus 32.91 for EnvACE-8B, and is clearly weaker on BFCL at 19.78 and VitaBench at 1.8.
  • Role of parameter sharing: The Per-role variant likewise generates environment responses during training, but does not share parameters between the two roles. As shown in Figure 3, EnvACE outperforms it at both scales, showing that knowledge learned during rehearsal indeed transfers to the acting policy.
  • New capability: After training, the policy itself can rehearse privately at test time. In Table 3, Figure 6, and Figure 9, EnvACE rehearsal clearly outperforms base-model rehearsal, a capability absent in the external-simulator paradigm.
  • Cost: In Li et al., environment feedback comes from proprietary reasoning models under an SFT-then-RL pipeline: the SFT stage uses GPT-5 and o4-mini to expand seed trajectories into 90K τ² trajectories; during RL, o4-mini is called at every turn to simulate the environment with up to a 60K-token context limit and again at task completion to judge success. That paper does not report API call counts or monetary cost, so we cannot provide a numerical comparison. EnvACE has no SFT stage, requires no synthetic SFT data, and starts RL directly from Qwen3-8B; nor does it call any external model during training: environment responses are generated on the training GPUs by the policy being trained, and the judge is a locally deployed open-source model, Qwen3-30B-A3B. As shown in the table below, EnvACE-8B trains for 470 steps on 16 H20 GPUs for ~1,276 GPU-hours in total, with rehearsal accounting for ~47% of per-step tokens.

W2. Practical Motivation and Cost

EnvACE removes the need to build and run executable environments, as well as the SFT stage, starting RL directly from Qwen3 models. We added empirical measurements of training cost in revised Appendix E.4: on 16 H20 GPUs, EnvACE-8B takes ~10 minutes per training step with a median of 597 s, totaling ~1,276 GPU-hours over 470 steps; each step processes ~1.33M tokens, of which rehearsal accounts for 47%. The full comparison is below:

ScaleMethodACT tokens/stepREHEARSE tokens/stepMedian time/stepGPUsTraining GPU-hAllocated Peak Memory
8BStandard GRPO786K—492 s161,07633.7 GB
8BPer-role780K918K753 s161,85734.9 GB
8BEnvACE704K627K597 s161,27630.6 GB
1.7BStandard GRPO764K—429 s1696725.5 GB
1.7BPer-role595K884K450 s161,08034.4 GB
1.7BEnvACE654K775K533 s161,24127.0 GB

Tokens/step is the sum of input tokens and generated tokens across all model calls within one training step. The Qwen3-30B-A3B judge runs as an independent service shared across all three methods and is not tallied separately. Under the same 16-GPU configuration, EnvACE-8B uses ~19% more training GPU-hours than standard GRPO, 1,276 versus 1,076, in exchange for a 5.5-point improvement on τ²-Bench and without requiring an extra simulator service; compared with Per-role, EnvACE-8B reduces per-step training time by ~21% and GPU-hours by ~31%. Peak memory measures the peak allocated memory per GPU during training to compare shared parameters against separate policies: EnvACE trains only one set of parameters, whereas Per-role trains two sets on the same GPUs; peak memory is lower for EnvACE at both scales, at 30.6 versus 34.9 GB for 8B and 27.0 versus 34.4 GB for 1.7B.

On the inference side, following Algorithm 1, the overhead of NN rehearsals consists of NN private simulations, NN self-evaluations, 1 summarization, plus 1 real execution. Empirically, for N=2N=2 parallel rehearsal, each task involves ~12 rehearsal calls and ~19K additional generated tokens, about 1.8× that of the final real execution, with wall-clock time ~2× that without TTS. None of this overhead requires interacting with external environments.

AuthorsOpenReview ↗

Author Response to Reviewer MjNb [Part 2/3]

W3. Direct Evidence and "World Model" Framing

Regarding direct evidence. Beyond our indirect evidence—where fixed-actor EnvACE-8B rehearsal in Appendix B significantly outperforms base rehearsal in Table 3 across budgets in Figure 6 and across all budgets on τ² in Figure 9, and Appendix A anticipates failures beforehand such as a missing "#" in an order ID in Figure 7 and switching from disallowed basic-economy writes to read-only queries in Figure 8—we added a direct response fidelity evaluation in revised Appendix E.3. Across 6,649 real tool calls on unseen τ²-Bench covering airline, retail, and telecom trajectories from EnvACE-8B, EnvScaler-8B, and Qwen3-8B, each model predicts the tool return from the real history and action under the rehearsal protocol without few-shot examples via greedy decoding, with 95% task-resampled CIs in brackets:

ModelStatus Acc.Error F1Field F1Read-after-Write F1
Qwen3-8B Base74.5 [72.7, 76.3]38.4 [33.0, 43.1]12.3 [11.5, 13.1]17.8 [15.1, 20.7]
EnvACE-8B85.5 [82.9, 87.7]64.4 [60.5, 68.0]16.4 [15.4, 17.6]25.3 [22.5, 28.4]

EnvACE clearly outperforms the base model on all metrics with non-overlapping CIs, holding consistently across trajectory sources. Read-after-Write F1 counts only read calls on objects previously modified by a write call, showing that EnvACE better tracks state changes across turns.

Regarding the world model framing. Your assessment rests on two premises: that generated observations receive only task-success rewards, and that no explicit constraint ties them to environment transitions. Neither matches EnvACE's actual training: the reward directly inspects generated tool returns, and per-task reference data explicitly constrains them. We also tested transition alignment and the "reward-favorable but incorrect" failure mode on real environments:

1. Rewards directly inspect generated tool returns. As detailed in §4.2, the LLM judge scores the full trajectory—including rehearsed tool returns—item by item against weighted per-task checklists. Across all 8,129 training tasks and 115,145 checklist items:

Evaluated TargetItem ShareWeight Share
Response to user59.4%60.0%
Tool call name & args19.8%21.2%
Tool return itself18.6%17.6%
Thinking process2.2%1.3%
  • 99.9% of tasks—8,120 out of 8,129—have at least one item checking the tool return itself, and ~36% of these require exact value matches against reference returns, e.g., converted_value == 41.75 and unit == 'meters'. Rehearsed returns are thus direct targets of the reward.

2. Reference data explicitly constrains generated observations. Each CM2 training task provides a reference dialogue with reference tool calls and returns, anchoring both ends:

  • Input side: Rehearsal conditions on this reference data; under Appendix D's protocol, calls matching a reference call in name and similar arguments reuse its return, and otherwise generate a return consistent with the reference.
  • Reward side: Another 8,251 user-response checklist items draw ground-truth evidence directly from reference tool returns, e.g., requiring total emissions 45,000 lbs and factor 2.5 lbs/mile in the reply to match the tool return.

3. "Reward-favorable but incorrect" responses are penalized, not rewarded. Simulating a failing call as succeeding or fabricating convenient numbers contradicts the reference, failing both the tool-return checks in 1 and the response checks in 2 and yielding lower reward. Testing this directly on unseen τ²-Bench confirms that EnvACE does not over-predict success on failing calls:

ModelPredicted Failure on 1,748 Failed CallsFalse Failure on 4,901 Successful Calls
Qwen3-8B Base29.6%8.8%
EnvACE-8B49.1%1.2%

EnvACE predicts genuine failures far more often while making fewer false alarms, showing training pushes rehearsal closer to the real environment rather than toward task-favorable outcomes.

4. Framing. Thus, EnvACE's objective imposes explicit reference-based constraints on generated observations, and our fidelity evaluation directly demonstrates this capability. We agree that reference data and an LLM judge cannot guarantee every return is exact; we report the full evaluation in revised Appendix E.3.

AuthorsOpenReview ↗

Author Response to Reviewer MjNb [Part 3/3]

Q1. Fig. 1

Fig. 1 was created by first generating a draft with GPT and then drawing it by hand. In the revised manuscript, we have replaced it with a cleaner, more information-dense schematic that directly contrasts where observations come from across the three rollout types.

Q2. Eq. 7 — Role Baselines and Terminal Actions

Let ni,rn_{i,r} be the number of outputs of role rr in trajectory ii; then μx,r=∑ini,rRi/∑ini,r\mu_{x,r} = \sum_i n_{i,r} R_i / \sum_i n_{i,r}. The two baselines are identical only when ni,ACTn_{i,\text{ACT}} and ni,REHn_{i,\text{REH}} are proportional across trajectories. In practice, this condition usually does not hold:

  1. As defined in §3, the action space A\mathcal{A} includes both structured tool calls and final answers. Terminal final answers and user-facing text replies belong to ACT without triggering rehearsal; they enter only Gx,ACTG_{x,\text{ACT}} while likewise inheriting RiR_i.
  2. A single action can contain up to 20 parallel tool calls in our implementation, and each call corresponds to one rehearsal, so one ACT output may correspond to multiple REHEARSE outputs.

Therefore, whenever the output ratio between the two roles varies across trajectories and correlates with reward, the two baselines differ. Sampling one step every 10 steps across training rollouts—47 steps × 16 task groups in total—we found μACT≠μREH\mu_{\text{ACT}} \neq \mu_{\text{REH}} in 59% of groups for EnvACE-8B, and in 75% and 63% for 1.7B and 4B, respectively. However, the difference is small: on the 0–1 reward scale, the mean of ∣μACT−μREH∣|\mu_{\text{ACT}} - \mu_{\text{REH}}| is 0.019 at 8B, 0.019 at 4B, and 0.027 at 1.7B, with a median no greater than 0.007 and a 90th percentile around 0.05–0.07. Each group has on average ~18 ACT outputs and ~10 REHEARSE outputs. In other words, the two baselines are not identical in the majority of groups.

Q3. Eqs. 7–8 — Weighting and Loss Averaging

The baseline averages over every output in role group Gx,rG_{x,r}, so each trajectory is weighted by its number of outputs under that role rather than by token count. This is a natural consequence of defining role groups in Eq. 6: each output is an individual optimization unit. For loss aggregation, advantages are computed as A=R−μA = R - \mu without dividing by the within-group standard deviation, consistent with Eq. 7; all tokens of the same output share that output's advantage; and the loss is aggregated via token-mean, with outputs from both roles jointly updating the shared parameters within the same step.

Reviewer_1rGDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The problem this work addresses is that training agents using RL require environments or simulators that can generate the rollouts. Prior works have used separate models for each role (actor vs. simulator), but this work uses a single shared policy that acts as both. The model is trained with GRPO on task-success rewards. The claimed contributions are the concept of world rehearsal, the training method EnvACE, and the experiment results across multiple benchmarks.

Critical strengths

The problem that this work tries to address is well-motivated. Verification is the main bottleneck in agent training using RL. Therefore, a method that can train a model to act both as a simulator and an actor is potentially high-impact. Another strength is that the authors evaluated models of different sizes, which is important for understanding if the method would work for larger models.

Critical weaknesses

My main concern is that the central claim that "EnvACE consistently outperforms strong baselines" across multiple benchmarks is not sufficiently supported by the experimental results. First, EnvACE does not achieve the best performance in terms of the Avg. performance on any of the individual benchmarks (Table 1). It wins only on an overall score by under one point. Since there are no variance estimates (no confidence intervals) for the results, it is unclear whether the gains are significant. Additionally, EnvACE-4B shows an even smaller gain compared to its base model.

Questions for authors

The paper's justification for a new training paradigm that only uses one policy for different roles requires evidence on either empirical performance superiority or other efficiency benefits. Are there any other benefits from using a shared policy?

8 comments
AuthorsOpenReview ↗

Author Response to Reviewer 1rGD [Part 1/3]

Thank you for your thoughtful review and constructive feedback. All revisions in the updated manuscript are highlighted in blue.

W1. Overall Performance and EnvACE-4B

Regarding overall performance. Our goal is to train an agent that is reliable across diverse tool environments, so we prioritize comprehensive performance across environments. Taken across all four benchmarks, EnvACE-8B is the best-performing method overall:

  • Ranked first on both aggregate metrics. The Overall metric in Table 1 aggregates three benchmarks: BFCL V4, τ²-Bench, and VitaBench. Without constructing any executable environments, EnvACE-8B achieves the highest Overall score of 32.91 among all methods with complete results, outperforming even the larger AWM-14B. On the fourth benchmark, FinMCP-Bench in Table 2, EnvACE-8B leads by an even clearer margin: TF1 reaches 46.78, which is 3.10 higher than the runner-up EnvScaler-8B, 4.28 higher than AWM-8B, and 5.54 higher than the base Qwen3-8B; tool precision reaches 54.04, exceeding the highest among all other methods, 39.47, by 14.57. EnvACE-8B is the only method ranking first on both metrics.
  • Consistently near the top on every benchmark. EnvACE-8B is the only method placing in the top three on BFCL V4, τ²-Bench, and VitaBench: it ranks 3rd on BFCL at 46.04, within 1.03 of EnvScaler-8B; achieves the 2nd-highest score on τ²-Bench at 36.7; and ranks 2nd overall on VitaBench at 16.0, which is the highest among all 7B–8B methods.
  • Other methods exhibit clear weaknesses. Simulator-8B scores 38.5 on τ²-Bench, but only 19.78 on BFCL, 1.8 on VitaBench, and 15.95 TF1 on FinMCP-Bench; AWM-14B scores high on BFCL and VitaBench, but reaches only 30.7 on τ²-Bench, ranking 6th despite having more parameters at 14B versus 8B.
  • Controlled comparison. Cross-method comparisons are also affected by differences in training data. A controlled comparison with the same training data and backbone is more informative: in Figure 3, the 8B model improves from 31.2 with standard GRPO to 36.7, a gain of 5.5 points.

In the revised manuscript, we have replaced "consistently outperforms strong baselines" in the Introduction with "achieving the best overall performance."

Regarding EnvACE-4B. EnvACE-4B outperforms its base model on all three benchmarks: from 41.80 to 41.88 on BFCL, from 27.9 to 28.5 on τ²-Bench, and from 9.6 to 10.8 on VitaBench. In revised Appendix E.2, we also added an evaluation of EnvACE-4B on FinMCP-Bench using the same hard subset as Table 2:

ModelTR (%)TP (%)TF1 (%)Multi-Step TF1 (%)Multi-Turn TF1 (%)
Qwen3-4B37.0130.7333.5825.5042.33
EnvACE-4B38.9632.2635.2928.4142.68

EnvACE-4B outperforms the base model across every metric, so 4B improves over its base model on all four benchmarks. Consistent with §5.4, the gains are largest at 8B, indicating that world rehearsal benefits more from stronger base models.

AuthorsOpenReview ↗

Author Response to Reviewer 1rGD [Part 2/3]

W2. Variance Across Runs

As described in §5.1, the non-TTS results in Table 1 are averaged over four independent runs, reported as Avg@4. The table below reports the mean ± std across the four runs. We have updated every cell in revised Table 1 to mean ± std—except ScaleEnv-8B, whose std cannot be reported as it is not open-sourced—and placed the mean ± std for the controlled comparisons of Figure 3 in Appendix E.1:

MethodWebMem.MultiNoLiveLiveIrrel.BFCL Avg.RetailTelecomAirlineτ² Avg.CrossDeliv.Inst.OTAVita Avg.Overall
Qwen3-1.7B1.88±0.2512.58±1.3715.75±0.4782.94±0.4774.06±0.4075.71±0.4830.89±0.324.6±0.511.6±3.615.0±2.610.4±1.50.0±0.03.7±1.03.3±1.50.7±0.51.9±0.514.41±0.53
Qwen3-4B8.25±0.8720.86±1.7337.25±0.8988.16±1.0081.55±0.4178.33±0.4441.80±0.4832.7±4.013.2±3.338.0±3.727.9±2.11.8±1.513.7±4.618.2±5.74.8±1.09.6±1.926.43±0.96
Qwen3-8B10.63±1.3622.15±3.5542.09±0.8588.09±0.4680.90±0.7179.58±0.3444.04±0.8141.2±4.020.8±2.928.0±2.830.0±1.91.2±1.515.5±3.324.8±3.94.0±2.211.4±1.428.48±0.84
Simulator-8B9.38±1.655.65±0.971.47±0.4832.46±0.7144.30±0.7986.54±0.7819.78±0.4349.3±3.215.1±2.551.0±6.938.5±2.70.0±0.01.2±1.64.8±2.81.0±0.81.8±0.820.03±0.94
TOUCAN-7B14.00±2.4815.97±2.1722.59±0.7676.11±1.1272.61±0.1476.89±0.3035.33±0.7130.0±1.710.1±0.927.0±3.522.4±1.30.5±0.66.9±1.43.0±0.80.8±1.02.8±0.520.18±0.52
EnvScaler-8B15.38±0.8019.52±1.5452.25±1.9386.40±0.5081.70±0.3476.03±0.4147.07±0.6848.0±2.316.2±0.834.5±5.932.9±2.16.0±0.820.8±3.329.5±2.57.0±1.915.8±1.231.92±0.84
AWM-8B15.00±2.3519.46±2.6342.81±2.0588.47±0.5480.64±0.6176.41±0.2944.29±0.9441.3±3.418.2±1.134.0±4.531.2±1.91.5±0.616.0±3.619.2±3.34.2±1.010.2±1.228.56±0.83
AWM-14B14.88±1.7021.88±2.5249.06±1.3989.46±0.3481.41±0.6881.59±0.9047.32±0.7541.9±2.519.1±1.131.0±5.330.7±2.08.0±0.824.8±3.833.8±3.911.8±1.519.6±1.432.54±0.85
ScaleEnv-8B–––––––50.927.237.538.53.026.323.87.015.0–
EnvACE-1.7B3.75±0.2915.70±0.3514.38±0.7983.13±0.1574.76±0.4278.14±0.1331.81±0.267.9±1.714.5±2.323.5±1.915.3±1.10.0±0.07.5±1.35.0±0.80.3±0.53.2±0.416.77±0.41
EnvACE-4B12.75±2.3524.74±1.0229.63±0.3487.86±0.4682.27±0.2984.79±0.4041.88±0.5336.8±4.112.7±2.336.0±3.028.5±1.91.5±0.818.0±3.419.0±3.04.5±2.110.8±1.327.06±0.77
EnvACE-8B12.25±1.2124.03±0.9345.29±1.0887.59±0.3981.20±0.1483.19±0.3346.04±0.4548.9±4.117.3±1.544.0±4.136.7±2.06.0±1.024.0±3.627.0±2.27.0±1.916.0±1.232.91±0.79
AuthorsOpenReview ↗

Author Response to Reviewer 1rGD [Part 3/3]

Q1. Additional Benefits of a Shared Policy

Beyond task performance, using a shared policy brings benefits in test-time capability, training efficiency, deployment, and rehearsal quality:

  1. Performance comes more from parameter sharing than merely from "adding a simulator." In Figure 3, all three settings use the exact same training data, judge, and rollout setup, differing only in the source of environment responses. At 8B, standard GRPO with a frozen external simulator achieves 31.2, Per-role—which also generates environment responses during training, but with separate parameters for the two roles—achieves 35.5, and EnvACE achieves 36.7; at 1.7B, the scores are 14.7, 14.0, and 15.3, respectively. At 1.7B, Per-role even underperforms standard GRPO, whereas parameter sharing performs best at both scales. This shows that the gain comes not merely from having a learnable simulator during training, but from a single set of parameters jointly learning acting and rehearsal: environment knowledge learned during rehearsal can be directly utilized by the acting policy.
  2. Test-time capability: the same model can rehearse privately without an extra simulator. With the acting policy fixed to EnvACE-8B, Table 3 shows that rehearsing with EnvACE is clearly better than rehearsing with the base model, scoring 40.9 versus 36.8 in parallel mode and 38.5 versus 34.9 in sequential mode, compared with 36.7 for Non-TTS. As the rehearsal budget increases in Figures 6 and 9, EnvACE continues to improve: BFCL Multi-Turn parallel rises from 42.12 to 49.50, and τ²-Bench parallel rises from 31.85 to 40.64. Training the two roles separately would require loading two models simultaneously at deployment; a shared policy requires only one.
  3. Training efficiency: more economical than separate policies. At the 8B scale under the same 16-GPU configuration, EnvACE reduces median training time per step by ~21%, from 753 s to 597 s, and reduces total GPU-hours by ~31%, from 1,857 to 1,276, compared with Per-role. Because EnvACE trains only one set of parameters, per-GPU peak memory is also lower: 30.6 versus 34.9 GB at 8B, and 27.0 versus 34.4 GB at 1.7B. Compared with standard GRPO, EnvACE also avoids separately deploying and maintaining a simulator service, as detailed in revised Appendix E.4.
  4. Rehearsal itself also improves. Because EnvACE's rehearsal role is optimized via RL alongside the acting role, its generated environment responses become more accurate with training. In revised Appendix E.3, comparing against ground-truth returns from real environments on τ²-Bench, which is unseen during training, EnvACE achieves 85.5% accuracy on predicting tool-return success versus error compared with 74.5% for the base model, and improves error F1 from 38.4 to 64.4.
  5. No reliance on executable environments. Training requires only tool schemas, reference dialogues, and checklists, without building or invoking executable environments for each domain. Methods such as EnvScaler, AWM, and ScaleEnv must first synthesize and execute environments; extending EnvACE to new tool domains requires only the corresponding data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.