- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EnvACE, a reinforcement learning method for tool-using LLM agents in which a shared policy alternates between generating actions and simulating the resulting environment responses. Both roles are optimized using trajectory-level task rewards with role-wise GRPO, allowing training rollouts to proceed without external environment calls. At inference time, the model can rehearse interactions privately and summarize them into a memory that guides execution in the actual environment. The method is evaluated on four tool-use benchmarks, with additional analyses of parameter sharing, model scale, and test-time rehearsal.
Critical strengths
- The central design is simple and addresses a practical bottleneck in agent training. Rather than maintaining a separate simulator or using response prediction only as an auxiliary objective, EnvACE incorporates response generation directly into the policy’s rollout and optimization. Sharing parameters between acting and rehearsal provides a concrete mechanism for environment-modeling experience to influence decision making, without requiring an executable environment for each training interaction.
- The empirical results support the usefulness of this design beyond a single benchmark. At the 8B scale, EnvACE improves τ²-Bench performance from 31.2% with standard GRPO to 36.7%, and outperforms the separate-policy variant at 35.5% (Figure 3). The test-time comparison is also informative: with the acting policy held fixed, parallel rehearsal using EnvACE performs better than rehearsal using the base model on both benchmarks in Table 3. These results support the value of learned rehearsal rather than simply adding more inference steps.
Critical weaknesses
- The evidence for accurately internalizing environment dynamics is less direct than the evidence for improved task performance. The parameter-sharing ablation and case studies are suggestive, but the paper does not quantitatively evaluate response fidelity or consistency across state-changing interactions. Since generated responses receive the same task reward as actions, it remains unclear how the training signal distinguishes faithful simulation from responses that merely make the imagined task easier to complete. This does not invalidate the downstream gains, but it limits the interpretation of what the internalized world model has learned.
- The claimed scalability advantage is not quantified in computational terms. Avoiding executable environments is valuable, but EnvACE also generates environment responses and uses an LLM judge; test-time rehearsal adds simulation, evaluation, and summarization. Section 5.1 reports the training configuration, but not comparative token or wall-clock costs. The results therefore establish effectiveness more clearly than the overall resource advantage over conventional agent training.
Questions for authors
- What information or checks ground the training reward when tool responses are generated by the policy itself? Appendix D mentions ground-truth reference calls for test-time simulation; please clarify what reference information is available during training and whether you have measurements of response fidelity or state consistency that help distinguish accurate simulation from task-favorable fabrication.
- Can you report generated-token counts and wall-clock costs for EnvACE versus the standard GRPO baseline, including reward judging, together with the inference overhead of the two-attempt rehearsal setting? This would help assess the practical tradeoff behind the scalability claim.
4 comments
Author Response to Reviewer 4vFn [Part 2/2]
W2 & Q2. Training and Inference Costs
We compiled the training costs in the table below and added them to revised Appendix E.4.
| Scale | Method | ACT tokens/step | REHEARSE tokens/step | Median time/step | GPUs | Training GPU-h | Allocated Peak Memory |
|---|---|---|---|---|---|---|---|
| 8B | Standard GRPO | 786K | — | 492 s | 16 | 1,076 | 33.7 GB |
| 8B | Per-role | 780K | 918K | 753 s | 16 | 1,857 | 34.9 GB |
| 8B | EnvACE | 704K | 627K | 597 s | 16 | 1,276 | 30.6 GB |
| 1.7B | Standard GRPO | 764K | — | 429 s | 16 | 967 | 25.5 GB |
| 1.7B | Per-role | 595K | 884K | 450 s | 16 | 1,080 | 34.4 GB |
| 1.7B | EnvACE | 654K | 775K | 533 s | 16 | 1,241 | 27.0 GB |
Tokens/step is the sum of input tokens and generated tokens across all model calls within one training step. The Qwen3-30B-A3B judge runs as an independent service shared across all three methods and is not tallied separately. Under the same 16-GPU configuration, EnvACE-8B uses ~19% more training GPU-hours than standard GRPO, 1,276 versus 1,076, in exchange for a 5.5-point improvement on τ²-Bench and without requiring an extra simulator service; compared with Per-role, EnvACE-8B reduces per-step training time by ~21% and GPU-hours by ~31%. Peak memory measures the peak allocated memory per GPU during training to compare shared parameters against separate policies: EnvACE trains only one set of parameters, whereas Per-role trains two sets on the same GPUs; peak memory is lower for EnvACE at both scales, at 30.6 versus 34.9 GB for 8B and 27.0 versus 34.4 GB for 1.7B.
As described in §5.1, the training setup uses 16 H20 GPUs for 470 steps. On the inference side, following Algorithm 1, the overhead of rehearsals consists of private simulations, self-evaluations, 1 summarization, plus 1 real execution. Empirically, for parallel rehearsal, each task involves ~12 rehearsal calls and ~19K additional generated tokens, about 1.8× that of the final real execution, with wall-clock time ~2× that without TTS; among this, rehearsal response generation itself accounts for only ~10% of the additional generation, while the main overhead comes from repeated exploration in the simulated environment. None of this overhead requires interacting with external environments. As shown in Figures 6 and 9, a moderate budget already yields clear gains.
Dear Reviewer 4vFn,
Thank you again for your comments and for taking the time to engage with our work.
As the discussion period is coming to an end, we wanted to check whether our responses have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.
If you feel that your concerns have been adequately addressed, we would also appreciate it if you could take our responses into account when finalizing your evaluation.
Dear Reviewer 4vFn,
Thank you again for your comments and for taking the time to engage with our work.
As the discussion period will end within one day, we wanted to check whether our responses have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.
If you feel that your concerns have been adequately addressed, we would also appreciate it if you could take our responses into account when finalizing your evaluation.
Author Response to Reviewer 4vFn [Part 1/2]
We are grateful for your time and helpful suggestions on our work. All corresponding updates have been highlighted in blue in the revised manuscript.
converted_value == 41.75andunit == 'meters'. Consequently, rehearsal returns that deviate from the reference receive lower rewards.Read-after-Write Field F1 counts only read calls on objects that previously underwent a write operation. EnvACE outperforms the base model on every metric with non-overlapping confidence intervals, and breaking down the statistics by trajectory source yields consistent conclusions. Across the 1,748 genuinely failed calls, EnvACE predicts failure 49.1% of the time versus 29.6% for the base model; across the 4,901 genuinely successful calls, its false-failure rate is only 1.2% versus 8.8% for the base model. This demonstrates that EnvACE's rehearsal responses are closer to true environment transitions rather than biased toward task-favorable outcomes. In addition, the consistent advantage of EnvACE rehearsal over base rehearsal in Table 3, Figure 6, and Figure 9 indirectly indicates that the responses generated by EnvACE are more helpful for real execution.