- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework that improves agent training by adapting executable training environments to the target model’s weaknesses. It introduces three components for analyzing failure trajectories, checking and refining generated environments, and organizing training feedback to guide future environment generation. The generated tasks are used for GRPO training. Experiments on two models and four benchmarks show improvements over two author-designed baselines, while ablations support the usefulness of the three components.
Critical strengths
-
The paper aims to address a meaningful problem. Generating useful training environments is important for improving agent capabilities. Adapting these environments to the model’s changing weaknesses is a useful direction.
-
The paper provides clear analyses of three failure modes. These cover behavioral analysis, environment construction, and feedback use. The proposed components address these problems, and the ablations support their usefulness.
Critical weaknesses
- The paper lacks comparisons with existing methods that address similar problems. It evaluates only two author-designed baselines, although its related work discusses several methods that address different aspects of these three failure modes. It does not compare against the prior approaches. This leaves the baseline comparison limited and makes it difficult to assess the benefit of addressing all three problems within the proposed framework.
Questions for authors
-
Could the authors compare EnvOpt with at least one relevant prior method under comparable settings? This would help clarify whether the proposed framework improves upon existing approaches. During the rebuttal, if the authors can add at least one relevant prior method under comparable settings and demonstrate a clear improvement, I would be willing to raise my score.
-
As an optional suggestion that does not affect my score, the current ablations already show that each component helps when added individually. I would also be interested in removing one component at a time from the full framework to understand its contribution when the other two are present.
5 comments
Response to the Author
Thank you for the additional experiments. Before updating my score, I would like to confirm that the comparison is made under comparable settings:
The cross-benchmark table in your response to Reviewer qxY8 suggests that EnvOpt is evolved separately for each benchmark using its public-split feedback. Were the baselines also trained per benchmark with equivalent information? If they were trained as general models, the jointly optimized EnvOpt seems the fairer comparison, and it is below FrogNano and RODS on τ²-Bench and below RODS on BFCL-v4. Please also specify the GRPO configuration and the data-generation LLM used for each baseline.
Also, AWM and EnvScaler fall below the base model on most benchmarks, unlike in their original papers. Could you comment on possible causes?
I will consider updating my score after these clarifications.
Thank you for these questions. We appreciate the opportunity to clarify the comparison settings and reproduction details.
1. Configuration of the dynamic baselines. Yes, both FrogNano and RODS are optimized separately for each benchmark. They use the same task splits and have access to the same information as EnvOpt, including public-split feedback. The way this information is used and organized follows each method’s design.
2. Clarification on jointly optimized EnvOpt. The joint experiment examines whether EnvOpt can optimize for a broader task distribution spanning multiple domains and generalize beyond the domains providing feedback, a setting closer to practical applications. Its total training-task budget remains approximately 1,000 tasks, rather than 1,000 tasks per domain. Joint optimization distributes the same total training budget across multiple domains, resulting in a substantially smaller training budget per domain than benchmark-specific optimization. A direct comparison would therefore not provide a fair assessment of the methods. Consequently, we use benchmark-specific runs for the additional baseline comparison and report joint optimization as a complementary experiment.
3. GRPO configuration and generation models. We use the same core GRPO configuration across methods, as summarized below.
| Configuration | Setting |
|---|---|
| Rollout batch size | 8 prompts |
| Rollouts per prompt | 8 |
| Global batch size | 64 trajectories per update |
| Learning rate | 3e-6, constant |
| Optimizer | Adam, β₁ = 0.9, β₂ = 0.98 |
| Weight decay | 0.1 |
| KL loss coefficient | 0.005, low_var_kl |
| PPO clipping | Lower: 0.2; upper: 0.28 |
| Entropy coefficient | 0 |
| Rollout temperature | 0.7 |
| Maximum context length | 65,536 tokens |
| Maximum response length | 16,384 tokens |
For dynamic generation, EnvOpt, FrogNano, and RODS use the same GPT-6 Astra for planner roles and GPT-5.5 for worker roles. This matches the planner/worker model configuration in our submission. For AWM and EnvScaler, we train directly on their released open-source datasets. These reproductions therefore inherit the original data-generation pipelines.
4. AWM and EnvScaler reproduction results. We have carefully checked both reproductions and followed their released setups as closely as possible within the shared training protocol. One possible explanation for the weaker results is the restricted data budget: our main comparison uses approximately 1,000 training tasks per method, which may be insufficient for these static training pools to deliver the gains reported at their original scales.
To examine this possibility, we also included AWM and EnvScaler with 3,000 tasks above in the comparison table, approximately three times the budget of the dynamic methods. Increasing the budget from 1,000 to 3,000 tasks does not produce consistent improvements across benchmarks. Time constraints during the rebuttal period prevented us from conducting larger-scale experiments, such as those involving 10,000 tasks. However, these results suggest that targeted dynamic generation can offer greater data efficiency within a fixed training-task budget.
We hope our responses address your concerns. Thank you again for your insightful feedback. :)
Thank you for recognizing the importance of adaptive environment generation and the usefulness of our failure-mode analysis. We address your request for comparisons with prior methods and your additional ablation suggestion below.
W1 & Q1. Additional Baseline Results
Thank you for this suggestion. We add comparisons with four prior methods on Qwen3-4B-Instruct-2507: FrogNano and RODS, which iteratively synthesize training data, and two static environment-scaling methods, AWM (Agent World Model) and EnvScaler.
To improve comparability, we match the total number of training tasks across methods. EnvOpt runs for 10 iterations with approximately 96 tasks per iteration, totaling around 1,000 tasks. We therefore use 1,000 training tasks for each baseline. To allow the static environment-scaling methods to benefit from larger data pools, we additionally evaluate AWM and EnvScaler with 3,000 training tasks each.
With a comparable or smaller training-task budget, EnvOpt outperforms both static environment-scaling and dynamic data-generation baselines on most benchmarks, trailing RODS only on BFCL-v4. These results demonstrate that EnvOpt is highly competitive.
The ACEBench-Agent reproduction for RODS was not completed due to time constraints.
[1] FrogNano: Training a 4B Coding Agent via Online Task Synthesis, arXiv, 2026
[2] RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents, arXiv, 2026
[3] Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning, ICML, 2026
[4] EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis, Findings of ACL, 2026
Q2. Component Ablation Results
Thank you for this suggestion. We add component ablations on Qwen3-4B-Instruct-2507, removing TBM, IEC, or TCR individually while retaining the other two components. These experiments complement the single-component ablations in the paper and assess each component's contribution when the other two are present. We report the base model and full EnvOpt for reference. The results are shown below.
Ablation results. Full EnvOpt outperforms every ablation on all four benchmarks, with gains of 1.20–8.91 percentage points. Removing IEC causes the largest drop on each benchmark, highlighting the importance of iterative environment construction. Removing TBM or TCR also consistently reduces performance, supporting the contribution of all three components.
Thank you again for your valuable feedback. We welcome any further questions or suggestions and are happy to provide additional clarifications and explanations. If our rebuttal aligns with your expectations, we kindly request that you consider raising the score. Thank you. :)