acceptodds
Withdrawn from ICLR 2027

ResponsiveEnv: Evolving Training Worlds for LLM Agents through Online Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM agents learn by interacting with their environments. Recent work on environment scaling seeks to improve agent learning by expanding the number and diversity of training environments. We instead obtain more useful experience from existing worlds by revising their conditions while preserving task intent. A limited pool of original worlds can then provide varied practice for the same tasks. Useful conditions depend on the learner's current capabilities. We therefore propose ResponsiveEnv, a framework that revises world conditions throughout training based on current learner trajectories. An environment editor simplifies or hardens selected worlds, and validated revisions replace them for subsequent training. We evaluate ResponsiveEnv on unedited ALFWorld and τ²-bench tasks. On ALFWorld, we edit 100 original worlds with Claude Opus 4.8, GPT-5.6 Sol, or a distilled 8B editor. Each editor raises the unseen-task success rate by 18 to 19 percentage points over static training on the same worlds. Editing these 100 worlds also outperforms training on 2,288 static worlds under the same learner-update budget. For the 8B editor, replacing the current learner's trajectories with success rates alone, or with the initial policy's trajectories, lowers the unseen-task success rate by 6.3 and 5.5 points, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Agent, Environment Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_upHeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ResponsiveEnv, a method for dynamic environment selection for training LLM agents using an environment editor. The editor simplifies or hardens the training environment according to the learner’s current needs. The paper asks whether revisiting modified environments can be more beneficial as training data than training on a larger set of environments, and compares the two methods under the same learner-update budget using ALFWorld and τ2\tau^2-bench.

Critical strengths

The authors present performance gains in most cases, which justifies their method, and I find the ablation study in Section 4.1 important for understanding the nature of this gain.

Critical weaknesses

The authors compare their method to previous methods only on the ALFWorld benchmark, which is also the only benchmark used to demonstrate the ablation studies. I find this significantly weakens the paper, as it is difficult to assess the generalisability of the proposed approach.

Questions for authors
  1. Could the authors repeat the ablation studies with the τ2\tau^2-bench tasks? Or provide a reasonable explanation for their absence?
  2. The authors used 100 original worlds, but the learner, in fact, encounters many more than that during training since they are being edited. However, the presented results compare with 100 static environments. Can the authors report how many environments the learner actually sees when using ResponsiveEnv? Can the authors clarify why they think it makes more sense to compare with 100 static worlds rather than the real number of different worlds?
  3. The only direct comparison between different methods presented in the paper uses GPT-5.6, which makes it hard to understand the contribution of ResponsiveEnv compared to the other methods. Can the authors expand the comparison to other editor models as well?
  4. Table 3 presents the results for a fixed-curriculum dataset. Can the authors clarify why these results do not belong in Table 1 alongside those for the other editor modes? For instance, it seems that Gemini-Flash loses in all 3 categories to the fixed curriculum.
Reviewer_CrnYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ResponsiveEnv repeatedly revises existing training worlds based on a learner’s current performance, while keeping task goals unchanged and the worlds solvable. An editor uses success rates and behavioral summaries to make selected worlds easier or harder; validated revisions replace earlier versions during training. On ALFWorld, editing 100 original worlds substantially improves held-out success over static training on the same worlds. The paper also reports results on τ²-bench and controls for editing strategy and learner feedback.

Critical strengths
  1. A useful alternative to expanding the number of training tasks. ResponsiveEnv adapts existing worlds to the learner rather than treating each world as fixed. On ALFWorld Unseen, the Opus-edited 100-world configuration reaches 86.15% success, compared with 67.21% for static training on the same 100 worlds (Tables 1 and 8). The size of this gain makes the approach worth investigating.
  2. Controls that help identify what matters. Fixed-curriculum, one-time-editing, random-editing, and learner-feedback comparisons test several simpler explanations for the result (Table 3; Appendix C). They provide evidence that repeated editing informed by the current learner is more useful than these alternatives under the reported protocols.
Critical weaknesses
  1. The central environment-efficiency comparison does not match total training resources. Editing 100 worlds exceeds static training on 2,288 worlds at the same number of learner updates, but editing requires additional learner episodes for checks and candidate evaluation. Relative to the static 100-world baseline, total learner episodes are 1.88× for the SFT-8B editor and 2.31× for the Opus editor; wall-clock costs also rise (Table 7). Without a comparison at matched total interaction or compute, the result supports efficiency in original-world count, but not necessarily overall training efficiency. This distinction is important to my assessment of the paper’s main claim.
  2. The reliability of the reported gains is not established across training runs and domains. Each configuration reports a single learner training run. The bootstrap intervals condition on those trained checkpoints and therefore do not capture training-run variability (Appendix A.7). The τ²-bench aggregate gains are smaller, and their reported confidence intervals include zero (Appendix B.5). I cannot tell whether the principal conclusions would hold consistently beyond the reported ALFWorld runs.
Questions for authors
  1. Can the authors provide independent training-seed results for the main ALFWorld comparisons and further evidence on the stability of the τ²-bench gains? Instance-bootstrap intervals alone do not answer this question.
  2. If total learner episodes, tokens, or wall-clock time include editing-side checks and probes, how does ResponsiveEnv compare with static training and the fixed curriculum at the same budget? This would materially affect the efficiency claim.
Reviewer_BtY2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Instead of scaling the number of training environments, the paper scales the world by repeatedly editing the world conditions of existing environments, in the spirit of curriculum learning. When a Qwen learner is trained with GRPO, the method outperforms the world-editing approaches of EnvHarness and RLAnything by 18-19 percentage points on unseen tasks.

Critical strengths

S1. A distilled SFT-8B editor matches the performance of frontier editors, which makes the method highly practical (Table 1, Table 2).

Critical weaknesses

W1. Limited methodological novelty. Fixing the goal and adapting only the world conditions to the current learner is the standard recipe in automatic curriculum research. Reverse curriculum (Florensa et al., 2017) keeps the goal fixed and moves the start state according to learner performance, ACCEL (Parker-Holder et al., 2022) mutates levels and selects them based on the learner, and EnvGen (Zala et al., 2024) has an LLM revise the environment configuration based on agent feedback. Section 5 claims three points of difference, but the first (fixed goal) is identical to reverse curriculum, the second (keeping edited worlds and re-editing them) is the same as level reuse in ACCEL, and the third (trajectory-conditioned editing) was already used by EnvHarness (Huang et al., 2026). If the contribution is the combination of these three elements, the closest methods, EnvGen and ACCEL, should be included as baselines in the 100-world setting of Table 1. Both are missing, and the random operations control (Table 3) has no regret-based selection, so it cannot stand in for ACCEL.

W2: Performance depends on the choice of editor. On ALFWorld Unseen, switching only the editor from Gemini 3.8 Flash to Opus changes the score by 10.73 points (Table 1), and this gap is far larger than the 3.40-point gain over the 2,288-world static baseline (Section 3.3). On τ²-bench, the domain in which improvements appear differs by editor, and SFT-8B falls below the static baseline on telecom (Table 2). Section 3.2 interprets these results as evidence that editor capability is rewarded, but no analysis is given of why the Gemini setting is weaker (whether due to the rate of invalid edits, the proportion of edited worlds outside the target band, or errors in editing direction). It is therefore impossible to judge whether the method works with weaker editors. Per-editor edit-quality metrics and a failure-case analysis are needed.

Questions for authors

Q1. In Table 3, why is the Seen success rate nearly identical between the success-rate-only variant and the full method?

Q2. What is the API cost of calling the frontier editors (Opus, GPT)? Can the total cost per training run be reported relative to the static baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.