- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces ENVHARNESS, a decorator style layer that reshapes a static training environment solely at the standard reset and step interface, so that the original human authored verifier is preserved intact. The framework instantiates three composable but non commutative wrappers: Stage, Contrac and Chain. To automate the selection and parameterization of these wrappers, the authors propose ENVRIGGER, a black box loop consisting of Observe, Diagnose, Write, and Validate stages, in which an LLM sharing the policy's backbone inspects rollout trajectories, synthesizes candidate wrappers targeting diagnosed weaknesses, and admits them through fresh rollouts under a challenging yet solvable criterion, iterating until acceptance or budget exhaustion.
Critical strengths
-
The paper is well motivated in identifying the static, agent agnostic nature of hand built training environments as a concrete limitation.
-
The empirical evaluation is comprehensive, spanning five benchmarks across four domains under both skill learning and rl.
Critical weaknesses
-
The core contribution essentially amounts to porting the decorator pattern from software engineering into RL environment construction and coupling it with an LLM as diagnoser loop to populate wrapper parameters, which positions the work as an engineering framework rather than a methodological advance and leaves its algorithmic novelty limited.
-
Although ENVRIGGER shares its backbone with the policy, it nonetheless performs an additional round of inference per task relative to the baselines, namely observing trajectories, diagnosing weaknesses, synthesizing wrappers, and validating them, whose outputs are subsequently distilled back into the policy via ReasoningBank; consequently, the observed gains may be attributable to this extra same backbone rollout and self distillation effect rather than to the policy conditioned diagnosis that the authors claim as the source of improvement.
Questions for authors
-
The paper rules out distillation from a stronger external model by sharing the backbone between ENVRIGGER and the policy, yet ENVRIGGER itself constitutes an additional round of same backbone inference whose outputs are distilled back into the policy via ReasoningBank, so how do the authors disentangle the contribution of policy conditioned environment reshaping from that of same backbone self distillation under the current setup?
-
The paper attributes the determinism of Stage to seeded reset combined with action replay, but WebArena and SWE bench Verified rely on live browser sessions and containerized test execution with external state, so what is the empirical failure rate of replay in these heavy weight environments, and does the Validate stage ever observe the same Stage producing divergent initial states across rollouts?
-
Since ReasoningBank's distillation is trajectory driven, environments that elicit richer success and failure contrasts, as ENVHARNESS is explicitly designed to do, likely yield larger and higher quality banks under the same acceptance threshold. Could the authors report the per row bank sizes, and where they differ, provide a controlled comparison that fixes bank size across skill sources?
5 comments
W2 & Q1: Policy-conditioned environment reshaping vs. same-backbone self-distillation
A: Thank you for this sharp question. To disentangle the two effects, we added a same-diagnosis baseline. It takes the Original trajectories plus exactly the diagnosis text produced by EnvRigger, and turns them into skills with the same model, extraction prompt, embedding and deduplication, without modifying any environment. In other words, it contains the output of the extra same-backbone inference and its distillation, but not the reshaped environments or the new trajectories they produce. The results are shown below:
| Skill Source | ALFWorld (Avg.) | WebArena (Avg.) | SpreadsheetBench (Pass@1) | SWE-bench Verified (Gemini 3.5 Flash, SR) | SWE-bench Verified (Sonnet 4.6, SR) | OfficeQA (EM) |
|---|---|---|---|---|---|---|
| Same-diagnosis | 64.7 | 38.8 | 48.65 | 49.9 | 71.9 | 55.00 |
| EnvHarness Envs | 68.3 | 41.6 | 49.15 | 52.6 | 72.4 | 56.20 |
EnvHarness outperforms the same-diagnosis baseline across benchmark settings (e.g., 68.3 vs. 64.7 on ALFWorld, and 41.6 vs. 38.8 on WebArena). These results indicate that policy-conditioned environment reshaping provides stronger training signals than distilling diagnoses without interactive environment changes. What the same-diagnosis baseline lacks is new experience: the trajectories the policy produces when acting in the reshaped environments, verified by the original evaluation criteria.
We also evaluated a "blind designer" baseline that reshapes environments from task descriptions alone, without observing policy trajectories. On 100 sampled ALFWorld envs, the blind designer's components passed validation in only 17% of cases, compared with 78% for EnvRigger, indicating that policy-conditioned feedback is essential for generating effective training environments.
Q2: Stage replay determinism in WebArena and SWE-bench Verified
A: Thank you for this practical question. Stage replay starts from a full environment reset before every episode (seeded reset in ALFWorld, a fresh container in SWE-bench Verified, and a snapshot restore of self-hosted websites in WebArena). Empirically, replay reached the exact same initial state in 100% of cases on ALFWorld and SWE-bench Verified, while on WebArena roughly 4% of cases exhibited minor state or timing divergence due to live browser/server dynamics. Even in those ~4% of cases, the environment still represents a valid, solvable task instance evaluated by the original verifier, so it remains usable for training and does not materially affect downstream learning or evaluation. We have added an empirical evaluation of Stage replay determinism across environments to Appendix L.
Q3: Bank size as a confounder: per-row sizes and fixed-size comparison
A: Thank you for the thoughtful question. The full skill-bank sizes are comparable across methods: 208 (Original), 209 (Same-diagnosis), 212 (Fixed wrappers), and 216 (EnvHarness) entries on ALFWorld, and 128, 129, 130, and 131 entries, respectively, on WebArena. The average entry length is also similar between Original and EnvHarness (65.6 vs. 64.7 tokens per entry on ALFWorld; 88.3 vs. 88.1 on WebArena). We further evaluated controlled comparisons across sub-sampled skill-bank sizes (52, 104, 208 entries) on ALFWorld (averaged over 3 random subsamples):
| Bank size | EnvHarness vs. Original | EnvHarness vs. Same-diagnosis | EnvHarness vs. GenEnv |
|---|---|---|---|
| 208 | +2.6 | +3.5 | +8.4 |
| 104 | +1.6 | +2.5 | +5.3 |
| 52 | +1.3 | +4.7 | +7.5 |
At every sub-sampled bank size (52, 104, 208), EnvHarness outperforms Original, the same-diagnosis baseline, and GenEnv, and EnvHarness's own performance stays flat across bank sizes. Therefore, the gain is not attributable to a larger bank size. We have added these results to Appendix J.
We sincerely thank the reviewer again for the time and suggestions. We hope you will consider these clarifications and updated evaluations in assessing our work.
Dear Reviewer QYCL,
Thank you for taking the time to revisit your review. We noticed that the confidence score was updated while the rating remained unchanged, so we wanted to kindly check whether our response and the new experiments (Appendices H, J, and L) have addressed your concerns.
If you have any remaining concerns or further questions, we would be very happy to discuss and provide additional clarification or results.
Thank you again for your time and feedback.
Best regards, The Authors
Thank you very much for your valuable comments and insightful questions. We have revised the manuscript to incorporate your suggestions, and we will address your concerns in detail below:
W1: Novelty beyond the decorator pattern
A: We appreciate the reviewer's perspective and agree that EnvHarness adopts a wrapper-style interface, which we consider a strength for generality. We note that the other reviewers view the contribution as going beyond the interface itself: Reviewer sJgB describes EnvHarness as "a well-motivated and coherent approach to environment scaling" and "an appealing alternative to constructing complete environments from scratch," and Reviewer wErm highlights that Stage is "a principled mechanism for initial state augmentation" that "constructs reachable starting states without directly manipulating simulator internals." Our contribution therefore lies not in the wrapper form itself, but in the following aspects:
[1] Zala, A., Cho, J., Lin, H., Yoon, J., & Bansal, M. (2024). EnvGen: Generating and adapting environments via LLMs for training embodied agents. COLM 2024 (arXiv preprint arXiv:2403.12014).
[2] Qi, Z., Liu, X., Iong, I. L., Lai, H., Sun, X., Zhao, W., ... & Tang, J. (2025). WebRL: Training LLM web agents via self-evolving online curriculum reinforcement learning. ICLR 2025 (arXiv preprint arXiv:2411.02337).
[3] Pan, J., Wang, X., Neubig, G., Jaitly, N., Ji, H., Suhr, A., & Zhang, Y. (2025). Training software engineering agents and verifiers with SWE-Gym. ICML 2025 (arXiv preprint arXiv:2412.21139).
[4] Jain, N., Singh, J., Shetty, M., Zheng, L., Sen, K., & Stoica, I. (2025). R2E-Gym: Procedural environments and hybrid verifiers for scaling open-weights SWE agents. COLM 2025 (arXiv preprint arXiv:2504.07164).
[5] Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., Khandpur, K., Zhang, Y., ... & Yang, D. (2025). SWE-smith: Scaling data for software engineering agents. NeurIPS 2025 (arXiv preprint arXiv:2504.21798).
[6] Tu, D., Hao, H., Yang, H., Chen, Y., Zhang, Y. K., ... & Cai, X. (2026). ScaleEnv: Scaling environment synthesis from scratch for generalist interactive tool-use agent training. ICML 2026 (arXiv preprint arXiv:2602.06820).
[7] Zhou, Y., Yang, Q., Lin, K., Bai, M., Zhou, X., Wang, Y. X., Levine, S., & Li, L. E. (2025). Proposer-Agent-Evaluator (PAE): Autonomous skill discovery for foundation model internet agents. ICML 2025 (arXiv preprint arXiv:2412.13194).
[8] Zhou, Y., Levine, S., Weston, J., Li, X., & Sukhbaatar, S. (2025). Self-challenging language model agents. NeurIPS 2025 (arXiv preprint arXiv:2506.01716).