- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Environment Duels, a framework in which language models both generate executable interactive environments and solve environments created by other models. Each environment includes a privileged hint, allowing the framework to measure solving performance, environment difficulty, and hint lift through matched evaluations with and without hints. Authors then revise their environments based on solver feedback. Experiments involving nine models examine the relationship between solving and authoring capabilities. A subsequent reinforcement learning study trains Qwen3.8-27B on environments from four authors and investigates whether learner-specific hint lift is associated with downstream training gains.
Critical strengths
-
An important research question. The paper examines models not only as problem solvers but also as creators of potentially useful training environments. This perspective is relevant to synthetic data generation and agent training.
-
A well-controlled comparison of hint effects. Using the same environment instances and action budgets in the hinted and unhinted conditions helps isolate the effects of hints. Separately examining attempts that hints rescue or harm also provides useful behavioral insights.
-
Attention to author–learner compatibility. The results suggest that the strongest solver or the author of the hardest environments is not necessarily the best source of training data for a particular learner. This is a practically meaningful observation.
-
An initial connection to downstream utility. Rather than relying solely on benchmark scores, the paper evaluates whether the generated environments improve performance on held-out tasks through reinforcement learning. This is a valuable step toward assessing their actual training utility.
Critical weaknesses
-
The incremental value of hint lift over task difficulty remains unclear.
Hint lift is defined as the difference between hinted and unhinted success rates and is therefore directly related to task difficulty. The current experiments do not establish whether it remains predictive of training gains after controlling for the target learner’s unhinted success rate. Moreover, hint length, information content, and wording can affect hint lift, whereas hints are not used during the actual RL training.
I recommend difficulty-matched comparisons and experiments with standardized hints. More importantly, the paper should directly compare downstream performance when training on environments selected randomly, selected by difficulty, and selected by hint lift.
-
The analysis of environment quality, task coverage, and information isolation is insufficiently systematic.
The paper includes generation constraints, execution checks, and some manual auditing. However, the additional auditing primarily targets instances on which all models fail under at least one hint condition, making it difficult to estimate the overall defect rate. The five broad task domains and the existing similarity analysis also do not adequately establish which skills the environments cover, how frequently underlying mechanisms are repeated, or whether difficulty arises from meaningful reasoning requirements rather than complicated rule descriptions.
I recommend random-sample quality audits, a more detailed analysis of skill and mechanism distributions, and quantitative verification that the anti-leakage constraints are satisfied in practice. Revised environments should also be evaluated using seeds or solver models not involved in providing feedback, to distinguish adaptation to observed instances from genuine generalization.
-
The connection between the complete Environment Duels process and target-model training gains remains underdeveloped.
The RL experiments mainly show that environments from different authors yield different training outcomes for Qwen. They do not establish the individual contributions of duel feedback, environment revision, or hint-lift-based selection. In particular, the paper lacks training comparisons using the same environments before and after revision, as well as budget-matched comparisons against ordinary one-shot environment generation.
Furthermore, being solvable with hints does not necessarily imply that hint-free RL can obtain enough successful trajectories and informative learning signals. Reporting training success rates, the proportion of rollout groups containing only failures, and learning curves would help clarify this connection. The training study also involves only one learner and four authors; additional learners, repeated training runs, and uncertainty estimates are needed to support the robustness of the conclusions.
Questions for authors
see in Critical Weaknesses part
13 comments
W1 (2)
Moreover, hint length, information content, and wording can affect hint lift, whereas hints are not used during the actual RL training.
We thank the reviewer for raising this point. We address information content, hint length and the role of hints in training below.
Hints cannot add missing information. Every environment and hint passes a self-review before admission, and its prompt is in Appendix F of the revision. Its hint check states that a hint "may explain strategy or direct attention to discoverable information, but must not contradict the rules or be the sole source of an otherwise unknowable requirement." A hint that supplies a missing rule is therefore rejected rather than counted as lift. Our response to W2(1) reports an independent audit of the validity of both environments and hints.
Hint length does not explain hint lift. Longer hints do not produce larger hint lift. Across the 90 environments, the number of words in a hint and its hint lift have Spearman ρ = −0.01, for both the original and the revised versions.
Hint lift marks a learnable insight. Hints are a measurement device, not a training input. Hint lift asks whether one piece of strategy, stated without the answer, turns failure into success on the same instance. A high lift therefore marks an environment whose difficulty lies in a learnable insight. During RL the learner has to discover that insight through its own play, and Finding 4 shows that hint lift measured before training is indicative of how much training on those environments improves the learner.
W2 (1)
The analysis of environment quality, task coverage, and information isolation is insufficiently systematic.
The paper includes generation constraints, execution checks, and some manual auditing. However, the additional auditing primarily targets instances on which all models fail under at least one hint condition, making it difficult to estimate the overall defect rate. …
I recommend random-sample quality audits, …
We thank the reviewer for the suggestion. Instead of a random sample, we audited every environment, so the audit measures the overall defect rate rather than the rate on all-fail instances only.
We audited the 88 environments not covered by the audit in Appendix A. The audit uses the same prompts as the verification in Appendix A, which are in Appendix F of the revision. The audit is white-box. Each reviewer receives the source code and the hint, and can execute the environment. To cover models beyond the frontier, we use three reviewers of different families and sizes, Claude-Opus-5, DeepSeek-V4.1-Flash and Qwen3.8-27B.
| Audit outcome | Environments |
|---|---|
| Valid by all three reviewers | 80 (90.9%) |
| Valid by majority (at least two of three) | 88 (100%) |
All three reviewers judge 80 of the 88 environments valid, and two of three judge the remaining 8 valid. We manually checked the 8 environments the models disagree on and found that the reported defects are minor and do not affect solvability. Even if these minor defects are counted, the overall defect rate is at most 8 of 88. Under majority vote, the defect rate is 0 of 88.
We sincerely thank you for your constructive comments. We are encouraged that you found the research question important and the link to downstream training a valuable step. We would like to address your concerns about hint lift beyond difficulty, environment quality and the link to training gains below.
W1 (1)
We thank the reviewer for the suggestion. We first run the selection comparison the reviewer proposes, then provide a regression analysis as supporting evidence.
Selecting by hint lift gives the best training gain. From the 800 environments of our RL experiments, we select three sets of 200 using Qwen3.8-27B's own success with and without the hint. Each set takes 10 environments from every author and domain. The random set samples uniformly. The high-lift set takes environments with relatively high hint lift. The difficulty-matched set samples at random but exactly matches the high-lift set's success without the hint. We train Qwen3.8-27B on each set with the same GRPO setup.
The average weights AIME, ACEBench-Agent, EvalPlus, LiveBench and Reasoning Gym equally. AIME averages AIME 2025 and 2026. Values are mean ± standard deviation over eight samples for AIME and three evaluation runs for the other benchmarks. We can see that training on the high-lift set gives the best average, 1.31 points above the difficulty-matched set and 1.20 above the random set. Matching difficulty alone does not reproduce this gain. This suggests that hint lift carries information beyond difficulty. We add this experiment to Section 4.2 of the revision (Table 4).
Supporting regression analysis. In Table 3, difficulty is measured on the learner itself, as Qwen3.8-27B's failure rate without the hint. The environments of GPT-5.6-Sol and DeepSeek-V4.1-Flash are equally difficult for Qwen3.8-27B (20.0), yet GPT-5.6-Sol's have a much higher hint lift (17.5 vs 5.0) and give a much larger training gain (+2.0 vs +0.1). We regress the training gain on each benchmark (four authors × five benchmarks) on hint lift and difficulty, with benchmark fixed effects and standard errors clustered by author.
The positive hint-lift coefficient is consistent with the selection results above. Given the four-author sample, we treat this regression as supporting evidence.