acceptodds
Under review as a conference paper at ICLR 2027

Hard Enough to Learn: Evolving Verified Tool-Use Environments with Code Agents

Abstract

Large language model (LLM) agents have emerged as a dominant paradigm in artificial intelligence, with their capabilities increasingly determined by their ability to invoke and orchestrate external tools. Reinforcement learning acquires that ability at scale, and it needs verified environments with evolving difficulty to keep the training signal rich. We build a factory that produces verified RLVR data with code agents. An agent authors a domain, tools with implementations, a seeded database, and declarative task templates. Deterministic code then instantiates each task, derives its verified gold tool-call chain. One deterministic predicate validates the gold, measures each task's difficulty, and serves as the reward, so no judge sits in the loop. The factory remains effective across different code agents and harnesses. Per-task difficulty estimates calibrate the corpus to a learnable band and let the code agents harden the saturated tasks. Reinforcement learning on the band-calibrated corpus improves Qwen3.5 tool use across scales. We evaluate these capabilities on GAIA2, WorkBench, -bench, BFCL, Workplace Assistant, and ToolSandbox, and assess capability retention on general benchmarks and IFBench. Training on this corpus outperforms training on randomly sampled task sets of equal and twice its size. The same corpus supports supervision beyond rewards. Supervised fine-tuning develops tool-use capabilities in weak non-Qwen base models from the Llama-3.2 and SmolLM families. We also demonstrate knowledge transfer through on-policy distillation within the generated environments. The 4B student achieves approximately 98% of the average performance of its 9B teacher across the evaluated benchmarks, while the 2B student outperforms a baseline trained with supervised fine-tuning followed by GRPO.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM, RLVR, Agents, Harness, Evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eM46OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses code agents to build stateful tool environments and deterministic programs to generate and verify training tasks. It then selects and revises tasks based on student performance. The generated data improves tool-use performance, with the strongest evidence coming from the 4B results.

Critical strengths
  1. The paper addresses an important bottleneck in tool-use RL. It connects environment code, task generation, reference trajectories, and rewards in a practical workflow that could be useful to other researchers.

  2. The 4B results are a strong part of the paper. Gains across six external evaluations, supported by task-level statistical analysis, show that the generated data has value beyond the training environments.

  3. The failure analysis is useful. Showing that weak models can improve benchmark scores by avoiding actions highlights an evaluation issue that matters beyond this framework.

Critical weaknesses
  1. The chosen difficulty band is not fully justified. Tasks above 50% success can still produce mixed rewards, while the main downstream selection comparison also varies corpus composition and training budget. The results demonstrate the recipe’s usefulness, but provide less evidence that this particular band explains the gains.

  2. Verification is closely tied to a specific reference solution. The checker matches the gold write sequence as well as the final state, which may exclude valid alternatives when independent writes can be reordered. The paper does not clearly establish when this restriction follows from the task’s requirements.

Questions for authors

Please see the weakness part.

2 comments

Thank you for highlighting the 4B transfer results and the failure analysis. Your concerns about the difficulty band and verification restrictions identify two places where our description needs to be more precise.

W1. The difficulty band and its downstream justification

You are correct that tasks above 50% success can still provide group-relative reward signal. For eight independent binary outcomes with success probability p, the probability of a mixed group is

P(mixed) = 1 − p⁸ − (1 − p)⁸.

It is approximately 90.0% at both p=0.25 and p=0.75, and 99.2% at p=0.5. Thus, the upper threshold is an empirical curriculum choice. Our observed band corresponds to one through four successes out of eight; five through seven successes can also be informative. We will correct the overly categorical explanation.

Our training logs also show how quickly this signal can disappear. In the main 4B run, the batch at update 15 already had 61 successful rollouts out of 64 (95.31%). With eight groups of eight, at least five groups therefore had 8/8 successes and zero group-relative reward signal. These observations motivate leaving room for learning below the saturation region. A related empirical choice appears in FrogNano, Section 3.1: its policy-calibrated coding-task pipeline targets p̂=0.5 in iterations 1–4 and 0<p̂≤0.5 in the revised fifth iteration. This supports targeting challenging but learnable tasks, without establishing that our exact band is universally optimal.

We also agree that the original aggregate comparison mixes corpus composition and training budget. We added three random controls, each containing 129 tasks with the same per-domain counts across the same 17 domains, sampled from the 1,354-candidate pool before in-band selection. Initialization, optimization recipe, training seed and number of updates were held fixed.

SelectionWorkBenchWorkplaceBFCL-MT
Random63.66 ± 3.9267.58 ± 2.1944.96 ± 1.71
In-band selection69.98 ± 0.4272.16 ± 1.4148.34 ± 1.55

W2. Valid alternative trajectories and write ordering

The action matcher checks required calls and arguments against unused calls in the trajectory; it does not impose a global call order. Read-only exploration need not reproduce the reference retrieval path. In a real fleet-dispatch task, for example, listing all available vehicles and then selecting class-D vehicles is accepted just as the reference class-filtered lookup is. Both executions receive reward 1.

However, checking the complete reward revealed a genuine limitation. Reversing the five independent maintenance writes passes action/argument matching and produces identical vehicle states, but changes the order of entries in an append-only maintenance log. The exact database hash therefore rejects that trajectory:

ExecutionAction/argument checkDatabase checkReward
Reference executionPassPass1
Alternative lookup, same writesPassPass1
Same writes in reverse orderPassFail: log ordering0

We will correct the phrase “in order” and distinguish order-insensitive action matching from order sensitivity that may remain in state serialization. When log order is not required by the task, rejecting an otherwise valid reordering is a limitation of our current verifier implementation. A suitable remedy is schema-declared order-insensitive comparison for fields where ordering is irrelevant, while retaining genuine workflow dependencies. Alternative write APIs are accepted only when the applicable contract permits them, not automatically because they reach a similar state.

Dear Reviewer eM4621,

As the rebuttal deadline approaches, we would be grateful for your feedback on our responses. We are happy to address any remaining questions.

If you feel that we have addressed your concerns, we would appreciate it if you could consider updating your assessment and score accordingly.

Best regards,
The authors

Reviewer_x36bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Authors introduce a pipeline for generating executable tool-use environments with coding agents. Inside setup, deterministic programs instantiate tasks, derive reference tool-call chains, and provide rewards without an LLM judge. The pipeline uses student success rates to select learnable tasks and make easy tasks more challenging. According to authors claim, their experiments show improvements across six tool-use benchmarks by training Qwen3.5 models from 0.8B to 9B with LoRA , with additional studies of supervised fine-tuning and on-policy distillation.on a small band-calibrated corpus improves six out-of-distribution tool-use benchmarks, while general capabilities are retained.

Critical strengths
  • Authors focus an important and timely problem in agentic training by generating environments that are both executable and challenging enough to support learning. Also the same verifier part makes the method further simpler to adapt in practical scenarios.

  • Their 4B results give convincing evidence of transfer beyond the generated environments, with gains on all six benchmarks whose paired bootstrap intervals exclude zero, which makes the corpus itself a useful contribution even apart from the task-evolution claims.
Critical weaknesses
  • The main comparison does not isolate the benefit of difficulty selection. Table 4 compares in-band corpora that differ in builder, size, and training duration, while the random baselines use only 198 tasks and fewer training steps than some selected runs. Figure 6 does match the budget, but it reports training signals rather than downstream performance. A controlled comparison with the same candidate pool, task count, and training budget, run with and without task evolution, would show whether the gains come from difficulty selection or simply from more data and compute.
  • The corpus is calibrated to the 4B student’s band but reused unchanged for 0.8B, 2B and 9B, contradicting the paper’s own premise that difficulty must be measured against the current student; this likely explains why 4B gains most while 9B gains are mostly within noise and 0.8B regresses on two benchmarks, yet the natural fix, re-calibrating the band per student, is never tried.


  • It is hard for reviewers to tell which model and which corpus each number comes from. For instance, section 4 says the training corpus has 198 tasks over 37 domains, but Section 5.2 says the main 4B model was trained on 129 tasks over 17 domains. Table 13 is presented as a bootstrap of the Table 2 models, yet its 2B gains match the OPD row of Table 12, not the SFT+GRPO model labeled Ours-2B in Table 2, which would give +20.1. It would be nice if the authors could clearly mention somewhere in the manuscript which corpus, training recipe and checkpoint produced each reported result, ideally in a single summary table.

Finally, it would be nice to understand the scaling characteristics of the method more clearly, in particular when it saturates and why. If further experiments across model sizes and compute make this clearer, I would be willing to increase my score.

Questions for authors
  1. Could authors report performance of the 4B model trained on base-corpus in-band tasks only (no hardening) and on the random pool with DAPO-style dynamic sampling at matched gradient steps, and compare with their Table 4?
  2. Adding seed runs for main 4B and 9B results would be helpful.
  3. Can the authors discuss the scaling behaviour of the evolved environments, for example how far the evolution loop can be scaled before the tasks stop being useful, and what causes that limit?
3 comments

Thank you for recognizing the value of our executable environments and the 4B transfer results. Below we address the requested controls, calibration, model identities, continued evolution and additional seed runs as requested in question 2.

W1 / Q1. Matched-budget controls and original tasks without evolution

To address the concern that the gains may reflect differences in data volume or training budget, we compared the 4B model trained on 129 in-band tasks with three random 129-task subsets drawn from the same 1,354-task candidate pool. Each random subset preserves the main corpus's domain quotas across the same 17 domains. All runs used the same base initialization, hyperparameters and budget of 32 optimizer updates. On Workbench, Workplace and BFCL-MT, the selected corpus outperforms random selection at matched task count, domain quotas and optimizer-update budget. Using the same setup, we also trained on the 52 original in-band tasks obtained from the 473-task pool before evolution.

Training corpusWorkBenchWorkplaceBFCL-MT
Base 4B62.459.241.0
Original in-band (52 tasks)67.17 ± 0.6068.75 ± 0.7847.12 ± 1.31
Random (129 tasks)63.66 ± 3.9267.58 ± 2.1944.96 ± 1.71
Random + DAPO61.30 ± 3.1867.49 ± 0.6745.13 ± 1.32
In-band + evolved (129 tasks)69.98 ± 0.4272.16 ± 1.4148.34 ± 1.55

The selected corpus exceeds the random-selection mean by 6.32/4.58/3.38 points on WorkBench, Workplace and BFCL-MT, respectively. Compared with original-pool filtering, evolution plus selection increases the corresponding means by 2.81/3.41/1.22 points. The random controls match the main corpus's size and domain quotas; the original-versus-evolved comparison assesses the complete corpus-building pipeline, including its change in size and composition. For random selection and random+DAPO, SD reflects three corpus draws with one fixed training seed. For the original and evolved in-band corpora, SD reflects three training seeds on each fixed corpus.

We additionally trained with DAPO-style dynamic sampling on the same three random subsets, with the same hyperparameters and 32 updates. The table shows no consistent downstream improvement from DAPO on these benchmarks. Additional sampling was needed on 31/32, 32/32 and 31/32 updates across the three runs. We generated 5.79 groups per retained training group on average (4,443 generated for 768 used), increasing rollout cost despite the matched optimizer-update budget.

W2. Calibration should depend on the student

The main 9B model in the paper uses the 198-task/37-domain corpus selected by 4B.

9B training variantCorpusWorkBenchWorkplaceBFCL-MT
4B in-band corpus198 tasks / 37 domains74.64 ± 1.9874.84 ± 0.8041.04 ± 1.15
9B in-band corpus265 tasks /47 domains73.40 ± 1.8277.19 ± 3.7446.58 ± 2.15

Relative to training on the 4B-calibrated corpus, 9B-specific calibration increases mean Workplace and BFCL-MT scores by 2.35 and 5.54 points, respectively, while reducing mean WorkBench by 1.24 points. The corpora contain 198 tasks and 265 tasks, respectively, so this is a comparison of selection strategies, not a fixed-size intervention. On 2,548 common synthetic tasks with eight measured attempts per model, average pass rates are 87.84% (4B) and 80.79% (9B). The downstream results support useful transfer of the 4B-selected corpus to 9B, while retaining the need for student-relative calibration. We retained the 4B-calibrated 9B model based on its higher GAIA2 macro-average.

For 0.8B, SFT+GRPO improves WB/WP/BFCL-MT (15.0/16.0/6.1 to 27.0/25.9/8.8), but GAIA2 decreases from 1.3 to 1.0 and τ³ from 24.2 to 21.6.

W3. Corpus, recipe and model identities are unclear

ModelTraining recipeTraining corpus
Main 4BGRPO129 tasks / 17 domains
Main 9BGRPO, selection calibrated on 4B198 tasks / 37 domains
9B calibration ablationGRPO, selection calibrated on 9B265 tasks / 47 domains
Main 2BOPD from a trained 4B teacher324 tasks / 49 domains
0.8B (Table 12)SFT warm-up, then GRPO977 SFT traces; 198 RL tasks / 37 domains

The main 4B model uses the 129-task/17-domain corpus from the Claude specification/code-agent pipeline in Table 3. The 217-task/40-domain 4B ablation and the main 9B corpus of 198 tasks/37 domains are different training splits of the same 228-task/45-domain selected pool. The 217-task 4B model reaches 51.5% on BFCL-MT; the 129-task model was retained as main 4B based on its higher GAIA2 score.

The main 2B model and its bootstrap correspond to OPD, as detailed in Table 12, p.18. The prose retained outdated SFT+GRPO gains; we thank you for identifying this and will align the Table 2 results, recipe labels, bootstrap references and discussion with OPD.

W4 / Q3. Continued evolution and its limits

We combined existing task pools from Claude- and DeepSeek-based generation pipelines and recalibrated difficulty against the trained 4B student. We selected 130 in-band tasks and added 25 initially-unsolved but teacher-solvable tasks, obtaining 155 tasks across 72 domains, compared with 129 tasks across 17 domains in the first stage.

BenchmarkMain 4BAfter recalibration + trainingGain (pp)
BFCL-MT47.3850.38+3.00
ToolSandbox49.0050.49+1.49
WorkBench70.3471.16+0.82
Workplace73.7674.28+0.52
GAIA2 Execution33.7532.71-1.04
GAIA2 Search57.0857.50+0.42

In an exploratory extension, we use GigaEvo (Khrulkov et al., 2025), an open-source LLM-driven evolutionary optimization framework. We integrated our method where GigaEvo mutates executable task generators and calibrates them against the current student: the model after the preceding training round. Its fitness gives credit to in-band tasks.

The evaluated checkpoints scored 62.4 → 69.5 → 70.5 → 72.9 → 68.7 → 74.2 on WorkBench. We supplied feedback about abstract task mechanics with benchmark-specific tool and domain names removed. WorkBench, therefore, served as a development benchmark. Improvements beyond the development benchmark support cross-benchmark transfer. For reference, Qwen3.6-27B achieves 77.4 on WorkBench.

BenchmarkBase 4BAfter evolutionGain (pp)
WorkBench62.4074.20+11.80
Workplace59.2071.67+12.47
BFCL-MT41.0047.50+6.50
GAIA2 Execution24.030.21+6.21
GAIA2 Search45.455.63+10.23
GAIA2 Ambiguity9.010.83+1.83

The checkpoint from the fourth round scoring 68.7 on WorkBench was rejected. Nevertheless, the loop can continue improving as task generators adapt to the stronger student.

Dear Reviewer x36b,

As the rebuttal deadline approaches, we would be grateful for your feedback on our responses. We are happy to address any remaining questions.

If you feel that we have addressed your concerns, we would appreciate it if you could consider updating your assessment and score accordingly.

Best regards, The authors

Reviewer_xmyZOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a factory that produces verified RLVR data with code agents. Such verified environments with evolving difficulty is helpful to train AI agents by keeping the training signal rich. Empirical results show significant gain compared to baselines.

Critical strengths

This work propose a concrete, inspectable environment-generation system. The builder creates tool implementations, a stateful database, and declarative task templates.

Critical weaknesses

The paper does not sufficiently distinguish its methodological contribution from established difficulty-aware synthetic-data generation. Prior work already covers instruction hardening, learner-dependent difficulty bands, and verified tool-use synthesis. For example:

  • Zhao et al. “Absolute Zero: Reinforced Self-play Reasoning with Zero Data.” 2025. arXiv:2505.03335. talks about learnability-driven synthesis with executable verification
  • Chen et al. “DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use.” 2026. arXiv:2603.11076. talks about student-success-based filtering specifically for tool use
  • Gao et al. “From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents.” 2026. arXiv:2601.22607.
Questions for authors

it seems you combined selection and rewriting questions (evolution) and then combined with baselines. What is the downstream benefit of evolving tasks and then filtering them, compared with filtering the original, unevolved task pool?

2 comments

Thank you for highlighting this distinction. We will clarify that our contribution is an executable environment factory, rather than difficulty-aware synthesis alone.

W1. Novelty relative to AZR, DIVE and AReaL-SEA

Our contribution is a coding-agent-agnostic, plug-and-play workflow for constructing executable RL environments. It is designed to work with agent-independent interface capable of editing repository files and running the acceptance checks, without relying on agent-specific orchestration. The portable interface consists of domain requirements, artifact contracts and executable tests: the agent authors tools, seeded state and declarative task templates within its existing development workflow. Environment construction therefore does not require users to implement a separate model-API orchestration system. Calibration and RL then consume the resulting executable artifacts.

AZR jointly trains a proposer and solver on program–input–output tasks, using execution-based verification and a learnability reward. DIVE executes real tools and reverse-derives tasks from the resulting traces, emphasizing tool and task diversity. Gao et al. (AReaL-SEA/EigenData) uses verification agents to refine synthesis/evaluation plans and also provides executable per-instance checkers.

The key design is what happens after code generation: deterministic programs instantiate tasks, derive gold tool-call chains by resolving templates against the seeded database, and replay-check those chains before admission. The same verifier scores student rollouts for calibration and RL, with no LLM judge in this loop. Measured difficulty guides rule-level changes while reusing existing tools and databases. Thus the contribution is the reusable artifact contract and verification pipeline, not the choice of model provider or subscription. Its empirical evidence is downstream improvement across six external benchmarks from only 129 selected training tasks; we do not claim a measured reduction in human effort or a direct sample-efficiency advantage over the cited systems.

Q1. Evolution plus filtering versus filtering the original pool

Filtering the original 473-task pool yields 52 in-band tasks; evolution expands the selected training corpus to 129 tasks. We trained on the original subset with the same initialization, hyperparameters and 32 optimizer updates as the main 4B model with 3 seeds:

Training corpusWorkBenchWorkplaceBFCL-MT
Base 4B62.459.241.0
Original in-band (52 tasks)67.17 ± 0.6068.75 ± 0.7847.12 ± 1.31
In-band + evolved (129 tasks)69.98 ± 0.4272.16 ± 1.4148.34 ± 1.55

Evolution plus filtering improves mean WorkBench, Workplace and BFCL-MT scores by 2.81, 3.41 and 1.22 points. This measures the benefit of the full corpus-building pipeline, including increased task count and changed composition. In the individual-run comparison, GAIA2 macro5 increases from 20.8 to 27.0.

Dear Reviewer xmyZ24,

As the rebuttal deadline approaches, we would be grateful for your feedback on our responses. We are happy to address any remaining questions.

If you feel that we have addressed your concerns, we would appreciate it if you could consider updating your assessment and score accordingly.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.