acceptodds
Under review as a conference paper at ICLR 2027

Agent Forest: Controlled Complexity Growth for RL Environments via a Multi-Agent Orchestration System

Abstract

Language models are increasingly expected to act as agents that use tools, follow user constraints, observe changing environment state, and complete long-horizon tasks. Training and evaluating these agents requires large collections of challenging, verifiable tasks, yet constructing such data at scale remains difficult. Existing synthetic task-generation methods frequently rely on single-pass prompting, producing tasks that are limited in complexity, weakly grounded in executable environments, and difficult to expand while preserving consistency. We introduce Agent Forest, a multi-agent framework for controlled complexity growth that systematically transforms trusted seed tasks into more complex descendants while preserving executability and verification guarantees. Agent Forest uses five specialized agents that increase complementary dimensions of interaction complexity: broader tool use, branching dependencies, long-range memory requirements, partial observability, and task composition. Rather than simply rewriting instructions to create superficial difficulty, these agents reason over available tools and environment state to modify the underlying interaction problem itself. Because transformations can accumulate and be applied in different orders, a single seed task can generate multiple descendants with distinct complexity profiles, forming a forest of task variants that grows both the scale and diversity of trusted agentic datasets. We apply Agent Forest to three agentic environments, -Retail, -Airline, and General-Agent, transforming trusted seed tasks into diverse descendants with varying complexity profiles. As interaction complexity accumulates, the resulting tasks become substantially more challenging for strong solver agents. On -Retail, Qwen 3.5 397B drops from 75.4% Pass@1 on the original seed tasks to 54.7% on fully transformed descendants. On -Airline, Gemma 4 31B drops from 66.0% to 28.6%, while GLM-5.2 drops from 72.0% to 53.6%. On General-Agent, Gemma 4 31B drops from 68.8% to 25.6%. Overall, accumulated complexity growth reduces Pass@1 by 18.4–43.2 percentage points relative to the original seed tasks, demonstrating that Agent Forest can systematically generate more complex agentic tasks while preserving executability and validity.

Primary area:
datasets and benchmarks
Keywords:
Agentic AI, Synthetic Data Generation, Agentic Task Generation, Multi-Agent Systems, Datasets and Benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xs6nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper hardens verified agentic tasks by passing them through five ReAct agents (tool expansion, branching, memory, partial observability, and compositionality). Each agent inspects the environment and rewrites both the instruction and the reference solution. A candidate is kept if its reference replays and an LLM judge accepts it; otherwise a repair loop tries to fix it. On τ²-Retail, τ²-Airline, and General-Agent, 212 seeds yield 823 descendants, and solver Pass@1 drops by 18–43 points after all five stages, including on lineages that survive every stage.

Critical strengths
  1. Grounding the transformations in the environment, rather than rewriting the prompt, is the right idea, and editing the reference solution together with the instruction keeps the output checkable. The recover-before-hide rule for partial observability is a nice touch.

  2. The complete-lineage results are convincing on difficulty. With the task set held fixed, Pass@1 still falls from 77.4% to 54.7% on τ²-Retail and from 71.8% to 25.6% on General-Agent.

Critical weaknesses
  1. My main concern is that a lower Pass@1 does not show the tasks are still valid; ambiguous or unsolvable tasks would produce the same drop. Replay only checks that the reference runs, the LLM judge is never validated, and in τ² the user simulator must still be able to supply whatever information was hidden. The General-Agent numbers also look odd: solvers pass 68.8% of seeds with 8.56 writes on average, while the references use 21.41. A human audit of sampled descendants, or at least an error analysis separating task defects from agent mistakes, is needed to support "preserving validity."

  2. Several framing claims are untested. The title targets RL environments, but there are no training experiments. Only one fixed agent order is run, so the "forest" is really a chain, the 823 descendants are nested versions of the same seeds, and diversity is never measured. There is also no comparison with plain instruction rewriting under the same validation, which is the baseline the paper argues against.


Post-rebuttal update

Thanks for the additional experiments. The cross-domain RL gains and the results with more solvers and a different agent order address several of my concerns. My main concern remains task validity: the new results leave open how often failures come from task ambiguity or defects. I’m keeping my rating at 2.

Questions for authors
  1. How is task success determined in each environment? On General-Agent seeds, reference solutions average 21.41 writes (Table 1), yet solvers pass 68.8% of tasks with 8.56 writes on average (Table 4), which suggests the success check is much looser than the reference.
  2. Can you report a human validation of sampled descendants per stage (solvable, unambiguous, reference correct), together with the LLM judge's agreement with it?
  3. How does a non-grounded rewriting baseline perform under the same validation and repair budget?
  4. On τ²-Retail, Qwen 3.5 397B is both the generator and the only solver. Does the drop hold for other solvers?

Suggestion for future work: training experiments (SFT or RL) on the generated tasks would directly support the RL framing.

4 comments
AuthorsOpenReview ↗

Response to Reviewer xs6n (Part 1)

We thank the reviewer for the detailed review and for recognizing that grounding the transformations in the environment is the right approach, that editing the reference solution together with the instruction keeps the output checkable (S1), and that the complete-lineage results are convincing on difficulty (S2). We report three new results and then address each weakness and question.

  1. RL training. GRPO training of Qwen 3.5 9B on General-Agent descendants improves Pass@1 on τ²-Retail and τ²-Airline, which were held out from training (W2).
  2. Additional solvers. GLM-5.2 and Qwen 3.5 9B on τ²-Retail, and Qwen 3.5 9B on τ²-Airline. Each τ² domain is now evaluated with three base solvers and one RL-trained solver (W1, Q4).
  3. Alternative agent order on τ²-Retail (W2).

W1. Validity of the transformed tasks

We agree that a lower Pass@1 alone does not establish validity. The validity of Agent Forest tasks rests on the four points below. None of them depends on the LLM judge, which acts as an additional filter that rejects descendants whose instruction, trajectory, and final state disagree.

1. Replay fixes a reachable target. On τ², a task is solved when the final database state matches the state produced by replaying the reference solution of that stage (Q1). Replay therefore does more than check that the reference runs. It defines the evaluation target, and every accepted descendant has an action sequence that reaches this target from a fresh environment state.

2. The transformation rules exclude unsolvable and artificial difficulty (Appendix B).

  • Partial Observability hides a value only after calling the recovery tool and observing the exact value in a tool response. It never hides a user-defined criterion (target category, requested amount, destination, threshold), never replaces a value with an undefined placeholder, and requires that the recovered value be used in a later action.
  • Tool Expansion and Branching may not invent writes to increase difficulty. Memory may not add "remember this" wording without changing the solution. Compositionality may not count duplicate reads or padding as composition.

3. The τ² user simulator does not need to supply hidden values. The simulator receives the instruction of the current stage. Partial Observability removes a concrete value from that instruction while keeping the user's intent ("My reservation is RES-123, please modify it" becomes "Please modify my current active reservation"). The value must be uniquely recoverable from the environment through a non-circular tool path (here, by looking up the user's reservations). The agent recovers it with tools and, after Memory, retains it until it is used.

4. Solver behavior matches tasks that are harder but solvable.

Table A. Complete-lineage Pass@1 (%), seed versus stage 5.

SolverRetail seedRetail stage 5Airline seedAirline stage 5
GLM-5.279.258.578.653.6
Qwen 3.5 397B77.454.7--
Gemma 4 31B--75.028.6
Qwen 3.5 9B + GRPO75.541.575.021.4
Qwen 3.5 9B66.037.764.321.4

If the transformations made tasks ambiguous or unsolvable, strong and weak solvers would fail those tasks alike, and their scores would move closer together. Table A shows the opposite. On τ²-Retail, the gap between the strongest solver (GLM-5.2) and the weakest (Qwen 3.5 9B) grows from 13.2 points on seed tasks to 20.8 points at stage 5, and the four solvers keep the same ranking. On τ²-Airline, the same gap grows from 14.3 to 32.2 points. Stronger solvers handle the added difficulty better, which is what we expect from tasks that are harder but solvable. GLM-5.2 still solves 58.5% (Retail) and 53.6% (Airline) of the fully transformed tasks. Across 11 configurations (five solvers, three environments, two agent orders), Pass@1 at every transformed stage is lower than at the seed stage, in both the all-accepted and the complete-lineage views.

AuthorsOpenReview ↗

Response to Reviewer xs6n (Part 2)

W2. Framing claims

Training on Agent Forest tasks (new result)

We trained Qwen 3.5 9B with GRPO on the 211 General-Agent descendants (8 rollouts per task, reward 1 if the episode passes the task verification and 0 otherwise) using a ReAct harness. We evaluated the model with the same harness on τ²-Retail and τ²-Airline, which were not used in training, with GPT-OSS 120B as the user simulator. This experiment addresses the RL framing and the reviewer's suggestion.

Table B. Pass@1 (%) of Qwen 3.5 9B before and after GRPO, all accepted tasks.

StageRetail beforeRetail afterΔAirline beforeAirline afterΔ
0 Seed64.071.1+7.160.072.0+12.0
1 +Tool Expansion58.364.8+6.552.663.2+10.6
2 +Branching55.461.4+6.054.162.2+8.1
3 +Memory52.257.6+5.445.551.5+6.0
4 +Partial Observability48.352.8+4.548.554.5+6.0
5 +Compositionality37.741.5+3.821.421.40.0
All stages54.460.1+5.748.956.6+7.7
  • Training transfers to held-out domains. Summed over all stages, the trained model solves 32 more tasks on Retail (335 versus 303 of 557) and 17 more on Airline (124 versus 107 of 219).
  • The stages show how far training transfers. The gains are largest on seed tasks and decrease as complexity accumulates. After training, stage-5 accuracy is 41.5% on Retail and 21.4% on Airline, so the fully transformed tasks remain open targets for further training.

Instruction-rewriting baseline

As the reviewer notes in S1, grounding the transformations in the environment, rather than rewriting the prompt, is the central design choice. A rewriting baseline without environment access must guess the database values that a new reference solution uses (order IDs, reservation IDs, amounts), and replay rejects candidates whose values do not exist or whose actions fail. TaskCraft reports this effect in a related setting: generating tasks from tool context raises the validation pass rate from 18.5% to 43.0% compared with direct LLM prompting (Shi et al., ICLR 2026, Table 4). Partial Observability cannot be performed by rewriting at all, because recover-before-hide requires calling the recovery tool and observing the value before it is hidden.

AuthorsOpenReview ↗

Response to Reviewer xs6n (Part 3)

W2 (continued). Agent order and the forest structure (new result)

The paper fixes the order to attribute the change between consecutive stages to one transformation while the task population stays fixed. The framework does not require a fixed order. We ran a second generation on 53 seeds sampled from the 114 τ²-Retail seeds, with the order Branching → Memory → Tool Expansion → Partial Observability → Compositionality. All other settings are unchanged.

Table C. Yield and reference-solution statistics under the alternative order.

LevelAgentInputAcceptedTool callsWritesDistinct tools
0Seed53534.431.403.08
1+Branching53377.031.355.08
2+Memory37348.471.385.59
3+Tool Expansion343211.311.446.84
4+Partial Observability323011.201.476.90
5+Compositionality302212.552.277.77

Table D. Pass@1 (%) under the alternative order. AA: all accepted tasks. CL: 22 complete lineages.

LevelAgentGLM-5.2 AAGLM-5.2 CLQwen 3.5 9B AAQwen 3.5 9B CL
0Seed81.186.464.268.2
1+Branching78.481.859.559.1
2+Memory76.577.355.954.5
3+Tool Expansion71.972.750.050.0
4+Partial Observability70.068.246.745.5
5+Compositionality59.159.136.436.4

The two runs show the forest structure.

  1. The framework runs under a different order without modification. It produced 155 verified descendants from 53 seeds, and Pass@1 decreases at every level for both solvers in both views.
  2. The path determines which descendants are generated. Branching applied to seeds discards 16 of 53 candidates (30%), while Branching applied after Tool Expansion discards 7 of 108 (6.5%). Tool Expansion adds observations on which a branch can condition, so the order of transformations changes the set of tasks that can be built. The first step also changes tasks differently: Tool Expansion first adds 3.79 tool calls and 2.35 distinct tools per task, while Branching first adds 2.60 tool calls and 2.00 distinct tools.
  3. Accumulated difficulty is reproducible across paths. At stage 5, GLM-5.2 reaches 59.1% under the new order and 58.5% under the original order. Qwen 3.5 9B reaches 36.4% and 37.7%.
  4. The effect of each transformation persists under reordering. Compositionality adds 0.80 writes per task in both orders (1.65 to 2.45, and 1.47 to 2.27). Partial Observability leaves reference length unchanged in both orders (11.49 to 11.54, and 11.31 to 11.20). Nested descendants. Nesting is part of the design. Because each stage extends the previous one, the complete-lineage evaluation can hold the task set fixed and attribute each change in Pass@1 to one transformation, which is the comparison the reviewer finds convincing (S2). Each stage is a different task with its own reference solution. On τ²-Retail, reference solutions grow from 4.82 to 12.62 tool calls and from 3.34 to 7.15 distinct tools per task (paper, Table 1), and the alternative order grows the same seeds into different intermediate tasks.

Questions

Q1. Task success. On τ²-Retail and τ²-Airline, a task is solved when the final database state matches the state produced by replaying the reference solution of that stage.

Q2. See W1.

Q3. See W2 (instruction-rewriting baseline).

Q4. Other solvers on Retail. Table E reports two solvers that took no part in task generation. Pass@1 decreases at every stage for both. From seed to stage 5, GLM-5.2 drops by 20.4 points (all accepted) and 20.7 points (complete lineages), close to the generator Qwen 3.5 397B (20.7 and 22.7 points, paper Table 2). Qwen 3.5 9B drops by 26.3 and 28.3 points. The decrease also holds on τ²-Airline for all four solvers in Table A.

Table E. Pass@1 (%) on τ²-Retail. AA: all accepted tasks. CL: 53 complete lineages.

StageAgentGLM-5.2 AAGLM-5.2 CLQwen 3.5 9B AAQwen 3.5 9B CL
0Seed78.979.264.066.0
1+Tool Expansion75.075.558.358.5
2+Branching72.373.655.456.6
3+Memory70.771.752.254.7
4+Partial Observability67.469.848.350.9
5+Compositionality58.558.537.737.7

Suggestion (training experiments). See W2, Table B.

We thank the reviewer again and hope these results address the concerns raised.

Thank you for your updated response. We are glad that the cross-domain RL gains, the additional solvers, and the second agent order address several of your concerns. We will take your remaining point on task validity into account in the revision.

Reviewer_Ye6LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Agent Forest, a pipeline that makes existing agentic tasks harder. It works in two parts.

Method. Five ReAct-style LLM agents (Tool Expansion, Branching, Memory, Partial Observability, Compositionality) rewrite each seed task. Each agent updates both the instruction and the reference solution, and inspects the environment's tools and state while doing so. Each candidate is validated by:

  • replaying its reference solution in the environment, and
  • an LLM judge.

Candidates that fail go through a repair loop.

Evaluation. The agents are applied in a fixed order to τ²-Retail, τ²-Airline, and General-Agent, growing 212 seeds into 823 accepted tasks. Pass@1 of 1–2 solver models drops by 18.4–43.2 points from seed to final stage, on both all accepted tasks and lineages that survive every stage.

Critical strengths

S1. The transformations are grounded in the environment, not just the text. Each agent queries the environment before changing anything. It then updates the instruction and the reference solution together, and the new solution is replayed in a fresh environment. This matters because it rules out the most common failure of LLM-rewritten tasks: a reference solution that cannot actually be executed.

S2. The evaluation controls for lineage filtering. The Complete-Lineage setting fixes the set of tasks that survive all five stages and still shows a consistent drop. So the drop in Pass@1 is not just because easier lineages were filtered out along the way.

Critical weaknesses

W1. Solvability and discriminability are not validated. The drop in Pass@1 may simply mean the tasks became ambiguous or unsolvable, since validation relies only on reference replay and an LLM judge, with no human audit. With only 1–2 solvers per environment and single-run results, there is also no evidence that the tasks discriminate between models.

W2. Limited novelty and no training experiments. Seed-based hardening of agentic tasks already exists (TASTE on τ²-bench, TaskCraft, TRACE, and General-Agent's own difficulty tiers), and the paper compares against none of these or any rewriting baseline. Despite the "for RL environments" framing, there is no training experiment, so the result is only a harder eval set.

Questions for authors

Clarification

  1. What fraction of Stage-5 failures are task errors rather than agent errors, based on a human audit?
  2. What is the Stage-5 pass@k of a frontier model?
  3. After Partial Observability and Memory, how is the τ² user simulator's instruction updated, and can the hidden information still be obtained?
  4. On Retail, Qwen 3.5 397B is both the generator and the solver. How do other solvers perform there?
  5. Why does Pass@1 increase after Branching, and after Compositionality on General-Agent? What is the variance across runs, given n=28 on Airline?
  6. How does Agent Forest differ from TASTE and from General-Agent's built-in task evolution?

Suggestion

  1. Train on the generated tasks and report held-out gains.
  2. Evaluate other agent orderings to support the "forest" claim.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Ye6L (Part 1)

We thank the reviewer for the careful review and for recognizing that environment grounding rules out non-executable reference solutions (S1) and that the complete-lineage evaluation controls for lineage filtering (S2). We report three new results and then address each weakness and question.

  1. RL training. GRPO training of Qwen 3.5 9B on General-Agent descendants improves Pass@1 on τ²-Retail and τ²-Airline, which were held out from training (W2, Q7).
  2. Additional solvers. GLM-5.2 and Qwen 3.5 9B on τ²-Retail, and Qwen 3.5 9B on τ²-Airline. Each τ² domain is now evaluated with three base solvers and one RL-trained solver (W1, Q4).
  3. Alternative agent order on τ²-Retail (Q8).

W1. Solvability and discriminability

Every fully transformed task has a valid solution. Each accepted descendant has a reference solution that replays from a fresh environment state and reaches the target state, and the semantic judge confirms that the instruction, trajectory, and final state agree. Partial Observability may hide only values that are uniquely recoverable from the environment through a non-circular tool path, and it may not hide values that define the user's intent (Appendix B and Q3).

The tasks separate models, and the separation grows with complexity.

Table A. Complete-lineage Pass@1 (%), seed versus stage 5.

SolverRetail seedRetail stage 5Airline seedAirline stage 5
GLM-5.279.258.578.653.6
Qwen 3.5 397B77.454.7--
Gemma 4 31B--75.028.6
Qwen 3.5 9B + GRPO75.541.575.021.4
Qwen 3.5 9B66.037.764.321.4

If the transformations made tasks ambiguous or unsolvable, strong and weak solvers would fail those tasks alike, and their scores would move closer together. Table A shows the opposite. On τ²-Retail, the gap between the strongest solver (GLM-5.2) and the weakest (Qwen 3.5 9B) grows from 13.2 points on seed tasks to 20.8 points at stage 5, and the four solvers keep the same ranking. On τ²-Airline, the same gap grows from 14.3 to 32.2 points. Stronger solvers handle the added difficulty better, which is what we expect from tasks that are harder but solvable. GLM-5.2 still solves 58.5% (Retail) and 53.6% (Airline) of the fully transformed tasks.

The decrease is consistent across solvers and settings. Across 11 configurations (five solvers, three environments, two agent orders), Pass@1 at every transformed stage is lower than at the seed stage, in both the all-accepted and the complete-lineage views. On τ²-Airline (n = 28), all four solvers drop from seed to stage 5, by 25.0 to 53.6 points.

W2. Training experiments and novelty

Training on Agent Forest tasks (new result)

We trained Qwen 3.5 9B with GRPO on the 211 General-Agent descendants (8 rollouts per task, reward 1 if the episode passes the task verification and 0 otherwise) using a ReAct harness. We evaluated the model with the same harness on τ²-Retail and τ²-Airline, which were not used in training, with GPT-OSS 120B as the user simulator.

Table B. Pass@1 (%) of Qwen 3.5 9B before and after GRPO, all accepted tasks.

StageRetail beforeRetail afterΔAirline beforeAirline afterΔ
0 Seed64.071.1+7.160.072.0+12.0
1 +Tool Expansion58.364.8+6.552.663.2+10.6
2 +Branching55.461.4+6.054.162.2+8.1
3 +Memory52.257.6+5.445.551.5+6.0
4 +Partial Observability48.352.8+4.548.554.5+6.0
5 +Compositionality37.741.5+3.821.421.40.0
All stages54.460.1+5.748.956.6+7.7
  • Training transfers to held-out domains. Summed over all stages, the trained model solves 32 more tasks on Retail (335 versus 303 of 557) and 17 more on Airline (124 versus 107 of 219).
  • The stages show how far training transfers. The gains are largest on seed tasks and decrease as complexity accumulates. After training, stage-5 accuracy is 41.5% on Retail and 21.4% on Airline, so the fully transformed tasks remain open targets for further training.
AuthorsOpenReview ↗

Response to Reviewer Ye6L (Part 2)

Comparison with related methods

As the reviewer notes in S1, Agent Forest changes tasks through environment interaction and replays every reference solution. The methods below differ in what they transform and how.

  • TASTE (Keren et al., 2026, arXiv:2605.28556) samples new tool sequences with an n-gram model trained on LLM validity judgments, selects representative sequences by clustering, and instantiates each as a new τ² task. It then evolves each task by adding decoy database records, policy pressure, and less cooperative user behavior, while keeping the gold tool sequence unchanged. Its goal is coverage of tool combinations for benchmark construction. Agent Forest instead transforms existing verified tasks, and each transformation changes the reference solution itself (additional tool calls, conditional actions, retained values, hidden values, dependent subtasks). Transformations accumulate along recorded lineages, and the descendants are also used for RL training. The two methods are complementary: TASTE's interaction-level evolution could be applied to Agent Forest descendants.
  • TaskCraft (Shi et al., ICLR 2026) builds question-answering tasks for information-retrieval tools. Depth-based extension hides an entity behind an additional tool call, width-based extension merges independent questions, and extended tasks are validated by linguistic analysis without execution. Agent Forest operates in stateful environments, replays every descendant, adds Branching and Memory, and requires dependencies between composed subtasks.
  • TRACE (Guo et al., 2025, arXiv:2510.00415) evolves GAIA and AIME seed tasks along directions that an LLM proposes through free exploration, and validates them by reproducing their trajectories. Agent Forest targets environments with database writes and a simulated user, and each step adds one specified dimension.
  • General-Agent tiers (Senghaas, 2026). A synthesizer writes a seed task and grows it through tiers t0 to t4 by extending the instruction, database, schema, and tools. A tier is accepted when the pass rate of a gating solver falls in a target band. Agent Forest keeps the tools and schema fixed, so it applies to existing benchmarks such as τ² whose environment cannot change. Its acceptance depends on replay and semantic validity, not on one solver's pass rate, and each step records the dimension it adds. Our General-Agent seeds come from the highest tier (t4), so they have already passed through General-Agent's own evolution. Agent Forest still reduces Gemma 4 31B's Pass@1 on them from 71.8% to 25.6% on the complete lineages. Rewriting baseline. A rewriting baseline without environment access cannot write reference solutions with valid database arguments, which is the failure the reviewer describes in S1. TaskCraft reports the same effect: generating tasks from tool context raises the validation pass rate from 18.5% to 43.0% compared with direct LLM prompting (Shi et al., Table 4). We will add TASTE, TRACE, and the General-Agent tiers to the related work.

Questions

Q1, Q2. See W1.

Q3. User simulator after Memory and Partial Observability. The τ² user simulator receives the instruction of the current stage. Memory restructures the scenario so that a required fact is obtained early and must be retained across later interactions before it is used. Partial Observability is applied on top of this. It removes a concrete value from the simulator instruction while keeping the user's intent ("My reservation is RES-123, please modify it" becomes "Please modify my current active reservation"). A value may be hidden only if it is uniquely recoverable from the environment through a non-circular tool path (here, by looking up the user's reservations). Values that express the user's intent are never hidden. GPT-OSS 120B (user simulator) receives only the transformed instruction, so it cannot reveal the hidden value. The agent must recover it with tools and, after Memory, retain it until it is used.

AuthorsOpenReview ↗

Response to Reviewer Ye6L (Part 3)

Questions (continued)

Q4. Other solvers on Retail. Table C reports two solvers that took no part in task generation. Pass@1 decreases at every stage for both. From seed to stage 5, GLM-5.2 drops by 20.4 points (all accepted) and 20.7 points (complete lineages), close to the generator Qwen 3.5 397B (20.7 and 22.7 points, paper Table 2). Qwen 3.5 9B drops by 26.3 and 28.3 points.

Table C. Pass@1 (%) on τ²-Retail. AA: all accepted tasks. CL: 53 complete lineages.

StageAgentGLM-5.2 AAGLM-5.2 CLQwen 3.5 9B AAQwen 3.5 9B CL
0Seed78.979.264.066.0
1+Tool Expansion75.075.558.358.5
2+Branching72.373.655.456.6
3+Memory70.771.752.254.7
4+Partial Observability67.469.848.350.9
5+Compositionality58.558.537.737.7

Q5. Increases after Branching and after Compositionality on General-Agent. Branching sometimes states a step of the task more explicitly (the condition and the resulting action), which can make that step easier for some solvers. These increases are one or two tasks on the complete lineages (Retail with Qwen 3.5 397B: 36 to 38 of 53; Airline with Gemma: 18 to 20 of 28), and they depend on the solver. On Retail, Branching lowers Pass@1 for GLM-5.2 and Qwen 3.5 9B, and when applied first (Q8) it lowers Pass@1 for both solvers. On General-Agent, Compositionality solves the same 10 tasks in the all-accepted view (10/41 to 10/39), and the rise comes from the smaller denominator. On the complete lineages it is one task (9 to 10 of 39). Relative to the seed, every transformed stage has lower Pass@1 in all 11 configurations, so the tasks remain harder than their seeds at every stage. For Airline (n = 28), see W1.

Q6. See W2.

Q7. See W2, Table B.

Q8. Other agent orderings. The paper fixes the order to attribute the change between consecutive stages to one transformation while the task population stays fixed. The framework does not require a fixed order. We ran a second generation on 53 seeds sampled at random from the 114 τ²-Retail seeds, with the order Branching → Memory → Tool Expansion → Partial Observability → Compositionality. All other settings are unchanged.

Table D. Yield and reference-solution statistics under the alternative order.

LevelAgentInputAcceptedTool callsWritesDistinct tools
0Seed53534.431.403.08
1+Branching53377.031.355.08
2+Memory37348.471.385.59
3+Tool Expansion343211.311.446.84
4+Partial Observability323011.201.476.90
5+Compositionality302212.552.277.77

Table E. Pass@1 (%) under the alternative order. AA: all accepted tasks. CL: 22 complete lineages.

LevelAgentGLM-5.2 AAGLM-5.2 CLQwen 3.5 9B AAQwen 3.5 9B CL
0Seed81.186.464.268.2
1+Branching78.481.859.559.1
2+Memory76.577.355.954.5
3+Tool Expansion71.972.750.050.0
4+Partial Observability70.068.246.745.5
5+Compositionality59.159.136.436.4

The two runs show the forest structure.

  1. The framework runs under a different order without modification. It produced 155 verified descendants from 53 seeds, and Pass@1 decreases at every level for both solvers in both views.
  2. The path determines which descendants are generated. Branching applied to seeds discards 16 of 53 candidates (30%), while Branching applied after Tool Expansion discards 7 of 108 (6.5%). Tool Expansion adds observations on which a branch can condition, so the order of transformations changes the set of tasks that can be built. The first step also changes tasks differently: Tool Expansion first adds 3.79 tool calls and 2.35 distinct tools per task, while Branching first adds 2.60 tool calls and 2.00 distinct tools.
  3. Accumulated difficulty is reproducible across paths. At stage 5, GLM-5.2 reaches 59.1% under the new order and 58.5% under the original order. Qwen 3.5 9B reaches 36.4% and 37.7%.
  4. The effect of each transformation persists under reordering. Compositionality adds 0.80 writes per task in both orders (1.65 to 2.45, and 1.47 to 2.27). Partial Observability leaves reference length unchanged in both orders (11.49 to 11.54, and 11.31 to 11.20). The same seed pool therefore grows along different paths into different tasks, while the complexity added by each path, measured by solver success, is stable.

We thank the reviewer again and hope these results address the concerns raised.

Reviewer_32AtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This study presents Agent Forest, a framework designed to automatically synthesize complex agentic tasks from established seed tasks. The methodology utilizes five specialized LLM-driven agents to introduce complexity across distinct dimensions: Tool Expansion, Branching Dependencies, Memory, Partial Observability, and Compositionality. During each transformation, the framework updates both the task instructions and their corresponding reference solutions, subsequently validating them through execution and LLM-based semantic checks. Additionally, an integrated repair mechanism is employed to repair any failed transformations.

Critical strengths

S1. Authors target an important problem. Automatically generating diverse and challenging tasks is a key component in building and evaluating LLM agents. A practical workaround is to take established seed tasks and simply modify them to generate synthetic ones.

S2. Decomposing the different aspects of task complexity is an intuitive approach to control different aspects of the generated tasks. This could be a good contribution if authors had provided evidence and appropriate evaluations to show improvements on each of these individual aspects.

Critical weaknesses

W1. I think the contributions of this work is very limited. The ReAct-syle reasoning (Yao et al. 2022) is used with 5 different prompts focusing on different aspects of task complexity in different orders. There is no fundamental contribution beyond this. These agents are applied sequentially, followed by execution-based validation and repair. While the overall pipeline may be practically useful, it is unclear what fundamentally new methodological contribution is introduced beyond combining existing techniques. The authors should better clarify the novelty of their approach and demonstrate why the proposed orchestration framework offers advantages over simpler alternatives.

W2. The evaluations are very limited. Since the evaluation pairs different solver models with varying environments, it is unclear whether the observed results truly generalize. Moreover,, relying primarily on reduced Pass@1 scores to claim increased complexity is problematic; the authors fail to distinguish whether these score drops stem from genuinely harder reasoning demands or merely longer trajectories and extra subtasks. The study also lacks statistical significance testing and an assessment of the generated tasks' diversity and quality. Ultimately, more rigorous experiments are essential to validate the proposed method's effectiveness.

W3. In the introduction and method section authors emphasize on different ordering of the agents working on seed tasks to make the complexity dimensions diverse, but then in the Experimental Setup section they mention that the ordering of the agent is fixed! This is clearly contradicting the earlier sections.

W4. There is no comparison to any existing baseline. There are other existing methods such as TaskCraft [1] for generating more complex tasks for agents, but no comparison exists to these methods other than citing them in the Related Work section. Without comparison, it is not clear if the proposed pipeline has any contribution beyond the existing methods.

W5. Choices in the paper are not justified. Authors use 5 different agents for different aspects of complexity in different orders. Why 5 agents? Since the agents are only different in their prompts given in the appendix, it is not clear if two or more of these prompts could be combined to achieve the same task complexity.

W6. Beyond tool calls, what measurable criteria establish that a generated task has more memory, branching, observability, or compositional complexity than its parent? There are no metrics or evidence provided for these claimed complexities.

W7. The paper provides high-level pseudocode and the instructions used for the five agents in the appendix. However, important implementation and evaluation details are missing, making it difficult to reproduce the reported results. Furthermore, the authors do not provide concrete examples of generated tasks, their corresponding reference solutions, or rejected tasks and the reasons for their rejection. These examples are important for understanding how the proposed transformations increase task complexity and for assessing whether the generated tasks are genuinely valid and executable. The authors should provide representative examples of accepted and rejected transformations, along with more detailed descriptions of the evaluation and validation procedures.

[1] Shi, D., Cao, J., Chen, Q., Sun, W., Li, W., Lu, H., ... & Zhou, W. (2026, April). Taskcraft: Automated generation of agentic tasks. In International Conference on Learning Representations (Vol. 2026, pp. 43714-43734).

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 32At (Part 1)

Thank you for the detailed review. We appreciate that you agree that the automatic generation of challenging agentic tasks is an important problem (S1) and that decomposing complexity into separate dimensions is an intuitive way to control generated tasks (S2). We address each weakness below and report three sets of new results:

  1. RL training on Agent Forest tasks. GRPO training of Qwen 3.5 9B on General-Agent descendants improves Pass@1 on τ²-Retail and τ²-Airline, two domains that were not used in training (Table R1).
  2. Additional solvers. GLM-5.2 on τ²-Retail, and Qwen 3.5 9B on τ²-Retail and τ²-Airline. The evaluation now covers 11 configurations of environment, solver, and agent order (Table R2).
  3. Alternative agent order. A second generation run on τ²-Retail with the order Branching → Memory → Tool Expansion → Partial Observability → Compositionality (Tables R4 and R5).

W1. Contribution beyond ReAct and prompts

Claim 1: Agent Forest changes the interaction problem and guarantees that every descendant remains executable.

In τ² and General-Agent, a task consists of an instruction, a set of tools, a database, and a reference solution whose replay produces the target database state. A valid descendant must call tools with arguments that exist in the database (order IDs, flight numbers, amounts), and its replay must reach a state consistent with the new instruction. A generator without environment access cannot know these values. Agent Forest agents query the environment before editing, write the new reference solution from observed tool outputs, and accept a descendant only after the reference solution replays from a fresh state and passes the semantic check.

Partial Observability shows why environment access is required. The agent may hide a value only after it has called the recovery tool and observed that value in the tool output (Appendix B, "recover before hide"). This condition cannot be enforced by text rewriting.

Claim 2: Verified descendants serve as RL environments, and training on them transfers to unseen domains (new result).

Each descendant has a replay-verified reference solution, so task success defines a binary reward without human labels. We trained Qwen 3.5 9B with GRPO on the 211 General-Agent descendants (8 rollouts per task, reward 1 if the episode passes the task verification and 0 otherwise) using a ReAct harness. We then evaluated the model with the same harness on τ²-Retail and τ²-Airline, which were not used in training. GPT-OSS 120B serves as the user simulator.

Table R1. Pass@1 (%) of Qwen 3.5 9B before and after GRPO, all accepted tasks.

StageRetail beforeRetail afterΔAirline beforeAirline afterΔ
0 Seed64.071.1+7.160.072.0+12.0
1 +Tool Expansion58.364.8+6.552.663.2+10.6
2 +Branching55.461.4+6.054.162.2+8.1
3 +Memory52.257.6+5.445.551.5+6.0
4 +Partial Observability48.352.8+4.548.554.5+6.0
5 +Compositionality37.741.5+3.821.421.40.0
All stages54.460.1+5.748.956.6+7.7

Two findings follow.

  • Training transfers across domains. Training on General-Agent descendants improves the model on two τ² domains it never saw. Summed over all stages, the trained model solves 32 more tasks on Retail (335 versus 303 of 557) and 17 more on Airline (124 versus 107 of 219).
  • The stages show how far training transfers. The gains are largest on seed tasks (+7.1 points on Retail, +12.0 on Airline) and decrease as complexity accumulates. Agent Forest therefore does more than report whether a model improved: it shows at which level of interaction complexity the improvement holds. After training, stage-5 accuracy is 41.5% on Retail and 21.4% on Airline, so the fully transformed tasks remain open targets for further training. The contribution is the formulation of agentic data generation as controlled, environment-grounded complexity growth over verified task lineages, the five transformation operators with their validity constraints, and the evidence that the resulting tasks are harder for solvers and usable for RL training. ReAct and execution replay are existing components. Agent Forest specifies how to combine them so that complexity grows along controlled dimensions while every task stays executable.
AuthorsOpenReview ↗

Response to Reviewer 32At (Part 2)

W2. Evaluation scope

W2(a). The results generalize across solvers, environments, and agent orders.

We added GLM-5.2 on τ²-Retail, and Qwen 3.5 9B on τ²-Retail and τ²-Airline. GLM-5.2 and Qwen 3.5 9B are now evaluated on both τ² domains. Table R2 reports all 11 configurations on the complete lineages, where the task population is fixed across stages.

Table R2. Complete-lineage Pass@1 (%), seed versus stage 5.

EnvironmentSolverLineagesSeedStage 5Δ
τ²-RetailQwen 3.5 397B (paper)5377.454.7-22.7
τ²-RetailGLM-5.2 (new)5379.258.5-20.7
τ²-RetailQwen 3.5 9B (new)5366.037.7-28.3
τ²-RetailQwen 3.5 9B + GRPO (new)5375.541.5-34.0
τ²-Retail, alt. orderGLM-5.2 (new)2286.459.1-27.3
τ²-Retail, alt. orderQwen 3.5 9B (new)2268.236.4-31.8
τ²-AirlineGemma 4 31B (paper)2875.028.6-46.4
τ²-AirlineGLM-5.2 (paper)2878.653.6-25.0
τ²-AirlineQwen 3.5 9B (new)2864.321.4-42.9
τ²-AirlineQwen 3.5 9B + GRPO (new)2875.021.4-53.6
General-AgentGemma 4 31B (paper)3971.825.6-46.2

Every configuration shows a decrease from seed to stage 5, between 20.7 and 53.6 points. The effect holds across model families and sizes (Qwen 3.5 9B, Gemma 4 31B, GLM-5.2, Qwen 3.5 397B), two agent harnesses (ReAct and OpenHands), two agent orders, and a solver trained with RL. GLM-5.2 took no part in task generation and shows the same decrease on τ²-Retail as the generator model Qwen 3.5 397B (20.7 and 22.7 points). The difficulty is therefore a property of the tasks and not of the generator.

W2(b). The decrease in Pass@1 does not follow the increase in trajectory length.

Table R3. Increase in reference-solution tool calls and mean change in complete-lineage Pass@1 across the base solvers on each environment (three solvers on each τ² domain, one on General-Agent).

EnvironmentTransformationAdded reference tool callsMean Δ Pass@1
τ²-RetailTool Expansion+3.79 (+79%)-6.9
τ²-RetailCompositionality+1.08 (+9%)-12.6
τ²-AirlineTool Expansion+2.61 (+92%)-7.2
τ²-AirlineMemory+0.83 (+12%)-14.3
General-AgentPartial Observability+0.02 (0%)-20.5

On τ²-Retail, Tool Expansion adds 3.5 times as many steps as Compositionality but causes about half the decrease. On τ²-Airline, Tool Expansion adds 3 times as many steps as Memory and causes half the decrease. Partial Observability on General-Agent leaves reference tool calls (43.80 and 43.82), writes (24.31 and 24.31), and distinct tools (10.37 and 10.39) unchanged, and reduces Pass@1 from 43.6% to 23.1% on the complete lineages.

Solver trajectories show the same pattern. On General-Agent, Gemma uses 40.19 tool calls on seed tasks and 40.41 on stage-5 tasks, while Pass@1 falls from 68.8% to 25.6%. The added difficulty comes from what the solver must recover, retain, and coordinate, not from the number of steps.

W2(c). The decrease is consistent across all configurations.

The complete-lineage evaluation compares the same lineages at the seed stage and at stage 5, so the decrease cannot come from a change in the task set. All 11 configurations in Table R2 show a decrease, across five solvers, three environments, two harnesses, and two agent orders. The decrease ranges from 6 to 18 tasks per configuration. Under the alternative order, Pass@1 decreases at every level for both solvers (Table R5).

W2(d). Every descendant is verified, and the forest produces diverse tasks.

Quality. Every accepted descendant passes two checks. First, its reference solution replays from a fresh environment state without errors. This check is deterministic and does not depend on an LLM. Second, an LLM judge (GPT-OSS 120B) confirms that the instruction, the executed trajectory, and the final state are consistent. Candidates that fail either check are repaired or discarded. The RL result in Table R1 provides further evidence of task quality: GRPO uses task success as the reward, and training on these rewards improves performance on two held-out domains.

Diversity. Diversity arises at two levels. Within a lineage, each transformation changes a different part of the interaction problem. On τ²-Retail, the number of distinct tools per task grows from 3.34 for seeds to 7.15 at stage 5, and writes per task grow from 1.54 to 2.45 (paper, Table 1). Across paths, different agent orders produce different descendants from the same seed pool (W3). After one transformation, the Tool-Expansion-first path adds 3.79 tool calls and 2.35 distinct tools per task, while the Branching-first path adds 2.60 tool calls and 2.00 distinct tools.

AuthorsOpenReview ↗

Response to Reviewer 32At (Part 3)

W3. Agent ordering

Clarification. Section 4 states that the order is fixed "for controlled stage-wise comparison". A fixed order lets us attribute the change between consecutive stages to one transformation while the task population stays fixed. The framework itself does not require a fixed order.

New experiment. We ran a second generation on 53 seeds sampled from the 114 τ²-Retail seeds, with the order Branching → Memory → Tool Expansion → Partial Observability → Compositionality. All other settings are unchanged.

Table R4. Yield and reference-solution statistics under the alternative order.

LevelAgentInputAcceptedTool callsWritesDistinct tools
0Seed53534.431.403.08
1+Branching53377.031.355.08
2+Memory37348.471.385.59
3+Tool Expansion343211.311.446.84
4+Partial Observability323011.201.476.90
5+Compositionality302212.552.277.77

Table R5. Pass@1 (%) under the alternative order. AA: all accepted tasks. CL: 22 complete lineages.

LevelAgentGLM-5.2 AAGLM-5.2 CLQwen 3.5 9B AAQwen 3.5 9B CL
0Seed81.186.464.268.2
1+Branching78.481.859.559.1
2+Memory76.577.355.954.5
3+Tool Expansion71.972.750.050.0
4+Partial Observability70.068.246.745.5
5+Compositionality59.159.136.436.4

The two runs show the forest structure.

  1. The framework runs under a different order without modification. It produced 155 verified descendants from 53 seeds, and Pass@1 decreases at every level for both solvers.
  2. The path determines which descendants are generated. Branching applied to seeds discards 16 of 53 candidates (30%), while Branching applied after Tool Expansion discards 7 of 108 (6.5%). Tool Expansion adds observations on which a branch can condition, so the order of transformations changes the set of tasks that can be built.
  3. Accumulated difficulty is reproducible across paths. At stage 5, GLM-5.2 reaches 59.1% under the new order and 58.5% under the original order. Qwen 3.5 9B reaches 36.4% and 37.7%.
  4. The effect of each transformation persists under reordering. Compositionality adds 0.80 writes per task in both orders (1.65 to 2.45, and 1.47 to 2.27). Partial Observability leaves reference length unchanged in both orders (11.49 to 11.54, and 11.31 to 11.20).

The same seed pool therefore grows along different paths into different tasks, while the complexity added by each path, measured by solver success, is stable.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your reviews. We have posted a response to each review. In addition to addressing your comments and questions, the responses report three new results:

  1. RL training. GRPO training of Qwen 3.5 9B on the generated General-Agent tasks improves Pass@1 on the held-out τ²-Retail and τ²-Airline domains.
  2. Additional solvers. GLM-5.2 and Qwen 3.5 9B on both τ² domains. The decrease in Pass@1 holds for solvers that took no part in task generation, and the gap between strong and weak solvers grows with task complexity.
  3. Alternative agent order. The framework runs under a different order of the hardening agents without any other modification. The order changes which tasks are generated, and in both orders the difficulty grows as the hardening agents are applied.

We would appreciate it if you could let us know whether our responses address your concerns. We are happy to provide clarifications or further details before the discussion period ends.

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.