- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper hardens verified agentic tasks by passing them through five ReAct agents (tool expansion, branching, memory, partial observability, and compositionality). Each agent inspects the environment and rewrites both the instruction and the reference solution. A candidate is kept if its reference replays and an LLM judge accepts it; otherwise a repair loop tries to fix it. On τ²-Retail, τ²-Airline, and General-Agent, 212 seeds yield 823 descendants, and solver Pass@1 drops by 18–43 points after all five stages, including on lineages that survive every stage.
Critical strengths
-
Grounding the transformations in the environment, rather than rewriting the prompt, is the right idea, and editing the reference solution together with the instruction keeps the output checkable. The recover-before-hide rule for partial observability is a nice touch.
-
The complete-lineage results are convincing on difficulty. With the task set held fixed, Pass@1 still falls from 77.4% to 54.7% on τ²-Retail and from 71.8% to 25.6% on General-Agent.
Critical weaknesses
-
My main concern is that a lower Pass@1 does not show the tasks are still valid; ambiguous or unsolvable tasks would produce the same drop. Replay only checks that the reference runs, the LLM judge is never validated, and in τ² the user simulator must still be able to supply whatever information was hidden. The General-Agent numbers also look odd: solvers pass 68.8% of seeds with 8.56 writes on average, while the references use 21.41. A human audit of sampled descendants, or at least an error analysis separating task defects from agent mistakes, is needed to support "preserving validity."
-
Several framing claims are untested. The title targets RL environments, but there are no training experiments. Only one fixed agent order is run, so the "forest" is really a chain, the 823 descendants are nested versions of the same seeds, and diversity is never measured. There is also no comparison with plain instruction rewriting under the same validation, which is the baseline the paper argues against.
Post-rebuttal update
Thanks for the additional experiments. The cross-domain RL gains and the results with more solvers and a different agent order address several of my concerns. My main concern remains task validity: the new results leave open how often failures come from task ambiguity or defects. I’m keeping my rating at 2.
Questions for authors
- How is task success determined in each environment? On General-Agent seeds, reference solutions average 21.41 writes (Table 1), yet solvers pass 68.8% of tasks with 8.56 writes on average (Table 4), which suggests the success check is much looser than the reference.
- Can you report a human validation of sampled descendants per stage (solvable, unambiguous, reference correct), together with the LLM judge's agreement with it?
- How does a non-grounded rewriting baseline perform under the same validation and repair budget?
- On τ²-Retail, Qwen 3.5 397B is both the generator and the only solver. Does the drop hold for other solvers?
Suggestion for future work: training experiments (SFT or RL) on the generated tasks would directly support the RL framing.
4 comments
Response to Reviewer xs6n (Part 2)
W2. Framing claims
Training on Agent Forest tasks (new result)
We trained Qwen 3.5 9B with GRPO on the 211 General-Agent descendants (8 rollouts per task, reward 1 if the episode passes the task verification and 0 otherwise) using a ReAct harness. We evaluated the model with the same harness on τ²-Retail and τ²-Airline, which were not used in training, with GPT-OSS 120B as the user simulator. This experiment addresses the RL framing and the reviewer's suggestion.
Table B. Pass@1 (%) of Qwen 3.5 9B before and after GRPO, all accepted tasks.
| Stage | Retail before | Retail after | Δ | Airline before | Airline after | Δ |
|---|---|---|---|---|---|---|
| 0 Seed | 64.0 | 71.1 | +7.1 | 60.0 | 72.0 | +12.0 |
| 1 +Tool Expansion | 58.3 | 64.8 | +6.5 | 52.6 | 63.2 | +10.6 |
| 2 +Branching | 55.4 | 61.4 | +6.0 | 54.1 | 62.2 | +8.1 |
| 3 +Memory | 52.2 | 57.6 | +5.4 | 45.5 | 51.5 | +6.0 |
| 4 +Partial Observability | 48.3 | 52.8 | +4.5 | 48.5 | 54.5 | +6.0 |
| 5 +Compositionality | 37.7 | 41.5 | +3.8 | 21.4 | 21.4 | 0.0 |
| All stages | 54.4 | 60.1 | +5.7 | 48.9 | 56.6 | +7.7 |
- Training transfers to held-out domains. Summed over all stages, the trained model solves 32 more tasks on Retail (335 versus 303 of 557) and 17 more on Airline (124 versus 107 of 219).
- The stages show how far training transfers. The gains are largest on seed tasks and decrease as complexity accumulates. After training, stage-5 accuracy is 41.5% on Retail and 21.4% on Airline, so the fully transformed tasks remain open targets for further training.
Instruction-rewriting baseline
As the reviewer notes in S1, grounding the transformations in the environment, rather than rewriting the prompt, is the central design choice. A rewriting baseline without environment access must guess the database values that a new reference solution uses (order IDs, reservation IDs, amounts), and replay rejects candidates whose values do not exist or whose actions fail. TaskCraft reports this effect in a related setting: generating tasks from tool context raises the validation pass rate from 18.5% to 43.0% compared with direct LLM prompting (Shi et al., ICLR 2026, Table 4). Partial Observability cannot be performed by rewriting at all, because recover-before-hide requires calling the recovery tool and observing the value before it is hidden.
Response to Reviewer xs6n (Part 3)
W2 (continued). Agent order and the forest structure (new result)
The paper fixes the order to attribute the change between consecutive stages to one transformation while the task population stays fixed. The framework does not require a fixed order. We ran a second generation on 53 seeds sampled from the 114 τ²-Retail seeds, with the order Branching → Memory → Tool Expansion → Partial Observability → Compositionality. All other settings are unchanged.
Table C. Yield and reference-solution statistics under the alternative order.
| Level | Agent | Input | Accepted | Tool calls | Writes | Distinct tools |
|---|---|---|---|---|---|---|
| 0 | Seed | 53 | 53 | 4.43 | 1.40 | 3.08 |
| 1 | +Branching | 53 | 37 | 7.03 | 1.35 | 5.08 |
| 2 | +Memory | 37 | 34 | 8.47 | 1.38 | 5.59 |
| 3 | +Tool Expansion | 34 | 32 | 11.31 | 1.44 | 6.84 |
| 4 | +Partial Observability | 32 | 30 | 11.20 | 1.47 | 6.90 |
| 5 | +Compositionality | 30 | 22 | 12.55 | 2.27 | 7.77 |
Table D. Pass@1 (%) under the alternative order. AA: all accepted tasks. CL: 22 complete lineages.
| Level | Agent | GLM-5.2 AA | GLM-5.2 CL | Qwen 3.5 9B AA | Qwen 3.5 9B CL |
|---|---|---|---|---|---|
| 0 | Seed | 81.1 | 86.4 | 64.2 | 68.2 |
| 1 | +Branching | 78.4 | 81.8 | 59.5 | 59.1 |
| 2 | +Memory | 76.5 | 77.3 | 55.9 | 54.5 |
| 3 | +Tool Expansion | 71.9 | 72.7 | 50.0 | 50.0 |
| 4 | +Partial Observability | 70.0 | 68.2 | 46.7 | 45.5 |
| 5 | +Compositionality | 59.1 | 59.1 | 36.4 | 36.4 |
The two runs show the forest structure.
- The framework runs under a different order without modification. It produced 155 verified descendants from 53 seeds, and Pass@1 decreases at every level for both solvers in both views.
- The path determines which descendants are generated. Branching applied to seeds discards 16 of 53 candidates (30%), while Branching applied after Tool Expansion discards 7 of 108 (6.5%). Tool Expansion adds observations on which a branch can condition, so the order of transformations changes the set of tasks that can be built. The first step also changes tasks differently: Tool Expansion first adds 3.79 tool calls and 2.35 distinct tools per task, while Branching first adds 2.60 tool calls and 2.00 distinct tools.
- Accumulated difficulty is reproducible across paths. At stage 5, GLM-5.2 reaches 59.1% under the new order and 58.5% under the original order. Qwen 3.5 9B reaches 36.4% and 37.7%.
- The effect of each transformation persists under reordering. Compositionality adds 0.80 writes per task in both orders (1.65 to 2.45, and 1.47 to 2.27). Partial Observability leaves reference length unchanged in both orders (11.49 to 11.54, and 11.31 to 11.20). Nested descendants. Nesting is part of the design. Because each stage extends the previous one, the complete-lineage evaluation can hold the task set fixed and attribute each change in Pass@1 to one transformation, which is the comparison the reviewer finds convincing (S2). Each stage is a different task with its own reference solution. On τ²-Retail, reference solutions grow from 4.82 to 12.62 tool calls and from 3.34 to 7.15 distinct tools per task (paper, Table 1), and the alternative order grows the same seeds into different intermediate tasks.
Questions
Q1. Task success. On τ²-Retail and τ²-Airline, a task is solved when the final database state matches the state produced by replaying the reference solution of that stage.
Q2. See W1.
Q3. See W2 (instruction-rewriting baseline).
Q4. Other solvers on Retail. Table E reports two solvers that took no part in task generation. Pass@1 decreases at every stage for both. From seed to stage 5, GLM-5.2 drops by 20.4 points (all accepted) and 20.7 points (complete lineages), close to the generator Qwen 3.5 397B (20.7 and 22.7 points, paper Table 2). Qwen 3.5 9B drops by 26.3 and 28.3 points. The decrease also holds on τ²-Airline for all four solvers in Table A.
Table E. Pass@1 (%) on τ²-Retail. AA: all accepted tasks. CL: 53 complete lineages.
| Stage | Agent | GLM-5.2 AA | GLM-5.2 CL | Qwen 3.5 9B AA | Qwen 3.5 9B CL |
|---|---|---|---|---|---|
| 0 | Seed | 78.9 | 79.2 | 64.0 | 66.0 |
| 1 | +Tool Expansion | 75.0 | 75.5 | 58.3 | 58.5 |
| 2 | +Branching | 72.3 | 73.6 | 55.4 | 56.6 |
| 3 | +Memory | 70.7 | 71.7 | 52.2 | 54.7 |
| 4 | +Partial Observability | 67.4 | 69.8 | 48.3 | 50.9 |
| 5 | +Compositionality | 58.5 | 58.5 | 37.7 | 37.7 |
Suggestion (training experiments). See W2, Table B.
We thank the reviewer again and hope these results address the concerns raised.
Thank you for your updated response. We are glad that the cross-domain RL gains, the additional solvers, and the second agent order address several of your concerns. We will take your remaining point on task validity into account in the revision.
Response to Reviewer xs6n (Part 1)
We thank the reviewer for the detailed review and for recognizing that grounding the transformations in the environment is the right approach, that editing the reference solution together with the instruction keeps the output checkable (S1), and that the complete-lineage results are convincing on difficulty (S2). We report three new results and then address each weakness and question.
W1. Validity of the transformed tasks
We agree that a lower Pass@1 alone does not establish validity. The validity of Agent Forest tasks rests on the four points below. None of them depends on the LLM judge, which acts as an additional filter that rejects descendants whose instruction, trajectory, and final state disagree.
1. Replay fixes a reachable target. On τ², a task is solved when the final database state matches the state produced by replaying the reference solution of that stage (Q1). Replay therefore does more than check that the reference runs. It defines the evaluation target, and every accepted descendant has an action sequence that reaches this target from a fresh environment state.
2. The transformation rules exclude unsolvable and artificial difficulty (Appendix B).
3. The τ² user simulator does not need to supply hidden values. The simulator receives the instruction of the current stage. Partial Observability removes a concrete value from that instruction while keeping the user's intent ("My reservation is RES-123, please modify it" becomes "Please modify my current active reservation"). The value must be uniquely recoverable from the environment through a non-circular tool path (here, by looking up the user's reservations). The agent recovers it with tools and, after Memory, retains it until it is used.
4. Solver behavior matches tasks that are harder but solvable.
Table A. Complete-lineage Pass@1 (%), seed versus stage 5.
If the transformations made tasks ambiguous or unsolvable, strong and weak solvers would fail those tasks alike, and their scores would move closer together. Table A shows the opposite. On τ²-Retail, the gap between the strongest solver (GLM-5.2) and the weakest (Qwen 3.5 9B) grows from 13.2 points on seed tasks to 20.8 points at stage 5, and the four solvers keep the same ranking. On τ²-Airline, the same gap grows from 14.3 to 32.2 points. Stronger solvers handle the added difficulty better, which is what we expect from tasks that are harder but solvable. GLM-5.2 still solves 58.5% (Retail) and 53.6% (Airline) of the fully transformed tasks. Across 11 configurations (five solvers, three environments, two agent orders), Pass@1 at every transformed stage is lower than at the seed stage, in both the all-accepted and the complete-lineage views.