- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a pipeline for extending tasks using an LLM by having them modify existing tasks. Static checks gate whether a modified task is accepted. SFT on successful rollouts on these new tasks improves model performance.
Critical strengths
- The pipeline generates more complex tasks from known-good tasks using LLMs to extend the tasks. The generated tasks require longer reference solutions, and SFT on successful rollouts on these tasks improves model performance.
Critical weaknesses
- It's not clearly established whether this specific method provides any gain over baselines, since there are no comparisons to baselines with equal amounts of compute. What is missing is a clean comparison to other tasks using the same token budget, successful rollouts from the same model, and running in the same harness. The comparison in Table 6 is not that, and shows the issue if this is not matched (e.g., the performance of Qwen3.5 drops if you train on Nemotron trajectories).
- None of the generated tasks are released, making it difficult to assess whether they are useful or are underspecified or obscure. There are three indications for why the generated tasks might be of poor quality. 1. The modifications (Tab. 11) are adding chores on top of tasks without adding intellectual challenge. 2. The task generation model is instructed to not put information in the instructions that can be discovered from the workspace. Whether something is discoverable is therefore wholly up to the judgement of a single model, which is the same model that designed the task. 3. The task quality control from App. E.1 is not implemented as described. Instead, only rigid, hard-coded checks (regex, length checks, count of the number of numbered steps) are used for issues instead of more thorough methods. Especially the last point is worrying, since the quality checks are very crude. Running the “quality checks” on TB-hard would reject 100% of tasks. Just 80% would be blocked by the regex detecting the numbered lists (
^\s*(?:\d+[\.\)]|\-\s+\*\*)fromsynthesis_pipeline.py).
Questions for authors
- What if you train on other successful rollouts, e.g., on successful Qwen rollouts on unmodified TMax-15K tasks? And how does this perform compared to other baselines?
- How did you ensure the tasks are of good quality beyond the metrics from the paper?
- Can you release the generated tasks?
8 comments
W2: We thank the reviewer for carefully examining our code and raising these concerns. We would like to clarify the following points and provide additional experimental evidence.
1. Task Complexity. Our 40 operators do more than simply add extra steps or outputs. For example, debugging, test failure repair, state migration, and Git forensics require agents to identify problems, understand their causes, and develop solutions. As shown in Appendix B.5, these operators introduce new challenges rather than simply increasing task length.
2. Requirement Discoverability. We agree that the generation model may incorrectly assume that certain information is available in the workspace. To address this, our pipeline checks the alignment between task instructions, workspace files, reference solutions, and verifiers. When inconsistencies are found, the pipeline can repair them rather than simply rejecting the task. Importantly, our independent human evaluation (Appendix G, Table 13) shows that 90–94% of RST tasks are publicly solvable, compared with 86–88% for single-pass baselines.
3. Quality Control. The regex and length checks mentioned by the reviewer are preliminary filters designed to prevent overly detailed instructions and trivial rewrites, rather than general measures of task quality. These filters are designed for RST task generation, not for evaluating existing benchmark tasks like TB-hard. Therefore, rejecting TB-hard tasks does not mean those tasks are of poor quality. Beyond these filters, we check operator compatibility, instruction–solution–verifier consistency, environment configuration, and whether reference solutions pass verification in fresh Daytona sandboxes. For recoverable failures, our pipeline uses validation feedback to repair the task, with at most two repair rounds, followed by revalidation. Furthermore, independent LLM and human evaluations (Appendix G, Tables 12–13) show that 86–92% of RST tasks are release-ready, compared with 82–84% for the baselines.
4. Additional Experiment: To directly evaluate whether our quality checks can detect hidden requirements, we sampled 20 tasks from each of R1, R5, R10, and R15, resulting in 80 tasks. For each task, we created two versions with the same target requirement:
- Discoverable: The requirement can be found in the public instruction or workspace.
- Hidden: The requirement is still checked by the verifier but cannot be found in either the public instruction or workspace.
We then applied our quality checks to both versions. A successful check should accept the discoverable version and flag the hidden version.
Table. Hidden Requirement Detection Across Recursive Rounds.
| Round | Task Pairs | Discoverable Accepted | Hidden Flagged | False Positives | False Negatives | Accuracy |
|---|---|---|---|---|---|---|
| R1 | 20 | 20/20 (100%) | 20/20 (100%) | 0 | 0 | 100% |
| R5 | 20 | 20/20 (100%) | 20/20 (100%) | 0 | 0 | 100% |
| R10 | 20 | 20/20 (100%) | 20/20 (100%) | 0 | 0 | 100% |
| R15 | 20 | 20/20 (100%) | 20/20 (100%) | 0 | 0 | 100% |
| Overall | 80 | 80/80 (100%) | 80/80 (100%) | 0 | 0 | 100% |
Results. Our quality checks correctly accepted all 80 discoverable versions and flagged all 80 hidden versions, achieving 100% accuracy across 160 test cases. This performance remains consistent from R1 to R15, suggesting that hidden-requirement detection remains effective across recursive rounds. More importantly, the results show that our checks can identify requirements missing from publicly available information, beyond simply filtering instruction formats. In our full pipeline, detected inconsistencies can also enter a bounded repair process, followed by revalidation.
Task Release. We agree that releasing generated tasks is important for independent assessment. As stated in our paper, the synthesized tasks and trajectories will be released upon publication.
We appreciate the reviewer's constructive suggestions and will include these additional experiments and clarifications in the revised appendix.
Q1: Comparison with Successful Rollouts from Other Tasks.
We thank the reviewer for this helpful question. To directly address this concern, we conducted a controlled SFT experiment using 5,000 successful Qwen3.5-27B trajectories collected from unmodified TMax-15K tasks and RST (R1–R3), respectively. Under the same rollout and training settings, RST achieves 43.45 ± 0.65% on TB2, compared with 40.82 ± 1.72% for TMax-15K (+2.63 percentage points). This suggests that the benefit of RST comes not simply from training on successful trajectories, but also from the additional learning challenges introduced by recursive task generation. Furthermore, Table 6 compares RST against five public SFT datasets under the same training-token budget, where RST achieves 46.82%, outperforming the strongest baseline at 42.70% (+4.12 percentage points). Together, these results show that RST provides useful training signals beyond those obtained from existing tasks and successful rollouts.
W1: We thank Reviewer RwGJ for the careful reading and constructive feedback. We agree that controlled comparisons are important for evaluating RST. We clarify our existing experiments and provide two additional comparisons using the same rollout model and harness.
1. Existing Controlled Ablation. Section 4.1 (Table 1) presents a 2×2 ablation using the same generation model, operator library, validation pipeline, and generation budget. With 3,000 training examples per condition and identical SFT settings, RST achieves 45.32% on TB2, compared with 43.07% for the strongest alternative. This supports the benefits of solution-first rewriting and recursive task reuse.
2. Purpose of Table 6. Table 6 evaluates existing public SFT datasets under identical training conditions, rather than comparing their original data-generation pipelines. We use the same base model, SFT settings, evaluation setup, and exactly 36,363,275 assistant-loss tokens for each dataset. RST achieves 46.82% on TB2, compared with 42.70% for the strongest public baseline. However, these datasets were originally collected using different rollout models and harnesses. We therefore conducted the following additional comparisons.
3. Additional Controlled Experiments. Following the reviewer's suggestion, we compared RST with TMax-15K [1] using the same Qwen3.5-27B rollout model, harness, and inference configuration.
(1) Controlled RL Comparison. We trained Qwen3.5-27B on 14,601 tasks from TMax-15K and an equally sized subset of RST tasks sampled from the 15 rounds (R1–R15). Both conditions used identical model initialization, rollout settings, and PPO configurations (Section 4.7 and Appendix H).
Table. Controlled RL Comparison on Terminal-Bench 2.
(2) Controlled SFT Comparison. We collected successful trajectories from TMax-15K and RST R1–R3 using the same Qwen3.5-27B rollout configuration: SGLang, Harbor, Daytona, temperature 1.0, top-p 0.95, up to 100 turns, and 8,192 generation tokens per call (Appendix H).
We selected 5,000 successful trajectories from each dataset and fine-tuned Qwen3.5-27B using identical SFT settings.
Table. Controlled SFT Comparison on Terminal-Bench 2.
Why We Do Not Match Tokens in These Additional Comparisons. Unlike Table 6, we intentionally match the number of tasks for RL and the number of successful trajectories for SFT, rather than forcing the same total token count. This is because task difficulty can affect how many steps an agent takes, how long its trajectories become, and what the model learns from them. Forcing equal token counts could require truncating or resampling trajectories, changing these natural differences. Our goal here is to compare the training value of the tasks and trajectories produced by each dataset under the same model, harness, and training configuration.
Results. RST achieves 48.31% vs. 47.57% in RL (+0.74 percentage points) and 43.45% vs. 40.82% in SFT (+2.63 percentage points). Notably, RL achieves substantially higher absolute performance than SFT from the same Qwen3.5-27B base model (41.20 ± 1.72). This may relate to the previously observed ~20% gap between pass@8 and pass@1, suggesting that RL can unlock existing capabilities by shifting probability toward successful solutions. In contrast, SFT is limited by the coverage of fixed trajectories, although our scaling results show that increasing SFT data can further improve performance.These findings suggest that combining SFT and RL may lead to better performance, as SFT helps the model learn more solution strategies, while RL improves its ability to use them successfully. Exploring this combination could be an interesting direction for our future work.
We appreciate the reviewer's suggestion and will include these experiments in the revised appendix.
References
[1] Ivison, H., et al. (2026). Tmax: A Simple Recipe for Terminal Agents. https://arxiv.org/abs/2606.23321