- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TMAX, a simple and open recipe for training terminal-using language-model agents. The authors construct TMAX-13K, a dataset of 13,565 synthetic terminal environments generated through a compositional taxonomy that explicitly controls domain diversity, task and command complexity, personas, fixtures, and verifier types, followed by an LLM-based verifier auditing and repair stage. They then train open-weight models using a relatively simple outcome-based DPPO recipe with several stability improvements, including an FP32 LM head and larger rollout groups. Experiments show that TMAX-13K outperforms a broad set of existing terminal-agent RL datasets under the same training setup, and TMAX-9B reaches 33% on Terminal-Bench 2.0, the strongest reported result among models below 10B parameters. The gains largely transfer across model sizes, Qwen model variants, agent harnesses, and additional tasks such as SWE-Bench Verified and AIME, suggesting that the proposed recipe improves general terminal/tool-use capabilities rather than only fitting a single benchmark.
Critical strengths
- The paper presents a simple yet well-designed recipe for generating large-scale terminal-agent RL environments, with explicit control over task diversity and difficulty, together with verifier auditing and repair. The resulting TMAX-13K dataset is substantially more balanced and challenging than several existing terminal-agent datasets.
- The proposed recipe consistently improves multiple model sizes and families, and the gains transfer across different agent harnesses and to additional tasks such as SWE-Bench Verified and AIME. In particular, TMAX-9B achieves strong Terminal-Bench performance relative to similarly sized open models.
Critical weaknesses
- It remains unclear whether TMAX-13K is intrinsically a better RL dataset or is better aligned with Terminal-Bench. The 13-gram decontamination analysis rules out direct lexical overlap, but does not exclude higher-level similarity in task archetypes, required skills, environment structure, domain composition, or difficulty. Since the main comparison against alternative RL datasets is conducted on TB-Lite/TB2.x, such distributional alignment could partly explain TMAX-13K's advantage. Although the SWE-Bench Verified and AIME experiments demonstrate transfer over the base model, they do not compare against models trained on alternative RL datasets.
- Only 7.7% of the initially generated environments are classified as clean, while 76.4% are judged to have overly weak verifiers. The authors repair the dominant failure modes and show that training on the repaired data improves downstream performance, but they explicitly do not perform a second rubric-based audit after repair. Therefore, the actual validity rate of the final TMAX-13K dataset remains unclear, including whether repairs fully resolve the original problems or introduce new ones.
- Dataset difficulty is primarily characterized using a single evaluator, Gemini-3-Flash-Preview, so it is unclear how well this calibration transfers across model strengths and families. More importantly, the gains become much smaller for the strongest tested model: Qwen 3.6 27B improves only modestly on TB2.1 and slightly degrades on TB-Lite, whereas substantially larger gains are observed for smaller models. This raises the possibility that TMAX-13K can only provide useful learning signal for relatively weak starting policies and may become saturated and less useful for stronger ones.
Questions for authors
- The paper reports that RL training often collapses after roughly 300 steps and presents several practical mitigations, including an FP32 LM head, larger rollout groups, and reduced infrastructure noise. While the ablations provide useful evidence that these interventions can improve stability, they do not identify the underlying failure mode that actually triggers collapse. for example, whether collapsing runs exhibit abnormal shifts in trajectory length or content, reward/advantage distributions, rollout diversity, tool-use behavior, log-probability mismatch, filtering rates, or other training statistics. Without such diagnostics, it is difficult to tell whether the proposed interventions address the same underlying pathology or merely suppress different symptoms. A targeted analysis comparing trajectories and training signals immediately before and after collapse, ideally across stable and unstable runs, would make the stability claims substantially more informative and help determine whether the proposed recipe is robust beyond the specific engineering setup.
- The authors state that DPPO appeared more stable than GRPO in early pilot experiments, but no quantitative comparison is provided. As a result, it is unclear whether the advantage of TMAX-13K is specific to its interaction with DPPO or whether the dataset remains superior under other commonly used RL objectives. At minimum, a controlled vanilla-GRPO comparison would substantially strengthen the generality of the conclusions.
- Although the authors acknowledge that the pipeline is completely synthetic and may struggle with settings such as large-scale codebases, the paper does not directly characterize how well the synthetic task distribution reflects real terminal-agent workloads. In particular, roughly uniform coverage across the nine predefined domains is a useful design choice, but does not by itself establish that the generated tasks match realistic user intents, workflows, failure modes, or environment constraints. Have you considered grounding the synthetic generation pipeline in real public issues/tasks while retaining its controllable composition?
Please also see the Weaknesses section for additional questions and concerns.
4 comments
Rebuttal - Addressing Reviewer Questions
Here, we directly answer your questions:
What is the underlying reason for training collapsing? Can you perform a targeted analysis comparing trajectories and training signals?
We provide further analysis and discussion of why training collapses in Appendix E.6. In general, we observed in earlier experiments that collapses in performance were often preceded by large spikes in logprob differences, and as such fixes to reduce such mismatch aided in reducing these crashes. We show that crashes in reward are preceded by logprob and gradient norm spikes, which the FP32 head, DPPO, and larger group size target fixing. We provide ablations and diagnostic figures in Appendix E.6 to ablate these changes (see Figures 8, 11, 12). Surprisingly, training on more stable infrastructure also reduced these spikes. This suggests they are partly caused by noisy rollouts (e.g., tool calls hanging due to slow sandboxes), and that non-robust infrastructure can appear as optimization issues. Reward drops were accompanied by shorter model outputs: in the 8-group size case, 11% of rollouts were cut short before the reward drop, and 38% were cut short afterward. The model also goes from about 20 to 13 bash calls per rollout.
Any experiments comparing DPPO and GRPO?
We provide a comparison in Appendix E.6, Figure 11 comparing GRPO and DPPO under matched settings from an earlier pilot experiment. While DPPO did not completely stop collapse, it did collapse later than GRPO, and when it did collapse, it did not collapse as severely, based on train reward. This provides quantitative evidence for our swap to DPPO. We did not compare all datasets using GRPO due to compute limitations, and we note that we believe that DPPO and GRPO are not very different (DPPO primarily improves the training-inference mismatch issue in GRPO training), suggesting that results would likely transfer to GRPO-like algorithms.
Have you considered grounding the synthetic generation pipeline in real public issues/tasks while retaining its controllable composition? How well does it match realistic user behaviours?
We consider grounding the pipeline in real repositories or tasks is an interesting extension for future work, potentially allowing for more complex tasks while controlling the composition of the dataset.
In Appendix F.4, we do a detailed comparison of Tmax’s distribution to Terminal-Bench Lite/2/4, 1k random samples from SWE-Chat (a dataset of user prompts to coding agents), and several baseline RL datasets. We find that Terminal bench has a significantly different distribution to both TMax and SWE-Chat, and that user tasks seem to focus mostly on software engineering tasks over e.g. data processing tasks. This suggests that focussing more on software engineering tasks could be useful for real-world users, although keeping a diverse set of tasks is useful for evaluations like Terminal Bench. We also note that one can easily vary the distribution of the dataset in our pipeline by simply adjusting the sampling weights when choosing axis values, and that a uniform prior is a fairly reasonable diversity-maximising choice.
Follow up to Reviewer JEiv
Thank you again for your detailed and thoughtful review. As the discussion period closes today (Nov 18), we wanted to kindly follow up.
Your questions led to several new results / analyses:
- Comparisons against models trained on other RL datasets on SWE-Bench Verified and AIME (Table 4).
- A post-repair audit of TMax-13K.
- Corrected 27B results with a pass-rate analysis (Appendix F.3).
- Collapse diagnostics and a GRPO comparison (Appendix E.6).
- A comparison with real user tasks (Appendix F.4).
We would really appreciate it if you could take a look at our response. If it addresses your concerns, we would be grateful if you would consider raising your score. We are happy to clarify anything before the deadline!
Thank you for the detailed rebuttal and additional experiments. The cross-dataset evaluations together with the additional RL stability diagnostics substantially address several of my original concerns. Remaining questions:
- I remain concerned about the reliability of the learning signal provided by TMAX-13K. approximately 34% of the final TMAX-13K environments are still not classified as clean, including around 21% with overly weak verifiers. Could the authors clarify the implications of this residual error rate for reward reliability?
- The first concern may also be related to the limited gains observed for the 27B model, the improvement on TB2.1 is only 0.3 percentage points, substantially smaller than the reported evaluation uncertainty. Could the authors clarify whether the gains for 27B are statistically distinguishable from noise?
I appreciate the authors' additional efforts, but I decide to maintain my original score.
Rebuttal - Addressing Reviewer Concerns
We thank the reviewer for noting that our recipe is simple and well-designed, and that our setup transfers across model sizes, families, and tasks. We start by addressing your concerns directly, and answer the questions in a second comment.
To investigate this, we evaluate the other models trained on other datasets on SWE-Bench Verified and AIME, and show the results in Table 4 (and full results in Table 16). We find that TMax has the highest AIME performance, and only falls behind on SWE-Bench Verified compared to SWE-Smith, which explicitly was constructed for SWE-Bench. As such, this suggests that the diversity and difficulty of Tmax can lead to improvements on non-terminal bench evaluations relative to other datasets. This also suggests that TMax’s diversity makes it a better choice relative to other datasets for a general set of tasks.
We also compare the distribution of domains between evaluation datasets and RL datasets (including TMax) in Appendix F.4. We find that TermiGen better matches Terminal-Bench Lite and 2.1 distributions, despite underperforming TMax, although the balanced distribution does result in TMax having more data science and scientific computing tasks than other baseline datasets.
Importantly, we primarily compare to other works that also target TB (e.g., TermiGen, OpenThoughts) as their primary evaluation. In this light, our purely synthetic data generation pipeline and ability to tightly control the distribution of our samples to ensure diverse tasks is a strength of our work. Future users of TMax could adjust the sampling ratios of task domains to better match particular distributions of interest to them (e.g., based on real-world user analyses).
We run the same auditor over our repaired dataset, and find that roughly 70% of the ‘verifier-too-weak’ samples now are classified as ‘clean’ (in the entire dataset, 9,014 of 13,565 are judged clean). Overall, in Tmax-13k, 66% samples are classified as ‘clean’, 21% have ‘verifier too weak’, and only ~4% now have answer leakage. We do not apply the repair stage to already-clean tasks, so it cannot make already-good tasks worse. Around 5% of tasks receive a different issue label after repair, but this is fairly small relative to the number of completely fixed tasks. We do not apply further repair rounds due to cost. We show in Table 6 that training on the post-repair data results in significantly stronger performance, showing that the repair stage is effective.
We first note that we found a minor bug around chat templates when evaluating Qwen 3.6 27b (we forgot to turn on preserved thinking). When we fix this, we find that Tmax training also improves Qwen 3.6 27b on TBLite and TB2.1:
Inspired by the reviewer, we also do additional analyses on the difficulty of TMax for Qwen 3.6 27B (and the other model sizes) by looking at the pass rates of groups during RL training (i.e., number of 0/32, 8/32, … , 32/32 correct groups). We add this analysis to Appendix F.3 and find that tmax-13k is not trivially solved by Qwen 3.6 27B, and in fact over 30% of tmax-13k prompts still provide signal during RL training (i.e., we have mixed-outcome rollouts). This suggests that TMax is still difficult enough for stronger models. We also note that our core focus is designing a strong academic setting for small-scale terminal agent training, and as such using smaller, weaker policies is in some ways a feature of our setting, not a bug. We do further investigation (see App. F.3 in the revised copy) and find that the 27B model does improve on many tasks, but also degrades on a substantial number due to it losing some ability to self-verify results. We leave investigating why this occurs to future work.