- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PostTrainBench-Pro, a benchmark for evaluating AI agents' ability to autonomously post-train language models under limited GPU compute budgets. It evaluates three domains: Math, Code, and Tool-use, including joint multi-domain training. The benchmark also introduces GPU-hour-based compute accounting and data contamination prevention mechanisms. Experiments across seven agents and three base models show that strong agents can approach human-level performance in single-domain post-training. However, multi-domain training remains challenging, and optimizing individual benchmarks does not always improve overall domain capabilities.
Critical strengths
- Realistic and Comprehensive Evaluation: The benchmark moves beyond single-task optimization to domain-level and multi-domain evaluation, providing a more realistic assessment of agents' autonomous post-training capabilities.
- Fair and Rigorous Experimental Design: The use of controlled GPU-hour budgets and infrastructure-enforced contamination prevention ensures fairer comparisons and more reliable evaluation results.
- Strong Empirical Insights: Extensive experiments across seven agents and three base models reveal important limitations of current agents, particularly benchmark overfitting, cross-domain interference, and the gap between autonomous and human-guided post-training.
Critical weaknesses
-
Limited Generalizability to Stronger Base Models: The benchmark evaluates agents' ability to improve relatively small base models, such as Qwen3-4B-Base, using powerful backbone models like Claude Opus-5 and GLM-5.3-Flash. However, this setting heavily relies on the substantial capability gap between the agent's backbone model and the model being fine-tuned. It remains unclear whether similarly significant improvements can be achieved when this gap is smaller, such as when fine-tuning Qwen3-32B-Base or other larger base models.
-
Limited Coverage of Real-World Agentic Coding Tasks: Although the benchmark includes several code-related tasks, it overlooks important agentic coding benchmarks, such as the TerminalBench and SWE-bench series. Incorporating these benchmarks would provide a more comprehensive evaluation of whether autonomous post-training can improve models' capabilities on complex, real-world software engineering and terminal-based tasks.
Questions for authors
Mentioned in Critical Weaknesses
I thank authors for their additional results and clarification. Weakness 1 and Question 1 are resolved. I am still somewhat split on Weakness 2 regarding contribution but I believe I have learned enough from the results that others will likely find the same. I have raised my score from weak reject to weak accept. Good luck.