acceptodds
Under review as a conference paper at ICLR 2027

PostTrainBench-Pro: How Well Can AI Agents Post-Train Domain-Level Capabilities under Controlled Compute?

Abstract

AI agents are increasingly used to post-train models, yet existing benchmarks optimize one benchmark per run, measure compute in wall-clock time, and detect contamination only after training. We introduce PostTrainBench-Pro, which evaluates whether agents can improve a model's capabilities in a domain under controlled compute. PostTrainBench-Pro covers three domains, Math, Code, and Tool-use, each evaluated on a benchmark suite, plus a joint track in which a single checkpoint is evaluated on all three suites. Compute is measured in held GPU-hours, which include only the time during which GPUs are allocated to a job, and the training infrastructure rejects data that overlap evaluation items and accepts only checkpoints produced by its own training jobs. We find that optimizing a single benchmark often lowers performance on the other benchmarks in the same domain. Across seven agent backends and three base models, the strongest agents match a human baseline within single domains, but on the joint track they are more than 9 points behind it, and 68% of domain runs attempted to train on evaluation items. Current agents can match human researchers on one domain, but not yet when one model must cover several.

Primary area:
datasets and benchmarks
Keywords:
Autonomous Post-Training, LLM Agents, Process Evaluation, Benchmarking, PostTrainBench-Pro

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VVvzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PostTrainBench-Pro, a benchmark for evaluating AI agents' ability to autonomously post-train language models under limited GPU compute budgets. It evaluates three domains: Math, Code, and Tool-use, including joint multi-domain training. The benchmark also introduces GPU-hour-based compute accounting and data contamination prevention mechanisms. Experiments across seven agents and three base models show that strong agents can approach human-level performance in single-domain post-training. However, multi-domain training remains challenging, and optimizing individual benchmarks does not always improve overall domain capabilities.

Critical strengths
  1. Realistic and Comprehensive Evaluation: The benchmark moves beyond single-task optimization to domain-level and multi-domain evaluation, providing a more realistic assessment of agents' autonomous post-training capabilities.
  2. Fair and Rigorous Experimental Design: The use of controlled GPU-hour budgets and infrastructure-enforced contamination prevention ensures fairer comparisons and more reliable evaluation results.
  3. Strong Empirical Insights: Extensive experiments across seven agents and three base models reveal important limitations of current agents, particularly benchmark overfitting, cross-domain interference, and the gap between autonomous and human-guided post-training.
Critical weaknesses
  1. Limited Generalizability to Stronger Base Models: The benchmark evaluates agents' ability to improve relatively small base models, such as Qwen3-4B-Base, using powerful backbone models like Claude Opus-5 and GLM-5.3-Flash. However, this setting heavily relies on the substantial capability gap between the agent's backbone model and the model being fine-tuned. It remains unclear whether similarly significant improvements can be achieved when this gap is smaller, such as when fine-tuning Qwen3-32B-Base or other larger base models.

  2. Limited Coverage of Real-World Agentic Coding Tasks: Although the benchmark includes several code-related tasks, it overlooks important agentic coding benchmarks, such as the TerminalBench and SWE-bench series. Incorporating these benchmarks would provide a more comprehensive evaluation of whether autonomous post-training can improve models' capabilities on complex, real-world software engineering and terminal-based tasks.

Questions for authors

Mentioned in Critical Weaknesses

Reviewer_pVx2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing full-stack post-training benchmarks fails to measure domain-generalization, gpu-time budget, and data integrity. This paper proposes PostTrainBench-Pro to fix those with in-domain validation, iso-gpu-time comparison, and harness with improved integrity enforcement. Author found that SoTA agents can post-train in a domain beating human levels. However, agents are still behind human baseline in cross-domain joint track.

Critical strengths
  1. The motivating insights hold (post-training is per domain, not per dataset; gpu-time is a separate resource from reasoning resource; train-on-eval is a pervasive failure mode).
  2. It is valuable to have experimental evidence that under more "proper" comparison, agents can beat human on performing post-training in one domain.
Critical weaknesses
  1. Lack of substantial qualitative analysis, besides two case studies in Appendix G. Why does some model win / lose? What are humans doing differently? e.g., more data augmentation? shorter but faster dev runs? different loss functions? Without these type of qualitative analysis, numbers alone can mask poor harness design as agent incapabilities.
  2. Somewhat necessary yet incremental contribution in comparison to PostTrainBench. The additions (data gates and domain eval) read to me like a patch to PostTrainBench. Gpu-time controlled experiments is more of a nice-to-have, i.e., users may care about e2e time more than GPU time, depending on what is their resource bottleneck.
Questions for authors
  1. Inside the agent sandbox, does agent have internet access to inspect relevant eval suite instances? Do they ever cheat by rephrasing eval instances as training samples? I am concerned about how reliable the proposed data boundary is.
1 comment

I thank authors for their additional results and clarification. Weakness 1 and Question 1 are resolved. I am still somewhat split on Weakness 2 regarding contribution but I believe I have learned enough from the results that others will likely find the same. I have raised my score from weak reject to weak accept. Good luck.

Reviewer_p9ssOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

Paper's question is reasonable and actual: "How Well Can AI Agents Post-Train Domain-Level Capabilities under Controlled Compute?". The benchmark uses small 2-4B base models and compute in H100 GPU hours. Evals are across 3 setups: math, code, tool usage (plus joint track with all three setups). Empirically the paper covers 7 (backend, harness) combinations, 3 base models, 63 domain campaigns, 3 joint campaings, manual human reference recipes. The authors also do substantial ablations for the budget and contamination checks. The main results are:

  • single benchmark optimization can damage other benchmarks in the same domain
  • the strongest agents approximately match the author's human reference when results are averaged over separate domains
  • the base reported joint agent is 9.3 points less than human joint reference
  • 43/63 domain runs have the literal data overlap
Critical strengths
  1. The benchmark have a clear and actual goal and tries to solve a useful problem rather that single benchmark optimization. Good side of the paper is that it goes further and forces the system to allocate data and compute across capabilities that may compete for the parameters. The data in Table 2 and "Joint and domain-only optimization " in Table 4 are useful for the post training community.
  2. A lot of experimental work done on the side of held GPU hour accounting, immutable snapshots, checkpoints, host side evaluation. The 7 x 3 x 3 domain matrix is broad for an autonomous post training study at the chosen cost. Useful observations found that wall clock time and actual held GPU time diverge a lot and agents collect benchmark items unless the infrastructure stops them.
Critical weaknesses
  1. PostTrainBench-Pro does not really show a domain level generalization. Agents may inspect and repeatedly evaluate on the exact configs that influence the final score. There is no held-out domain evaluation. The data gate only rejects literal prompt overlap via 8-gram containment threshold.
  2. The detector and full benchmark implementations are not included in the supplement. Also the detector is not validated for false positives or false negatives.
  3. I did not find the number of seeds reported or repeated evals, seems like the authors use only one run for their experiments. Therefore standard deviation or error are not reported which makes hard to infer the conclusions from the comparison tables.
Questions for authors
  1. Can you please report the performance on hidden domain benchmarks that ere never visible to the agents and were never looked into during the runs?
  2. Can you please do a manual or semantic audit of the contamination gate's false negative and false positive rates?
  3. Can you please repeat all joint-track runs and then report uncertainty from training and evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.