- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Auto-Research Arena, a benchmark that recasts Karpathy’s autoresearch LM-pretraining testbed as nine constrained optimization problems. Each track minimizes one efficiency target (parameters, tokens, updates, FLOPs/token, peak memory, training time, KV-cache memory, decode latency, or request latency). A candidate is admissible only if it reaches BPB < 1.05 and stays within resource limits on the other metrics.
Seven harnesses are compared with Claude Opus 5 and GPT-6, each with 100 charged experiments per track at 10 minutes of training per candidate, and results are aggregated with a margin-aware Elo. The harnesses are AutoScientists, Arbor, GEAR, Heuresis, the authors’ Sequential and Beam Search, and TPE.
Critical strengths
- The proposed task is well motivated. With the current development in auto research, the field needs a reliable benchmark to evaluate the performances.
- The objectives evaluated are broad, covering aspects from training to inference efficiency. The setup (Karpathy’s autoresearch language-modeling testbed) provides quantifiable feedback signals that are grounded in real tasks.
Critical weaknesses
- The paper claims to test auto research capabilities. However, they only test on the task of LLM pretraining. This is a very limited domain as auto research models are now employed in all major scientific domains. Also, a big part of auto research is research idea proposal and enhancement. The paper's evaluation only focuses on performance optimization. Overall, the focus is a very narrow task which could weaken the overall claim that an auto research benchmark is being developed.
- The reported results may not be very reliable. All results are reported against one single run. No confidence intervals are reported.
- The paper is reusing an established testbed with variations. Limited novelty is displayed.
Questions for authors
See weaknesses.
2 comments
Dear Reviewer mMWF,
We have clarified Arena’s contribution beyond single-objective evaluation and added repeated-search experiments to assess ranking stability. We have also addressed your other concerns and revised the manuscript accordingly. We would appreciate it if you could revisit these updates and consider updating your rating if they resolve your concerns. Thank you again.
Author Response
Overclaim autoresearch focus.
Arena is explicitly scoped to autonomous research in language modeling, as stated in the title and limitations, rather than research across all scientific disciplines.
We respectfully disagree that Arena only addresses performance optimization rather than idea proposal and refinement. Efficiency is the research objective; proposing and refining ideas is how agents pursue it. Section 4.2 documents the proposed ideas closely related to ongoing research directions, including looped Transformers and cross-layer K/V sharing.
Lack of multiple runs for ranking reliability.
We kindly refer to common question 1 in the general response for details.
Limited Novelty.
We respectfully disagree that building upon an established testbed, by itself, implies limited novelty. For example, MLE-bench (ICLR 2025) builds on existing Kaggle competitions, while the Automated LLM Speedrunning Benchmark (NeurIPS 2025) builds on the existing NanoGPT Speedrun benchmark.
Arena’s novelty lies in its multi-target evaluation framework: nine complementary efficiency targets with corresponding quality and resource constraints. Moving beyond validation-loss minimization, this framework enables controlled study of how research objectives shape harness performance and idea discovery. The resulting searches uncover a substantially broader range of ideas while also aligning with active research directions (Sec. 4.2). We further demonstrate that these discoveries can be adapted and combined in a subsequent search to improve model quality (Sec. 4.3).