acceptodds
Under review as a conference paper at ICLR 2027

Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

Abstract

As autonomous research systems become more capable, benchmarks must both evaluate their performance and support meaningful research progress. We introduce Auto-Research Arena, a multi-target benchmark for evaluating autonomous research systems and discovering ideas in language modeling. Arena supports diverse research objectives within a shared experimental environment by separating admissibility from improvement: model quality and resource constraints determine whether a result qualifies, while the selected target measures progress among admissible results. We instantiate Arena in language model pretraining with nine targets spanning model, data, training, and inference efficiency, and evaluate seven research harnesses using two foundation models. Arena clearly differentiates existing harnesses across research objectives, with several sophisticated methods outperformed by our simple sequential search baseline, indicating substantial headroom in current auto-research systems. Beyond evaluation, Arena reveals a substantially broader frontier of model and system ideas aligned with active research directions. Reusing these discoveries in a new quality-driven search yields lower training loss under matched data and FLOPs and lower held-out BPB at the same model size. Together, Arena provides a common framework for benchmarking autonomous research and expanding its scope in language modeling.

Primary area:
datasets and benchmarks
Keywords:
Auto-research, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mMWFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Auto-Research Arena, a benchmark that recasts Karpathy’s autoresearch LM-pretraining testbed as nine constrained optimization problems. Each track minimizes one efficiency target (parameters, tokens, updates, FLOPs/token, peak memory, training time, KV-cache memory, decode latency, or request latency). A candidate is admissible only if it reaches BPB < 1.05 and stays within resource limits on the other metrics.

Seven harnesses are compared with Claude Opus 5 and GPT-6, each with 100 charged experiments per track at 10 minutes of training per candidate, and results are aggregated with a margin-aware Elo. The harnesses are AutoScientists, Arbor, GEAR, Heuresis, the authors’ Sequential and Beam Search, and TPE.

Critical strengths
  1. The proposed task is well motivated. With the current development in auto research, the field needs a reliable benchmark to evaluate the performances.
  2. The objectives evaluated are broad, covering aspects from training to inference efficiency. The setup (Karpathy’s autoresearch language-modeling testbed) provides quantifiable feedback signals that are grounded in real tasks.
Critical weaknesses
  1. The paper claims to test auto research capabilities. However, they only test on the task of LLM pretraining. This is a very limited domain as auto research models are now employed in all major scientific domains. Also, a big part of auto research is research idea proposal and enhancement. The paper's evaluation only focuses on performance optimization. Overall, the focus is a very narrow task which could weaken the overall claim that an auto research benchmark is being developed.
  2. The reported results may not be very reliable. All results are reported against one single run. No confidence intervals are reported.
  3. The paper is reusing an established testbed with variations. Limited novelty is displayed.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Author Response

Overclaim autoresearch focus.

Arena is explicitly scoped to autonomous research in language modeling, as stated in the title and limitations, rather than research across all scientific disciplines.

We respectfully disagree that Arena only addresses performance optimization rather than idea proposal and refinement. Efficiency is the research objective; proposing and refining ideas is how agents pursue it. Section 4.2 documents the proposed ideas closely related to ongoing research directions, including looped Transformers and cross-layer K/V sharing.

Lack of multiple runs for ranking reliability.

We kindly refer to common question 1 in the general response for details.

Limited Novelty.

We respectfully disagree that building upon an established testbed, by itself, implies limited novelty. For example, MLE-bench (ICLR 2025) builds on existing Kaggle competitions, while the Automated LLM Speedrunning Benchmark (NeurIPS 2025) builds on the existing NanoGPT Speedrun benchmark.

Arena’s novelty lies in its multi-target evaluation framework: nine complementary efficiency targets with corresponding quality and resource constraints. Moving beyond validation-loss minimization, this framework enables controlled study of how research objectives shape harness performance and idea discovery. The resulting searches uncover a substantially broader range of ideas while also aligning with active research directions (Sec. 4.2). We further demonstrate that these discoveries can be adapted and combined in a subsequent search to improve model quality (Sec. 4.3).

Dear Reviewer mMWF,

We have clarified Arena’s contribution beyond single-objective evaluation and added repeated-search experiments to assess ranking stability. We have also addressed your other concerns and revised the manuscript accordingly. We would appreciate it if you could revisit these updates and consider updating your rating if they resolve your concerns. Thank you again.

Reviewer_GczJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a benchmark based on Karpathy's autoresearch testbed: a small LM pretraining program. The original testbed optimizes a single goal, validation loss. Arena instead casts each task as constrained optimization consisting of two research requirements (admissibility and improvement) and nine research objectives. To be admissible, a program must meet certain requirements like a performance threshold and target-specific resource limits.

The contributions are as follows:

  1. A benchmark built on Karpathy’s autoresearch testbed (~50M-parameter LM pretraining, 10-minute training runs) with nine efficiency targets. These cover model size (parameters), training efficiency (tokens, updates, FLOPs, peak memory, time) and inference efficiency (KV cache, decode latency, long-prompt latency).
  2. An evaluation of six LLM-based harnesses and a TPE baseline with Claude Opus 5 and GPT-6 Astra, with results aggregated by Elo.
  3. A catalogue of 32 findings grouped into nine idea families, and experiments reusing these discoveries to improve model quality.

Validation:

  1. Harness benchmark (Table 2). Each harness receives 100 charged experiments per objective. The authors report that no harness dominates, that their Sequential Search baseline ranks second with both foundation models, and that TPE ranks last.
  2. Idea frontier (Section 4.2). Discoveries are mapped to related prior work, e.g., looped Transformers with depth-specific residual gains, cross-layer K/V sharing and Adam-mini-style optimizer states.
  3. Idea reuse (Section 4.3). A quality-driven search given an idea library, with one-hour training, reduces training loss from 2.737 to 2.608 at matched data and from 2.700 to 2.649 at matched FLOPs. Held-out BPB falls from 0.9295 to 0.9112 at the same model size.
  4. Direct transfer (Table 4). Programs found with 10-minute training are retrained for 60 minutes. Some keep their advantage for 4-5x the search horizon, but by 60 minutes they are worse than the reference (e.g., 0.9478 vs 0.9295).
  5. Research-process studies (Section 4.4). These vary the BPB threshold and the per-candidate training budget, and continue searches across objectives.
Critical strengths
  1. Separating admissibility from improvement is a sound and reusable design. Single-metric benchmarks such as autoresearch recognize only one kind of progress. This proposed benchmark lets a smaller model or lower inference memory count as progress at fixed quality, within one environment. Non-target limits are set to the reference program’s own costs, so gains cannot be bought by regressing elsewhere. This matters to my assessment because the formulation is not tied to this task and is likely the paper’s most lasting contribution, even if the specific rankings change.
Critical weaknesses
  1. The reliability of the harness rankings is not established (Table 2). No variance is reported: each candidate is trained once with seed 42 and there is no indication of repeated search runs. Rankings also change substantially between foundation models (AutoScientists ranks first with Opus 5 but fifth of six with GPT-6). Budget use further confounds the GPT-6 results, since AutoScientists and Arbor used on average only 30.2 and 14.7 of 100 charged experiments. This is the main reason for my rating, since ranking harnesses is the benchmark’s core purpose, and the claims that Arena is discriminative and that simple search is competitive cannot currently be separated from run-to-run noise.
  2. The discovery and reuse claims exceed their evidence (Section 4.3). The quality-driven search is compared only with the unmodified reference, so without a matched search that lacks the idea library, the improvement cannot be attributed to the discovered ideas rather than to running a 305-candidate quality search. All results also come from a single ~50M-parameter testbed. This matters because the paper presents Arena as both evaluating and advancing auto-research, and the evidence for the second half shows that a library-guided search can improve on the reference, but not that the library is responsible for the improvement.
Questions for authors
  1. Repeated runs. Can you report at least three search runs per harness on a subset of objectives, with the resulting variance in per-objective ranks and Elo?
  2. Budget use. Why did GPT-6 AutoScientists and Arbor use only 30.2 and 14.7 charged experiments on average? Were these early stops, integration failures or harness decisions, and how do the rankings change if these runs are rerun?
  3. No-library control. Can you run the quality-driven search with the same budget and one-hour training but without the idea library? If the library-guided search clearly outperforms this control, I would raise my score. Was held-out BPB used both to select candidates and to report the final result?
  4. Thresholds. How was BPB < 1.05 chosen, what is the reference program’s BPB after 600 seconds, and do the rankings hold at BPB < 1.02?
  5. Correctness. Can you explain the mechanism behind the 1.000 KiB KV-cache result (Sequential Search, Opus 5), and confirm it passes the correctness checks?
4 comments
AuthorsOpenReview ↗

Author Response

Repeated Runs.

We kindly refer to common question 1 in the general response for details.

Reason for harnesses' early stopping.

We kindly refer to common question 2 in the general response for details.

No-library control.

No-library control run.
We thank the reviewer for this suggestion. We conducted a no-library control using the same search protocol and budget, including the same one-hour per-candidate training limit, with only the idea library removed. Results are shown below; the search with the idea library produced a program with lower training loss and validation BPB.

ProgramFinal training loss ↓Original validation BPB ↓
With idea library2.5440.9112
Without idea library2.5750.9173

Fresh BPB test. Yes, the original validation BPB was used for search feedback, candidate selection, and reporting the final result. To provide an independent evaluation, we first selected the best program from each search using the original validation BPB and then evaluated them on a new test set that had not been used for training or search. We repeated training and evaluation for both selected programs and the reference program using three seeds (0, 42, and 4242) for a more comprehensive evaluation. The table below reports mean ± standard deviation across training seeds.

ProgramFresh test BPB ↓
With idea library0.91097 ± 0.00124
Without idea library0.91621 ± 0.00030
Reference0.92900 ± 0.00119

Both searches improve upon the reference. The search with the idea library produces a better program than the matched no-library control, reducing training loss from 2.575 to 2.544 and validation BPB from 0.9173 to 0.9112. On the independent test set, it further achieves a 0.00524 lower mean BPB than the no-library control, consistently outperforming it across all three training seeds. For context, DeepSeek-V3 [1] reports BPB reductions of 0.003 (0.727 to 0.724) and 0.004 (0.656 to 0.652) in its load-balancing ablations. Muon [2] reports a training-loss reduction of 0.014 (2.734 to 2.720) and a validation-loss reduction of 0.023 (2.812 to 2.789) in its update-scaling study, while Sophia [3] reports a validation-loss reduction of 0.020 (2.921 to 2.901) on GPT-2 Small. These results show that the idea library provides a meaningful improvement over this quality-driven search.

Reference: [1] DeepSeek-AI, "DeepSeek-V3 Technical Report," 2024. https://arxiv.org/abs/2412.19437 [2] Liu et al., "Muon is Scalable for LLM Training," 2025. https://arxiv.org/abs/2502.16982 [3] Liu et al., "Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training," ICLR 2024. https://arxiv.org/abs/2305.14342

Explanation of Quality Threshold.

The reference program’s BPB after 600 seconds is 1.047. We use BPB < 1.05 to allow limited quality degradation while exploring structural and efficiency improvements.

Ranking at BPB < 1.02. A threshold of 1.02 defines a different optimization problem within the same framework. Since the reference program achieves 1.047 BPB, the search must first reduce BPB below 1.02 before any efficiency improvement can qualify as admissible progress. In contrast, at 1.05, the reference is already admissible, allowing the search to directly pursue efficiency improvements while maintaining the quality requirement. Therefore, these are two different optimization problems.

In Section 4.4, Table 5 provides a targeted ablation with Sequential Search and Beam Search explicitly run under both thresholds on parameter and memory efficiency. For both objectives, Sequential Search performs better at 1.05, whereas Beam Search performs better at 1.02. These results demonstrate that harness rankings can depend on the quality constraint. Additionally, a threshold at 1.05 allows broader idea discovery. At BPB < 1.05, Sequential Search’s parameter-efficiency winner removes an attention sublayer and shares K/V projections across layers, while its memory-efficiency winner shares value-embedding tables and replaces elementwise Adam second-moment estimates with one estimate per row. In contrast, its memory-efficiency winner at BPB < 1.02 primarily reduces activation storage through checkpointing and custom backward reconstruction. These examples suggest that the quality slack at 1.05 provides more room to explore structural model and optimizer changes.

Reason for harnesses' early stopping under GPT6.

We refer the reviewer to Common Question 2 in our general response for details. The lower experiment counts for AutoScientists and Arbor resulted from their own stopping rules. After rerunning both harnesses with GPT-6, the overall ranking remains unchanged.

Sequential Search KV-Cache results correctness.

We confirm it passes the correctness check. We kindly refer to common question 3 in the general response for details.

Dear Reviewer GczJ,

The requested no-library control shows that the library-guided search found a better program, with its advantage confirmed across three training seeds on fresh test data. We have also addressed your other concerns and revised the manuscript accordingly. We would appreciate your reconsideration of the rating if these additions resolve your concerns. Thank you again.

Dear authors,

Thank you for the additional experiments and the revision.

The no-library control and independent test set address my main concern about the reuse claim, and I appreciate that the comparison is now in Section 4.3. The repeated searches show the broad ranking pattern holds, though exact positions still shift between runs (e.g., AutoScientists' mean rank of 3.00 ± 2.00 in Table 11).

I’m raising my score to 3.

One thing I'd still like to see in the final version: a sentence in the paper, as acknowledged in the rebuttal, that harness rankings can depend on the quality threshold (Table 5).

Dear Reviewer GczJ,

Thank you for taking the time to review our responses. We are glad the additional experiments addressed your main concerns and appreciate your positive assessment of our work. We will explicitly clarify that harness rankings can depend on the quality threshold (Table 5) in the final version.

Reviewer_z87vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Auto-Research Arena, a benchmark for autonomous research systems in language modeling. Its design choice is to separate admissibility from improvement: each of nine objectives is a constrained optimization problem (Eq. 1) that minimizes one resource metric — parameters, tokens/updates/time/FLOPs/peak memory to reach quality, KV-cache state, decode latency, long-prompt latency — subject to a shared validation-BPB threshold of 1.05 plus target-specific resource caps. The instantiation extends Karpathy's autoresearch testbed (631M-token ClimbMix pool, one A100, 600 s candidate training, 100 charged experiments per search).

Three sets of results follow. (1) Seven harnesses (AutoScientists, Arbor, GEAR, Heuresis, plus the authors' Sequential and Beam Search baselines, plus a TPE baseline) are run on all nine objectives under Claude Opus 5 and GPT-6 Astra and aggregated with a margin-aware Elo (Appendix D); the authors report that no harness dominates, that TPE transfers poorly, and that simple Sequential Search ranks second under both LLMs. (2) A catalogue of 32 findings in nine idea families, with a worked example (looped Transformer with depth-specific residual gains) related to concurrent work. (3) Reuse experiments: an idea library seeds a quality-driven search that lowers training loss 2.737→2.608 at matched data, 2.700→2.649 at matched FLOPs, and held-out BPB 0.9295→0.9112 at the same 50.33M size; separately, six efficiency-discovered programs are retrained for 60 minutes. §4.4 adds studies of the quality threshold, evaluation-depth/search-breadth allocation, and cross-target continuation.

Critical strengths

The objective specification is precise, auditable, and well-motivated. Appendix A pins down each of the nine tracks to a level that would let a third party reimplement them: data-pool size (631.2M tokens), what counts as a token (consumption including repeats, not unique positions), what the FLOPs counter includes and excludes (matrix/conv/attention; not optimizer updates, elementwise ops, or normalization), peak memory as PyTorch-allocated rather than reserved, latency as median over 30 timed passes after three warmups, and a non-trivial inference-correctness contract (total-variation distance ≤ 0.05 across all 513 scored positions, with the request state required to own all retained KV data on-GPU). Appendix B.4's charging rule — an experiment is charged once a GPU-work-start marker fires, so crashes and OOMs consume budget — closes a loophole most agent benchmarks leave open, and Arena scoring the best recorded admissible candidate rather than the harness's self-selected endpoint is the right call. The appendix is also candid about scope in places where it would have been easy not to be (C.2.1, C.2.2).

Critical weaknesses
  1. The benchmark's deliverable is a ranking, but every number in it comes from a single run with no uncertainty quantification — and the paper's own data show the ranking is unstable. All searches use seed 42 (§4, line 395), one search per (harness, objective, foundation model); Table 2's "Charged experiments (mean)" is a mean over the nine objectives, not over repeats. Nowhere in 29 pages is a variance, standard deviation, confidence interval, or repeated run reported. Yet Elo is printed to one decimal and every headline claim is ordinal: AutoScientists "achieves the highest aggregate Elo" (1729.4 vs Sequential Search's 1687.6 — 41.8 points), Sequential Search "ranks second overall with both foundation LLMs", GEAR ranks first under GPT-6. Appendix D's 4,000 permutations average over comparison order only; they do not touch measurement noise, so the statistical machinery gives no handle on the quantity that matters. The instability is visible in the paper's own table: AutoScientists goes from 1st of 7 under Opus 5 (1729.4) to 5th of 6 under GPT-6 (1298.7), and GEAR from 4th (1622.4) to 1st (1835.3). The authors read this as complementary strengths, but at n=1 it is not separable from run-to-run variation, and the quantization makes that worse — training time is scored at 30-second checkpoints, where four Opus harnesses tie at exactly 270 s, so a single checkpoint of jitter flips three pairwise outcomes. The paper's own limitation ("noisy system measurements require care when interpreting small differences," line 741) concedes the premise while the main claims rest on exactly such differences. Table 8 shows what this costs: the text asserts "longer settings yield better BPB but more parameters," but its own 6,000 s row (7.86M params, BPB 1.1518) is both smaller and worse than the 4,800 s row (11.54M, 1.0607), breaking the stated trend at n=1 per setting.

  2. Search budget is not equalized across harnesses, which directly confounds the paper's flagship conclusion. §4 states "Each search receives 100 charged experiments," but Table 2's own column shows actual mean consumption ranging from 14.7 to 100.0: under GPT-6, Arbor 14.7, AutoScientists 30.2, Heuresis 86.8, Beam Search 95.8, Sequential Search 100.0, GEAR 100.0; under Opus 5, Arbor 54.0, AutoScientists 84.7, Sequential Search 87.7, Beam 96.9, GEAR/Heuresis/TPE 100.0. The score is the best recorded admissible value, a statistic that is monotone non-increasing in the number of experiments, so a harness that halts early is scored on a strictly truncated search and cannot catch up. This falls precisely on the claim the paper leads with: "SIMPLE ≠ WEAK" (Fig. 1) and "substantial headroom in current auto-research system design" are drawn from Sequential Search outranking harnesses that it outspent by up to 6.8× (100.0 vs Arbor's 14.7), and from TPE — which used the full 100.0 — being compared against AutoScientists at 30.2 under GPT-6 and 84.7 under Opus. Appendix B.6 specifies stopping policies only for the authors' own two baselines; nothing states why the third-party harnesses terminated early, whether early termination was a harness decision or an integration failure, or how many of their charged experiments were crashes. No budget-matched comparison is reported anywhere: Fig. 3 shows best-so-far trajectories for 3 of 9 objectives, Opus only. The trajectories are recorded (Appendix B.1), so a best-at-common-k table is computable from existing logs — but as submitted, the paper's central evaluative conclusion is confounded by a variable its own table documents.

Questions for authors
  1. Budget normalization. Can you report each harness's best admissible value at a common k (say k = 15, 30, 50) for all nine objectives and both foundation models, and recompute Elo at matched k? Does Sequential Search still rank second? Separately, why did Arbor stop at 14.7 and AutoScientists at 30.2 charged experiments under GPT-6 - harness-side stopping, integration error, or repeated failures?

  2. Ranking uncertainty. For even a subset (e.g. three harnesses x three objectives x three seeds), what is the spread of the target values, and what Elo interval does it imply? Given the 41.8-point Opus 5 gap between AutoScientists and Sequential Search, and the AutoScientists 1st-to-5th / GEAR 4th-to-1st swing across the two foundation models, what evidence distinguishes a harness-model interaction from run-to-run noise?

  3. The KV-cache track. Sequential Search reports 1.000 KiB against a 33,790 KiB reference - a 33,790x reduction, and an exactly round number. Appendix A defines this metric as a retained-allocation delta that "includes allocator rounding (approximately 1 MiB in the reference measurement)", yet ten of the thirteen reported harness values fall below 1 MiB. What is the measurement resolution, and are the sub-MiB orderings inside it? This matters for the ranking: since the Elo margin multiplier is a capped log-ratio, that single entry yields a near-maximal- weight win against all six rivals on that objective, so I would like to know whether Sequential Search's second-place finish survives its removal. Relatedly, the inference correctness check compares the cached path against the same model's full forward pass, which a model that has stopped attending to context passes trivially - with BPB < 1.05 (barely above the reference) as the only guard. What prevents this track from rewarding the elimination of context dependence, and did you inspect the 1.000 KiB program for this?

  4. Section 4.4 versus the appendix. The main text concludes that "Arena enables systematic study of how autonomous research develops", but C.2.1 states the budget-allocation prefixes "establish neither an optimal allocation nor a causal or matched-quality effect", and C.2.2 states there is "one search per receiving objective" and that without matched restarts they "establish neither a causal benefit of transfer nor a transfer-success rate". Please reconcile these, or restate Section 4.4 and the conclusion as illustrations rather than controlled studies. Relatedly, the text asserts that "longer settings yield better BPB but more parameters", but the 6,000 s row of Table 8 (7.86M parameters, BPB 1.1518) is both smaller and worse than the 4,800 s row (11.54M, 1.0606).

  5. Idea-frontier count and quality transfer. What is the operational criterion for a "distinct finding", and who applied it - a single author, or multiple annotators with a measured agreement? Since "32 findings versus Karpathy's 2" compares across different harnesses, budgets, and counting procedures, can you provide a matched- budget single-objective control run counted the same way? For Section 4.3, the matched-FLOPs gain is 1.9% and the BPB gain 1.97% from one run - what is seed variance at this scale? And how should readers weigh that single winning program against Table 4, where all six directly transferred programs are worse than the reference at 60 minutes (0.9365-0.9765 versus 0.9295), with the headline 43.3% / 45.4% savings obtained by linear interpolation of the reference curve?

4 comments
AuthorsOpenReview ↗

Author Response (part 1/3)

Budget normalization.

We added comparisons at caps of 15, 30, and 50 charged experiments across all nine objectives. The aggregate rankings are shown below; detailed per-objective results for each harness’s best admissible value will be included in the revised appendix. Sequential Search’s competitiveness is already evident at the 15-experiment cap: it ranks second under GPT-6 at every cap, including the full budget, and third under Opus 5 at the 15-, 30-, and 50-experiment caps, rising to second at the full budget.

The overall rankings are also broadly stable across budgets: for each foundation model, the same three harnesses occupy the top three positions at every cap and in the full-budget results, with changes mainly in their relative ordering.

MethodOpus 5: 15Opus 5: 30Opus 5: 50Opus 5: FullGPT-6: 15GPT-6: 30GPT-6: 50GPT-6: Full
AutoScientists2 (1686.2)1 (1730.3)2 (1685.8)1 (1729.4)4 (1408.1)4 (1363.6)5 (1340.7)5 (1298.7)
Sequential Search3 (1644.5)3 (1650.8)3 (1679.7)2 (1687.6)2 (1635.7)2 (1626.4)2 (1649.4)2 (1702.4)
Arbor1 (1689.2)2 (1729.9)1 (1718.0)3 (1632.4)1 (1661.3)3 (1551.4)3 (1498.1)3 (1459.6)
GEAR4 (1539.9)4 (1577.3)4 (1630.1)4 (1622.4)3 (1631.1)1 (1830.6)1 (1839.3)1 (1835.3)
Beam Search6 (1340.6)5 (1370.6)5 (1411.1)5 (1438.6)6 (1282.5)6 (1284.7)4 (1358.4)4 (1423.3)
Heuresis5 (1539.6)6 (1366.8)6 (1331.3)6 (1350.4)5 (1381.3)5 (1343.3)6 (1314.0)6 (1280.6)
TPE7 (1059.9)7 (1074.3)7 (1044.0)7 (1039.2)————

Reason for harnesses' early stopping under GPT6.

We kindly refer to common question 2 in the general response for details. Both harnesses stopped according to their own stopping rules. Rerunning them did not change the overall ranking.

Ranking uncertainty.

Ranking uncertainty Analysis. We refer the reviewer to Common Question 1 for the multi-run search results and ranking stability analysis.

Cross-model ranking differences. The changes between Opus 5 and GPT-6 appear partly related to differences in harness behavior across foundation models. Under GPT-6, AutoScientists frequently triggers its stagnation-based stopping rule, conducting only 30.2 experiments on average, compared with 84.7 under Opus 5. GEAR, meanwhile, uses the full budget under both models and performs substantially better relative to other harnesses under GPT-6. Despite these changes, the relative ordering of Sequential Search, Arbor, Beam Search, and Heuresis is preserved across both models. We also reran AutoScientists and Arbor under GPT-6, and all six harnesses retained their original rankings (please refer to Common Question 2 for details). Together, these results suggest that the cross-model ranking differences partly reflect consistent differences in harness behavior, particularly AutoScientists’ earlier stopping under GPT-6.

AuthorsOpenReview ↗

Author Response (part 2/3)

Questions regarding Sequential Search KV-Cache results.

KV-Cache measurement resolution. PyTorch’s native CUDA caching allocator rounds small allocations in 512-byte (0.5KiB), which is also our measurement resolution. The approximately 1 MiB mentioned in Appendix A refers to the allocator-rounding contribution in the reference program’s measurement, not a universal measurement resolution. The reported sub-MiB orderings are well separated at this granularity: the smallest gap within either foundation-model group is 10 KiB, between 58 and 68 KiB under Opus 5, which is 20 times the allocation increment. We will clarify this distinction in the revised manuscript.

Ranking excluding the KV-cache objective. We recomputed the Opus 5 Elo rankings after excluding the KV-cache objective, using the same scoring procedure across the remaining eight objectives. Sequential Search remains second, as shown in the table below (we report rank(Elo) for each method). Its second-place ranking therefore does not depend on the exceptional KV-cache result.

MethodWith KV-cache (9 objectives)Without KV-cache (8 objectives)
AutoScientists1 (1729.4)1 (1731.7)
Sequential Search2 (1687.6)2 (1646.9)
Arbor3 (1632.4)3 (1639.8)
GEAR4 (1622.4)4 (1632.8)
Beam Search5 (1438.6)5 (1432.7)
Heuresis6 (1350.4)6 (1362.7)
TPE7 (1039.2)7 (1053.4)

Context dependence of the 1kiB kv-cache program. The 1 kiB program retains attention to previous tokens. As detailed in Common Question 3 of our general response, it stores a packed history of 625 token IDs and reconstructs K/V during decoding. Cross-layer K/V sharing reduces reconstruction cost, while each layer retains its own queries and attends to previous tokens. The small memory footprint therefore comes from compact history storage and recomputation.

Section 4.4 versus the appendix confusion.

Scope of Section 4.4. Our statement that Arena “enables systematic study” concerns its capability as an experimental framework. The budget-allocation experiment compares search outcomes under different allocations of a fixed total compute budget, while the cross-target continuations demonstrate examples of accumulating improvements across objectives. These findings illustrate the framework’s use; the scope stated in Appendix C.2 concerns stronger conclusions about optimal allocation and the causal benefit or success rate of transfer. We will clarify their scope in Section 4.4 and replace “controlled studies” with “illustrative analyses” in the conclusion.

Correction to the Table 8 discussion. We agree that the original wording incorrectly suggests a monotonic trend. We will revise both Section 4.4 and the table caption to state: “Compared with the shortest, 1,200-s setting, all three longer settings produce larger models with better BPB, but neither metric varies monotonically with per-candidate training time.” Specifically, the 6,000-s setting produces a smaller model with worse BPB than the 4,800-s setting. We will revise the paper accordingly.

Idea-frontier count and quality transfer

Idea selection Criterion. We inspect the source-code differences between each winning program and its starting reference and count distinct model, optimization, or systems mechanisms active in the winning program, rather than changes to an existing mechanism’s numerical settings. For example, replacing independent Transformer blocks with a shared block reused across depth qualifies. Merely reducing the number of independent blocks does not qualify because it changes only the existing architecture’s depth setting. Within each objective, different numerical configurations of the same mechanism count as one idea. In practice, we defined the above criteria and used coding agents to identify findings and merge duplicates. Human authors then verified the agent-generated results to produce the final catalogue.

Breadth of discovered ideas. We will remove the numerical comparison with Karpathy’s two findings from Section 4.2 and instead emphasize the breadth of the discoveries. These range from model architecture and optimization to training-memory management and inference systems, with examples such as looped Transformers, reduced optimizer-state storage, and K/V sharing and reconstruction. Arena therefore recognizes progress across a broader range of research goals than a single-objective benchmark.

AuthorsOpenReview ↗

Author Response (part 3/3)

Clarification on Table 4. Table 4 directly extends programs discovered under 10-minute training to 60 minutes by stretching their learning-rate schedules by a factor of six, without further search. This simple extension does not improve final BPB over the reference: all six programs perform worse at 60 minutes. We therefore conducted a new search to combine the discovered ideas and adjust the necessary hyperparameters for longer training. The resulting program achieves better final BPB: 0.9112 versus the reference’s 0.9295.

The headline savings measure the resources needed to reach the same BPB during training. The 43.3% token saving compares 1.004B tokens for the discovered program with an estimated 1.771B tokens for the reference at the same BPB of 0.9593. Similarly, the 45.4% time saving compares 25 minutes with an estimated 45.8 minutes at the same BPB of 0.9647. The reference costs are estimated by linear interpolation between measured checkpoints. These results show that the transferred programs can reach intermediate quality levels with fewer resources, even though the reference achieves better quality by 60 minutes.

Seed variance of Idea Combination Search. We retrained the baseline and selected design with two additional seeds, 0 and 4242, without further search or tuning. The improvements persist in both additional seeds. Matched-token and matched-FLOPs budgets are fixed at the original selected-design run’s endpoints (seed 42); final BPB uses the original stopping rule.

Metric ↓Reference ProgramSelected designMean reductionRelative reduction
Train loss, matched FLOPs2.613 ± 0.0030462.545 ± 0.0018320.067652.589%
Train loss, matched tokens2.672 ± 0.0032292.545 ± 0.0018320.12694.749%
Final BPB0.9304 ± 0.0010290.9119 ± 0.00068400.018471.985%

Values are mean ± SD across three seeds. Reductions are computed from unrounded means.

Reporting correction. We identified transcription errors in the matched-budget training-loss results reported in Section 4.3 and will correct them in the revised manuscript. The corrected values are 2.675 → 2.544 at matched tokens (4.9% reduction) and 2.617 → 2.544 at matched FLOPs (2.8% reduction). The held-out BPB result (0.9295 → 0.9112) remains unchanged. These corrections do not alter our conclusions, which are further supported by the additional seed evaluations above.

Timing resolution for training time to target.

We re-evaluated each method’s original winning program for the training-time objective using Opus 5, checking BPB at 5-second rather than 30-second intervals. We then recomputed the aggregate Elo ratings using the updated time-to-quality measurements. The original four-way tie at 270 s separates into 255 s (GEAR and Beam Search), 260 s (Arbor), and 270 s (Heuresis). All seven harnesses retain their original overall rank, with Elo changes of at most 6.24 points, showing that the aggregate ranking remains stable under finer validation intervals.

Overall rank (unchanged)MethodTime: 30-s resolutionTime: 5-s resolutionOriginal EloUpdated Elo
1AutoScientists240 s240 s1729.371726.12
2Sequential Search210 s205 s1687.611693.85
3Arbor270 s260 s1632.371629.82
4GEAR270 s255 s1622.411625.98
5Beam Search270 s255 s1438.611443.24
6Heuresis270 s270 s1350.451345.51
7TPE510 s500 s1039.171035.49

Dear Reviewer z87v,

We have added repeated-search results reporting rank and Elo variability, common-budget comparisons, and an analysis showing that excluding the KV-cache objective leaves the overall ranking unchanged. We have also addressed your other concerns and revised the manuscript accordingly. We would appreciate it if you could review these additions and consider updating your rating if they resolve your concerns. Thank you again for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (part 1/2)

We thank the reviewers for their constructive feedback. We are encouraged that they find Arena well motivated (z87v, mMWF) and recognize its precise and auditable objective specifications (z87v), its sound and reusable separation of admissibility from improvement (GczJ), and its broad coverage of training and inference efficiency objectives (mMWF).

We will incorporate the additional experimental results, clarifications, and corrections presented below into the revised manuscript.

We address the common questions from the reviewers as follows:

Common Question 1 (z87v, GczJ, mMWF): Repeated searches and ranking stability.

Given the scale of the original benchmark and the limited token credits and time available during the rebuttal period, we assess ranking stability through two complementary experiments using Opus 5: (1) three independent searches across three harnesses and three objectives, and (2) three searches across all seven harnesses on one objective.

(1) Repeated searches across three objectives. We evaluated Sequential Search, Beam Search, and GEAR on the training-updates, training-FLOPs, and VRAM objectives across three independent runs. Elo is recomputed separately for each run across these three objectives.

MethodMetricRun 1Run 2Run 3
Sequential SearchRank (Elo)1 (1560.1)1 (1565.2)1 (1541.6)
Updates ↓295236282
FLOPs (M/token) ↓56.6657.8564.27
VRAM (GiB) ↓0.5670.8310.523
Beam SearchRank (Elo)3 (1405.8)2 (1509.5)2 (1481.2)
Updates ↓335336286
FLOPs (M/token) ↓69.8565.8767.24
VRAM (GiB) ↓0.9510.7460.782
GEARRank (Elo)2 (1534.1)3 (1425.3)3 (1477.2)
Updates ↓306315322
FLOPs (M/token) ↓62.2372.8455.97
VRAM (GiB) ↓0.4340.9650.941

Runs 2 and 3 reproduce the same ranking, and Sequential Search ranks first in all three runs. Sequential Search achieves a mean Elo of 1555.6 ± 12.4, compared with 1478.9 ± 54.4 for GEAR and 1465.5 ± 53.6 for Beam Search. Despite variation in individual objective values, Sequential Search consistently leads the aggregate ranking with substantially lower run-to-run variability.

(2) Repeated searches across all seven harnesses. We additionally evaluated all seven harnesses on the VRAM objective across three runs, using a common cap of 40 candidate evaluations. The table reports each run's VRAM and rank, together with the mean and standard deviation of both quantities.

MethodRun 1Run 2Run 3Mean VRAM ± SDMean rank ± SD
AutoScientists0.671 (1)1.295 (3)2.289 (5)1.418 ± 0.8163.00 ± 2.00
Sequential Search0.952 (4)0.937 (2)1.540 (2)1.143 ± 0.3442.67 ± 1.15
Arbor0.680 (2)0.298 (1)0.362 (1)0.447 ± 0.2051.33 ± 0.58
GEAR0.770 (3)3.637 (5)2.189 (3)2.199 ± 1.4343.67 ± 1.15
Beam Search1.913 (5)5.605 (6)4.640 (6)4.053 ± 1.9155.67 ± 0.58
Heuresis2.071 (6)3.409 (4)2.199 (4)2.560 ± 0.7384.67 ± 1.15
TPE6.388 (7)16.477 (7)6.388 (7)9.751 ± 5.8257.00 ± 0.00

Several ranking patterns persist across all three runs. Arbor remains in the top two, Sequential Search achieves the second-best average rank, Beam Search ranks fifth or sixth, and TPE ranks last throughout. The standard deviation of VRAM is also near or below 1 GiB for most harnesses, suggesting relatively consistent performance across runs. Among the 21 method pairs, Runs 2 and 3 preserve 16 (76.2%) and 15 (71.4%) of the original Run 1 orderings, respectively. Moreover, 14 of 21 pairs (66.7%) retain the same ordering across all three runs, compared with an expected 25% under independent random rankings. These results support the stability of the broader ranking patterns across independent searches.

Our original benchmark spans nine objectives and two foundation models, involving thousands of candidate experiments, making full replication computationally expensive. Reporting results from single searches is also not unique to our work: GEAR [1], for example, reports its main comparisons from one search per method without repeated-search variance estimates. Beyond this precedent, we conducted two complementary repeat studies to directly assess ranking stability. These results reproduce Sequential Search's leading position in the three-objective comparison and preserve most pairwise orderings across all seven harnesses on VRAM, providing additional empirical support for the reliability of our main ranking conclusions.

Reference: [1] Jeddi et al., "GEAR: Genetic AutoResearch for Agentic Code Evolution," arXiv:2605.13874, 2026.

AuthorsOpenReview ↗

General Response (part 2/2)

Common Question 2 (z87v, GczJ): Early Stopping of harnesses under GPT6

The lower experiment counts for AutoScientists and Arbor using GPT-6 resulted from the methods’ own stopping rules, despite a budget of 100 charged launches per task. AutoScientists triggered its stagnation rule—no KEEP decisions in the last ten experiments—on eight of nine tasks; the remaining KV-cache task exhausted its budget, yielding an average of 30.2 launches. Arbor terminated exploration based on its coordinator’s diminishing-returns criterion, guided by an approximately 5% incremental-improvement threshold for merging candidates. For example, its Data search stopped after five candidates plus the reference because follow-up candidates did not meet that threshold; the best additional improvement was only 0.20%. These method-controlled termination decisions produced Arbor’s average of 14.7 launches.

To assess whether the original rankings depended on these particular runs, we reran Arbor and AutoScientists and recomputed Elo using the new results while keeping the other harness results unchanged. In the reruns, Arbor averaged 16.4 charged evaluations per objective compared with 14.7 originally, and AutoScientists averaged 22.2 compared with 30.2 originally.

HarnessOriginal EloUpdated EloRank (before → after)
GEAR1835.31824.71 → 1
Sequential Search1702.41695.52 → 2
Arbor1459.61484.93 → 3
Beam Search1423.31385.84 → 4
AutoScientists1298.71333.45 → 5
Heuresis1280.61275.76 → 6

All six harnesses retain their original rankings. The reruns confirm that early stopping is a consistent behavior of these harnesses.

Common Question 3 (z87v, GczJ): Sequential Search KV-Cache results correctness

The candidate retains compact token IDs instead of K/V tensors and reconstructs K/V during decoding. In the eight-block model, block 0 uses its own K/V, blocks 1–2 share block 1’s K/V, and blocks 3–7 share block 3’s K/V, with attention windows of 1, 239, and 384 previous tokens, respectively. Each block retains its own queries, output projection, MLP, and residual updates. These bounded dependencies require only 625 token IDs, losslessly packed into 1 KiB including bookkeeping. Sharing enables reconstruction without recomputing every upper block’s historical outputs.

The candidate passes the benchmark’s correctness checks against the same model’s full-sequence forward pass, using identical sharing and attention windows. It also meets the quality threshold (BPB 1.04748 < 1.05) and request-latency ceiling (649 ms < 750 ms).

Connection to prior work. The 1 KiB program explores two active research directions: reducing redundancy across attention layers and trading computation for memory. CLA [1] and LiSA [2] exploit cross-layer redundancy through K/V sharing and attention-weight sharing, respectively. Our program uses nonuniform K/V sharing while retaining layer-specific queries. It also reconstructs K/V rather than caching them, as explored in KV-Direct [3], but stores compact token IDs instead of residual representations. This combination enables the program to reduce persistent decoding state to 1 KiB while satisfying the quality and latency constraints.

Reference: [1] Brandon et al., "Reducing Transformer Key-Value Cache Size with Cross-Layer Attention," NeurIPS 2024. [2] Mu et al., "Cross-layer Attention Sharing for Pre-trained Large Language Models," TACL 2026. [3] Qasim et al., "The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference," 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.