- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RSI-Index, a benchmark measuring autonomous LLM R&D capabilities across four tasks: compression, language model training, harness engineering, and post-training. It evaluates 19 frontier agent systems and analyzes their research performance, behavioral patterns, and progress over time.
Critical strengths
-
Important and timely benchmark. The paper directly evaluates autonomous AI R&D capabilities through realistic, long-horizon research tasks with independently verified outcomes.
-
Insightful behavioral analysis. The findings on premature termination, experimental efficiency, and difficulty accumulating improvements provide valuable insights into current limitations of autonomous research agents.
Critical weaknesses
-
Limited connection to recursive self-improvement. The benchmark measures one-off improvements to existing AI systems, but does not demonstrate whether agents can continuously improve their own research capabilities. Therefore, extrapolating these results to predict RSI milestones lacks sufficient justification. Overall, I think the benchmark is timely and valuable, but extending the results to RSI seems not convincing enough.
-
Limited benchmark representativeness and calibration. The index relies on only four tasks, with reference results obtained under potentially different evaluation and resource settings. It is unclear whether the resulting scores, particularly the 0.6 milestone, reliably reflect frontier AI research capabilities. Furthermore, I think using the
-
Insufficient statistical reliability and speculative extrapolation. Each agent is evaluated with only one research run per task, making rankings sensitive to execution variability. Extrapolating these results to predict autonomous frontier AI development timelines is insufficiently supported.
Questions for authors
-
How sensitive are the model rankings and proposed RSI milestones to alternative task selections, reference values, and normalization schemes?
-
Could the authors conduct multiple independent research runs for representative agents to quantify performance variance and the reliability of the projected trends?
3 comments
Thank you again for the constructive feedback, we hope our response, including the sensitivity analysis, addresses your concerns before the discussion phase comes to a close. We're happy to clarify anything further.
Response to authors
Thanks for the response. I slightly support the acceptance for this paper. But in my view, RSI can only be truly studied with frontier-level compute, since the conclusions in toy tasks may not be generalized to frontier models (like when you use fable 5.2 to develop fable 5.5). Anyway, I think the paper is interesting but a little bit overclaimed. I will keep my score.
We thank the reviewer for recognizing RSI-Index as important and timely.
Connection to RSI Frontier labs tie RSI to automated AI R&D: OpenAI defines a model “capable of recursively self improving” as “fully automated AI R&D” [1], and Anthropic’s automated AI R&D threshold “is intended to capture the onset of dramatic recursive self-improvement” [2]. A model building its successor can’t be measured at the frontier without frontier-scale compute, so we measure this capability on smaller-scale stages of LLM development. The improvements are not one-off: each run is 12–30 hours of iterative experiments (Sections 5.2–5.4), and the index tracks whether each new release is a better researcher than the last (Figure 1), the measurable counterpart of continuous improvement. We will make this explicit in Section 1.
Representativeness and calibration Our four tasks are distinct stages of LLM development, and labs measure this capability with tasks of the same scale: Anthropic re-runs a small-model training task on every release [3], and OpenAI evaluates GPT-6 Astra on NanoGPT, a single-H100 LM-training task [4]. Results at this scale are relevant to the frontier: the Muon optimizer [5] came from NanoGPT speedrunning and was used to train Kimi K2, and byte-pair encoding, a common LLM tokenizer, began as a compression algorithm [6].
Our references used more resources than the agents receive. For a fairer comparison, we adapted the January 2026 modded-nanogpt record to LM Training and scored it with the same harness and hardware: 0.8458 BPB versus the 0.85 reference, and only one of 19 agents beats it. The 0.6 milestone means gains over the references comparable to AlexNet, BERT or the move from LSTMs to Transformers (Section 5.1). Future versions will add stages such as data curation.
Extrapolation We extrapolate a capability milestone, the date at which the trend reaches 0.6 (agents clearly beating the best known results), conditional on the trend continuing; it is not a prediction of RSI itself. The trend is robust: 0.30 index points per year for the leading developer (95% CI 0.17–0.42; Figure 1, Table 5), significant under the alternatives below.
Sensitivity We varied the three choices as requested, extending Appendix C.
The same top three (Opus 5.5, Fable 5.1, Opus 5) lead in every case except without Post-training, where Grok 4.6 replaces Opus 5.
Independent runs We agree repeated runs are valuable. Runs are expensive, so each system has one research run per task, but each task score limits evaluation noise: Compression is an exact compressed size, LM Training the median of three retrainings, and Harness Engineering and Post-training the mean of three evaluations. The trend across releases, each from separate runs, remains significant. We will add rank intervals to Table 2, and in future versions we plan to repeat runs for representative models.
References
[1] OpenAI, “Preparedness Framework (Version 2),” 2025.
[2] Anthropic, “Responsible Scaling Policy, Version 3.4,” 2026.
[3] M. Favaro and J. Clark, “When AI builds itself,” Anthropic, 2026.
[4] OpenAI, “GPT-6 Astra System Card,” 2026.
[5] K. Jordan et al., “Muon: An optimizer for hidden layers in neural networks,” 2024.
[6] R. Sennrich, B. Haddow and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016.