acceptodds
Under review as a conference paper at ICLR 2027

RSI-Index: Measuring Autonomous LLM R&D in Frontier Agent Systems

Abstract

Recursive self-improvement requires AI systems that can do AI research autonomously, but this capability is measured primarily by model developers, on their own models and internal tasks. We introduce RSI-Index, an external benchmark that measures how much of the research behind developing LLMs an agent can do on its own. Its four tasks cover pre-training, post-training and harness engineering, and each scores the agent's result against the best known result. Across 19 models from eight providers, the strongest, Claude Opus 5.5, scores 0.373. Performance has risen steadily across model releases, although gains so far come mainly from combining established techniques. If current trends continue and generalize to frontier development, within a year, autonomous AI agents could drive improvements in frontier language-model performance comparable to major human-led advancements.

Primary area:
datasets and benchmarks
Keywords:
ai, llm, agents, rsi, autoresearch, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7AinOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RSI-Index, a benchmark measuring autonomous LLM R&D capabilities across four tasks: compression, language model training, harness engineering, and post-training. It evaluates 19 frontier agent systems and analyzes their research performance, behavioral patterns, and progress over time.

Critical strengths
  • Important and timely benchmark. The paper directly evaluates autonomous AI R&D capabilities through realistic, long-horizon research tasks with independently verified outcomes.

  • Insightful behavioral analysis. The findings on premature termination, experimental efficiency, and difficulty accumulating improvements provide valuable insights into current limitations of autonomous research agents.

Critical weaknesses
  • Limited connection to recursive self-improvement. The benchmark measures one-off improvements to existing AI systems, but does not demonstrate whether agents can continuously improve their own research capabilities. Therefore, extrapolating these results to predict RSI milestones lacks sufficient justification. Overall, I think the benchmark is timely and valuable, but extending the results to RSI seems not convincing enough.

  • Limited benchmark representativeness and calibration. The index relies on only four tasks, with reference results obtained under potentially different evaluation and resource settings. It is unclear whether the resulting scores, particularly the 0.6 milestone, reliably reflect frontier AI research capabilities. Furthermore, I think using the

  • Insufficient statistical reliability and speculative extrapolation. Each agent is evaluated with only one research run per task, making rankings sensitive to execution variability. Extrapolating these results to predict autonomous frontier AI development timelines is insufficiently supported.

Questions for authors
  • How sensitive are the model rankings and proposed RSI milestones to alternative task selections, reference values, and normalization schemes?

  • Could the authors conduct multiple independent research runs for representative agents to quantify performance variance and the reliability of the projected trends?

3 comments

We thank the reviewer for recognizing RSI-Index as important and timely.

Connection to RSI Frontier labs tie RSI to automated AI R&D: OpenAI defines a model “capable of recursively self improving” as “fully automated AI R&D” [1], and Anthropic’s automated AI R&D threshold “is intended to capture the onset of dramatic recursive self-improvement” [2]. A model building its successor can’t be measured at the frontier without frontier-scale compute, so we measure this capability on smaller-scale stages of LLM development. The improvements are not one-off: each run is 12–30 hours of iterative experiments (Sections 5.2–5.4), and the index tracks whether each new release is a better researcher than the last (Figure 1), the measurable counterpart of continuous improvement. We will make this explicit in Section 1.

Representativeness and calibration Our four tasks are distinct stages of LLM development, and labs measure this capability with tasks of the same scale: Anthropic re-runs a small-model training task on every release [3], and OpenAI evaluates GPT-6 Astra on NanoGPT, a single-H100 LM-training task [4]. Results at this scale are relevant to the frontier: the Muon optimizer [5] came from NanoGPT speedrunning and was used to train Kimi K2, and byte-pair encoding, a common LLM tokenizer, began as a compression algorithm [6].

Our references used more resources than the agents receive. For a fairer comparison, we adapted the January 2026 modded-nanogpt record to LM Training and scored it with the same harness and hardware: 0.8458 BPB versus the 0.85 reference, and only one of 19 agents beats it. The 0.6 milestone means gains over the references comparable to AlexNet, BERT or the move from LSTMs to Transformers (Section 5.1). Future versions will add stages such as data curation.

Extrapolation We extrapolate a capability milestone, the date at which the trend reaches 0.6 (agents clearly beating the best known results), conditional on the trend continuing; it is not a prediction of RSI itself. The trend is robust: 0.30 index points per year for the leading developer (95% CI 0.17–0.42; Figure 1, Table 5), significant under the alternatives below.

Sensitivity We varied the three choices as requested, extending Appendix C.

VariedAlternativesKendall’s τ\tau with published ranking0.6 crossing (Anthropic)
ReferencesAll shifted to score 0.4 or 0.6 instead of 0.50.99, 0.99Mar 2027, Feb 2028
Task selectionDropping one task0.87–0.96Mar–Nov 2027
NormalizationLinear instead of logarithmic scale0.98Oct 2027
Ceilings0.4 or 0.8 BPB instead of 0.6, and 100% or 95% instead of 99%0.95–0.96Jul 2027

The same top three (Opus 5.5, Fable 5.1, Opus 5) lead in every case except without Post-training, where Grok 4.6 replaces Opus 5.

Independent runs We agree repeated runs are valuable. Runs are expensive, so each system has one research run per task, but each task score limits evaluation noise: Compression is an exact compressed size, LM Training the median of three retrainings, and Harness Engineering and Post-training the mean of three evaluations. The trend across releases, each from separate runs, remains significant. We will add rank intervals to Table 2, and in future versions we plan to repeat runs for representative models.

References

[1] OpenAI, “Preparedness Framework (Version 2),” 2025.
[2] Anthropic, “Responsible Scaling Policy, Version 3.4,” 2026.
[3] M. Favaro and J. Clark, “When AI builds itself,” Anthropic, 2026.
[4] OpenAI, “GPT-6 Astra System Card,” 2026.
[5] K. Jordan et al., “Muon: An optimizer for hidden layers in neural networks,” 2024.
[6] R. Sennrich, B. Haddow and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016.

Thank you again for the constructive feedback, we hope our response, including the sensitivity analysis, addresses your concerns before the discussion phase comes to a close. We're happy to clarify anything further.

Reviewer_7AinOpenReview ↗

Response to authors

Thanks for the response. I slightly support the acceptance for this paper. But in my view, RSI can only be truly studied with frontier-level compute, since the conclusions in toy tasks may not be generalized to frontier models (like when you use fable 5.2 to develop fable 5.5). Anyway, I think the paper is interesting but a little bit overclaimed. I will keep my score.

Reviewer_23qnOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper creates a suite of four tasks related to LLM development that the authors take strong LLM agent performance on to be indicative of a model capable of recursive self-improvement. The authors build development and evaluation environments for these tasks and evaluate 19 models within them. They find strong performance differences across models, with models rarely achieving state-of-the-art results. They extrapolate the progress over time on their index for frontier developers to estimate when models will develop solutions better than current-day SOTA results and compare their forecasts to frontier developer estimates of automated research timelines. They provide additional analysis of alternative configurations and the behavior of agents in their experiments.

Critical strengths

The post-training task is well designed, allowing full internet access and assessing completed artifacts on held-out data. It provides a realistic environment for post-training and a strong measure of LLM agent ability at the task.

Critical weaknesses

This work has little actual relation to recursive self-improvement (RSI). This work covers neither self-improvement nor, as mentioned in the limitations, a closed development loop, the two key elements of RSI. The task score also tests agent performance with respect to the present-day state of the art, so their extrapolation represents when agents will be able to develop solutions that exceed current capabilities rather than push the frontier forward, which is a more challenging task that the authors ultimately aim to study.

The first two tasks test capabilities which are not critical to modern frontier model development. These tasks are rarely the subject of current AI research, so I wouldn’t necessarily expect development capabilities on them to translate to frontier LLM development. The age of these state-of-the-art results and agents’ ability to reach near or above the reference in as little as 30 minutes is indicative of the lack of serious study or the simplicity of combining existing methods on these problems.

Questions for authors

What is the novelty of these benchmarks over existing work? Your related work points to past benchmarks covering similar tasks, so what do you see as the gap in the literature that your tasks fill, and why would a meta-analysis of existing work be insufficient to study your research questions?

For the harness engineering task, do agents generally modify aspects of the harness beyond the prompt? If not, this task is fairly simple and there are already strong methods like GEPA.

How do you interpret model performance nearing or exceeding the state of the art on some of your tasks?

Can you perform some checks to see how aware the models already were of top solutions to the research tasks? Analyzing chain-of-thought summaries, if they are saved in your logs, or prompting the agents about their awareness of the tasks up front would be helpful to check for data contamination.

5 comments

We thank the reviewer for the careful review and for recognizing the design of our Post-training task.

Relation to RSI Our paper introduces RSI as an AI system that builds a more intelligent successor, which cannot be measured directly at the frontier because it requires frontier-scale compute. Frontier labs instead tie RSI to automated AI R&D. OpenAI defines models “capable of recursively self improving” as “fully automated AI R&D” [1], and Anthropic states that its automated AI R&D threshold “is intended to capture the onset of dramatic recursive self-improvement” [2]. RSI-Index measures this capability, not the closed loop, and tracks its improvement across model releases from multiple providers.

Relevance of the tasks We acknowledge these tasks are smaller than frontier-scale training, but they test the same research judgement, and many frontier methods were first found on small tasks like these.

  • Anthropic measures progress toward recursive self-improvement with a similar small-model training task, re-run on every model release [3], and OpenAI evaluates GPT-6 Astra on NanoGPT, a single-H100 LM-training task “intended to measure LLM pretraining ability” [4].
  • The Muon optimizer [5] was established in NanoGPT speedrunning, which inspired our LM Training task, before it was used to train frontier models such as Kimi K2, and GPT-4’s performance was predicted from models trained with “no more than 1/1,000th the compute” [6].
  • Compression is tightly coupled to language modeling [7]: byte-pair encoding began as a compression algorithm [8], and sparse attention, whose variants are used in frontier models such as MiniMax-M3 [9], was introduced with enwik8 results [10].
  • Both the Compression and LM Training tasks reward efficient language modeling, a key driver of AI progress [11].

Clarifying the research budget and the reference Both Compression and LM Training allow 24 hours of research; the 30-minute limit applies only to retraining the submitted LM Training program during scoring. In other words, the agent has 24 hours to optimize a 30-minute training.

To address the age of the LM Training reference, we adapted and scored a recent human result: the January 2026 modded-nanogpt speedrun record. It is the latest record that can be adapted to our single-GPU task; later records use eight GPUs. The record only slightly beats our reference (0.8458 vs 0.85 BPB), which suggests the reference remains a strong bar. The record itself combines existing methods, yet only one of 19 agents beats it.

Novelty and meta-analysis Most of our tasks build on established task types; our contribution is running every system on tasks anchored to the best known results, so that improvement can be tracked across releases and providers without the measurement saturating. Published results cannot be pooled for this, and existing benchmarks cannot simply be re-run, because many are already saturated or are not about LLM development.

In terms of novelty, Harness Engineering is, to our knowledge, the first benchmark task that asks an agent to build a grader harness. Building evaluations is a core step of AI research itself: OpenAI lists writing “evaluations to judge model performance” among the steps its researchers perform [12].

Today, AI research capability is measured mainly by frontier developers, on their own models and internal tasks. For example, Anthropic’s measures cover only its own models, and it publishes “limited amounts on these internal measures for competitive reasons” [13]. Thus, progress can’t be compared across labs or tracked independently. RSI-Index fills this gap: it evaluates 19 systems from eight providers under the same task configs, and tracks each provider’s improvement over time.

Tracking progress over time requires tasks that (1) are relevant to LLM development, (2) are anchored to the best known result, (3) are not saturated, and (4) run under the same configuration on each model. Existing benchmarks each miss at least one: most target long-horizon software engineering or ML broadly rather than LLM development (e.g., FrontierSWE, MLE-bench, AIRS-Bench); others score pass/fail against fixed targets, the starting code or budget-matched experts rather than frontier references (e.g., SWE-Marathon, RE-Bench); many are saturated, as OpenAI notes of its own earlier measures, which “were also becoming saturated” [4]; and published results use different scaffolds, budgets and benchmark versions, so they cannot be compared across providers or over time. RSI-Index is designed to meet all four criteria, building on established problems where possible (the Hutter Prize setting, nanogpt language modeling and PostTrainBench’s design).

Harness Engineering and GEPA In Harness Engineering, the agent improves the harness of a fixed judge model that grades full-stack web applications built by coding agents. Models modify far more than the prompt: all 19 replace the starting judge harness with their own code (up to 5000 lines of code), and 16 models built a browser agent that operates the application. A better prompt is insufficient, because the task asks the judge to decide whether a web application supports each step of a user workflow, which is hard to do without operating it.

To test prompt optimization, we ran GEPA (with Claude Opus and GPT-6.1 Sol optimizers) on a judge that makes a single model call per application. Each run made about 40 full passes over the training applications, similar to the amount that models typically made across full 12 hour runs. Neither optimization run found a better prompt: the judge scores 61.5% on the held-out set, two points above always answering “pass”, while 17 of 19 agents beat it (median 74.0%, best 81.3%).

Fixed references and performance near the state of the art Like OpenAI’s self-improvement thresholds, our references are fixed: OpenAI’s High threshold is defined “relative to those researchers’ 2024 baseline”, and its Critical lagging indicator as a generational model improvement “in 1/5th the wall-clock time of equivalent progress in 2024” [1]. Fixed references keep results comparable over time; a moving reference would confound agents’ own capability with AI-assisted human progress. Our 0.6 level is still a high bar, requiring margins over the best known results comparable to AlexNet, BERT and the move from LSTMs to Transformers.

Exceeding the best known result does not require brand-new ideas: DeepSeek concluded for DeepSeek-V4.1-Flash that “the marginal return of engineering the data and environment pipeline substantially exceeds that of algorithmic novelty in post-training” [14]. The agents’ gains likewise come from combining and tuning existing techniques. RSI-Index will track how far newer models move past the best known results.

Awareness of top solutions We prompted the tested models; none could reproduce a top solution to our tasks. Only one model recalled the LM Training reference score (0.85 BPB), and its own submission does not beat it. Remembering the reference also gives no direct recipe for our task as it is a different Transformer architecture. Top models also know general techniques such as the Muon optimizer, which the January 2026 NanoGPT public record uses. This is general knowledge rather than contamination: agents still have to apply, tune and combine techniques within the budget, and only Opus 5.5 beats the record.

On Compression, top models can describe the reference compressor (NNCP), but none of the compressors they build matches this performance (best 1.31 vs 1.24 BPB reference).

References

[1] OpenAI, “Preparedness Framework (Version 2),” 2025.
[2] Anthropic, “Responsible Scaling Policy, Version 3.4,” 2026.
[3] M. Favaro and J. Clark, “When AI builds itself,” Anthropic, 2026.
[4] OpenAI, “GPT-6 Astra System Card,” 2026.
[5] K. Jordan et al., “Muon: An optimizer for hidden layers in neural networks,” 2024.
[6] OpenAI, “GPT-4 Technical Report,” 2023.
[7] G. Delétang et al., “Language Modeling Is Compression,” ICLR 2024.
[8] R. Sennrich, B. Haddow and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016.
[9] Lai et al., “MiniMax Sparse Attention,” arXiv:2606.13392, 2026.
[10] R. Child, S. Gray, A. Radford and I. Sutskever, “Generating Long Sequences with Sparse Transformers,” arXiv:1904.10509, 2019.
[11] A. Ho et al., “Algorithmic progress in language models,” arXiv:2403.05812, 2024.
[12] OpenAI, “Research acceleration: The view inside OpenAI,” 2026.
[13] Anthropic, “Claude Opus 5 System Card,” 2026.
[14] DeepSeek-AI, “DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression,” 2026.

Thank you again for the constructive feedback, we hope our response addresses your concerns before the discussion phase comes to a close. We're happy to clarify anything further.

Reviewer_dVUBOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

They introduce a new benchmark for automated AI research, called RSI-Index. RSI-Index has four tasks: compression (building a lossless compressor for 100 MB of Wikipedia), training a small language model on 800 MB of Wikipedia, building a harness for an llm-as-a-judge and post-training a 35B parameter model for finance. On RSI-Index, a score of 0.5 corresponds to the best human solution on the task, a score of 1.0 corresponds to the theoretical maximum score on this task.

Critical strengths
  1. good analysis with lots of details, including analysis of how the agents behaved. E.g. "sequential post-training", where they investigated what happens if the agents first need to post-train for one task, then for another. Also great to see the many different models being tested.
  2. important topic. Testing how well AI agents can do autonomous research is important.
Critical weaknesses
  1. the novel tasks are toyish. Arguably the most relevant task for frontier AI research here is post-training, which was covered in PostTrainBench (Rank et al., 2026). The compression and lm training tasks are much simpler, arguably toyish and have limited relevance for frontier research. The harness engineering task might be interesting, although there are only limited details provided on what kind of web-apps they use etc.
  2. the benchmark is closed and it is not possible to run it by yourself. The harness engineering task needs data which the authors don't give freely and the post-training task needs a private benchmark which is not available freely. For open research, open benchmarks are crucial. I would consider increasing the score if the benchmark is made available completely openly.
Questions for authors
  1. Why call it RSI? It is imo rather automated AI research, but not really RSI, which would mean that an agent improves a model of it's own capabilities. The models here are much weaker than frontier models.
  2. Could the tasks be made more realistic and closer to real world frontier research?
  3. Would it be possible to make the benchmark more open? (see weakness 2)
8 comments

We thank the reviewer for recognizing our thorough analysis and hope to address any concerns below.

Why call this RSI? Our paper introduces RSI as an AI system that builds a more intelligent successor, which cannot be measured directly at the frontier because it requires frontier-scale compute. Frontier labs instead tie RSI to automated AI R&D. OpenAI defines models “capable of recursively self improving” as “fully automated AI R&D” [1], and Anthropic states that its automated AI R&D threshold “is intended to capture the onset of dramatic recursive self-improvement” [2]. RSI-Index measures this capability, not closed-loop model training, and tracks its improvement across model releases from multiple providers. We will make this framing explicit in Section 1.

“Toy-ish” tasks We agree our tasks may not be representative of frontier-scale training and will say so in the Limitations. However, they test the same research judgement that frontier-scale work requires, and many frontier methods were first found on small tasks.

  • Frontier labs themselves use small-scale tasks like ours to measure progress toward recursive self-improvement: Anthropic re-runs a small-model training task on every model release [3], and OpenAI evaluates GPT-6 Astra on NanoGPT, a single-H100 LM-training task “intended to measure LLM pretraining ability” [4].
  • The Muon optimizer was first established as part of NanoGPT speedrunning and used to train frontier-scale models such as Kimi K2 (1 trillion parameters) and GLM-4.5. Our LM Training task is similarly inspired by the NanoGPT speedrun [5].
  • Many frontier scaling decisions are derived from smaller experiments. Parts of GPT-4’s performance were predicted based on “models trained with no more than 1/1,000th the compute of GPT-4.” [6]
  • Compression, as we note in our paper, is tightly coupled to language modeling [7]. Byte-pair encoding, a common tokenizer for LLMs, was initially a compression algorithm [8]. Sparse attention, whose variants are used in frontier models like MiniMax-M3 [9], was introduced with enwik8 results [10], the same dataset we use for our Compression task.
  • Neither the LM Training nor the Compression task is easy for the tested models. Only one model (Opus 5.5) has passed the LM Training reference.

We ran additional experiments to test if gains transfer beyond our setup. We re-trained the LM Training submissions of Opus 5.5, Fable 5.1, GPT-6 Astra and Grok 4.6 on four new 1 GB datasets from other domains, and separately with ten times the training time on 11 GB of web text. We used RTX PRO 6000 GPUs rather than the H100s used for official scoring. Opus 5.5 is first on four of five datasets (GPT-6 Astra is slightly ahead on Python code) and Grok 4.6 last on all; only the closely matched Fable 5.1 and GPT-6 Astra swap places. With ten times the training time, submissions improve by about 0.14–0.15 BPB. Gains therefore transfer across data domains and to longer training, not just within our task setup, which we take as empirical evidence that the task measures general training efficiency.

Held-out BPB after retraining (lower is better; not comparable with official H100 scores):

SubmissionWikipedia (enwik9, original task data)Educational web text (FineWeb-Edu)Python code (CodeParrot-clean)Japanese web text (FineWeb-2)Math web text (OpenWebMath)
Opus 5.50.8631.0800.4831.0191.051
GPT-6 Astra0.8821.0990.4791.0421.073
Fable 5.10.8891.0890.5061.0441.071
Grok 4.60.9371.1350.5421.0851.117

Overlap with PostTrainBench As our paper acknowledges, PostTrainBench first measured agents’ post-training ability. Our task differs in scale and realism: we use a 35B model with 480 GPU-hours, compared to 1.7B–4B models and ten GPU-hours. We measure against an agentic benchmark rather than single-turn benchmarks like GSM8K, and use a private held-out test set, reducing risks of contamination and cheating.

Harness Engineering clarification The model under test writes a harness for a small, fixed judge model (Gemini 3.5 Flash Lite). The judge grades full-stack web applications against a set of functional user workflows. The apps include personal tools (e.g., a habit tracker), small-business apps (e.g., finding and reserving parking) and internal tools (e.g., approving procurement requests), many requiring authentication, databases and payments. We will clarify this further in the paper.

More realistic tasks Future versions of the benchmark will add tasks that start from stronger baselines and existing research codebases, use larger budgets where possible, and cover new areas such as data curation and inference optimization.

Openness We agree that open research and open benchmarks are important. Our Compression and LM Training tasks already use public data. The Harness applications and the Post-training test set use private data, which we don’t currently plan to release, to avoid leakage and keep the benchmark trustworthy over time. We commit to expanding the task descriptions in the paper.

References

[1] OpenAI, “Preparedness Framework (Version 2),” 2025.
[2] Anthropic, “Responsible Scaling Policy, Version 3.4,” 2026.
[3] M. Favaro and J. Clark, “When AI builds itself,” Anthropic, 2026.
[4] OpenAI, “GPT-6 Astra System Card,” 2026.
[5] K. Jordan et al., “Muon: An optimizer for hidden layers in neural networks,” 2024.
[6] OpenAI, “GPT-4 Technical Report,” 2023.
[7] G. Delétang et al., “Language Modeling Is Compression,” ICLR 2024.
[8] R. Sennrich, B. Haddow and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016.
[9] Lai et al., “MiniMax Sparse Attention,” arXiv:2606.13392, 2026.
[10] R. Child, S. Gray, A. Radford and I. Sutskever, “Generating Long Sequences with Sparse Transformers,” arXiv:1904.10509, 2019.

Thank you for your response!

I still have serious concerns about the openness of the benchmark.

Half of the benchmark is effectively closed (the post-training and harness engineering tasks). The post-training task is closed because it relies on a closed, proprietary benchmark. The harness engineering task is fully closed. Those can therefore not be properly judged on their quality.

Further, the two open tasks are not representative of the whole benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.