- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces RSIBench-Code, 105 coding tasks taken from LLM development work such as post-training, training infrastructure and serving. The tasks are built by an LLM (GPT 5.6 Sol) on top of the public verl and SGLang codebases, each with a hidden grader and a reference solution. Current agents score around 36–44%, and the authors argue that this kind of code is a bottleneck for recursive self-improvement (RSI).
Critical strengths
-
A relevant and under-served target. The tasks live inside real LLM training and serving code and go beyond bug fixing, for example writing tests and running small experiments.
-
Open and transparent. All tasks and graders are released, together with starter and reference baselines, which makes the benchmark easy to inspect and reuse.
Critical weaknesses
-
The RSI claim is not supported, and the novelty is unclear. The paper shows that the tasks are hard, but not that solving them leads to better models or agents, so the link to RSI is a hypothesis rather than a result. The construction method (mining tasks from coding sessions and having LLMs write tasks and graders) is close to existing benchmarks such as SWE-smith and ProdCodeBench, and the most similar recent benchmarks are barely discussed. The related work also does not mention the original key RSI algorithms: the Gödel Machine (Schmidhuber, 2003) and recursive meta-evolution (Schmidhuber, 1987).
-
The scores are not yet reliable. The graders are written by an LLM and no human check is reported. Some reported scores also do not match what the released graders can produce, which needs an explanation. In addition, the model comparison is uneven (different numbers of attempts and different harnesses), and the gaps between the top models are within the noise, so the ranking is not established.
References
[1] J. Schmidhuber. "Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements." arXiv:cs/0309048, 2003.
[2] J. Schmidhuber. "Evolutionary Principles in Self-Referential Learning, or On Learning How to Learn: The Meta-Meta-... Hook." Diploma thesis, Technische Universität München, 1987.
[3] J. Yang, K. Lieret, C. E. Jimenez, et al. "SWE-smith: Scaling Data for Software Engineering Agents." NeurIPS 2025.
[4] "ProdCodeBench: A Production-Derived Benchmark for Evaluating AI Coding Agents." arXiv:2604.01527, 2026. Later version titled "REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage".
[5] L. Ding, et al. "Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?" arXiv:2609.10226, 2026.
[6] W. Chen, et al. "Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?" arXiv:2604.10547, 2026.
[7] A. Nangia, et al. "ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?" arXiv:2602.19594, 2026.
Questions for authors
- (Clarification) Is there evidence that solving these tasks leads to better models or agents downstream? For example, does a higher score predict that an agent's code changes improve a real training or serving run? If not, could you present the RSI link as a hypothesis?
- (Clarification) Were the graders checked by people, and how often do they reject correct solutions or accept wrong ones?
- (Clarification) Can you explain the Figure 8 scores that appear inconsistent with the released graders, such as the 14% starter score for prompt-rollout-allocation-study-easy? The grader version and score breakdown for this row would help.
- (Suggestion) Repeated runs for all models, plus a comparison in the same harness, would make the ranking more convincing.
- (Suggestion) Please discuss how RSIBench-Code differs from Φ-Bench, Agent² RL-Bench and ISO-Bench, for example in its coverage of test writing and controlled experiments (Table 1). The related work should also discuss the original key RSI algorithms, the Gödel Machine (2003) and recursive meta-evolution (1987).
8 comments
(Weakness 1 and Question 5) Novelty relative to SWE-smith and ProdCodeBench, discussion of Φ-Bench, Agent² RL-Bench and ISO-Bench, and citation of the original RSI algorithms.
Our task source and evaluation differ from existing work. SWE-smith synthesizes bugs in public repositories. REAP, the later version of ProdCodeBench, keeps only single-turn conversations and excludes code changes that come from more than one conversation. Unlike regular debugging, one LLM development task may be scattered across several sessions, so a complete task is hard to collect, extract and filter, and we reconstruct each complete task from this scattered context using curated prompts and our generation pipeline. REAP also keeps only tasks that existing fail-to-pass tests can verify. LLM development code often has no tests for the target behavior, so our framework writes and validates hidden evaluators, including evaluators for test writing and controlled experiments that existing tests cannot check. This is why we built a complete and curated task generation framework rather than filtering existing changes and tests.
Φ-Bench, Agent² RL-Bench and ISO-Bench are also close to our setting. Compared with them, RSIBench-Code tasks come from real enterprise LLM development work rather than from public pull requests, masked code or existing benchmarks. We sampled tasks and found no related tasks or code available online, so there is no task leakage into the training data of the evaluated models. Without public tasks or solutions to start from, task generation becomes much harder at every stage, from collecting complete tasks to filtering useful ones and generating reliable evaluation tasks. To address this, we provide the full task generation pipeline, which also lets companies build private benchmarks from their own development tasks and codebases.
Thank you for pointing out the missing works. We have compared RSIBench-Code with all the works you list and added the missing ones. In the first paragraph of Appendix G.2 (Generating Executable Tasks for Self-Improvement), we have added the following sentence after CursorBench, which also derives tasks from real coding sessions.
"ProdCodeBench, later extended as REAP, curates tasks from single-turn conversations with a production coding assistant and keeps only tasks that existing fail-to-pass tests can verify."
At the end of the same paragraph, we have added the following contrast.
"[RSIBench-Code] reconstructs complete development tasks that may span several sessions, and writes and validates hidden evaluators for behavior that existing tests do not cover."
At the end of the first paragraph of Appendix G.3 (Evaluating Agents on LLM-Development Tasks), we have added the following sentences.
"Closest to our setting, Φ-Bench evaluates kernel completion, feature implementation, and end-to-end optimization of LLM infrastructure, with tasks built from public pull requests, issues, and masked repository code. Agent² RL-Bench evaluates whether agents can build post-training pipelines that improve a base model on six public benchmarks, and ISO-Bench evaluates whether agents can reproduce inference optimizations from merged pull requests in vLLM and SGLang. Unlike these benchmarks, RSIBench-Code draws its tasks from enterprise LLM-development work rather than from public pull requests, masked code, or existing benchmarks, and a check of sampled tasks found no related tasks or code available online (Section 4)."
The full task generation pipeline, which lets companies build private benchmarks from their own development tasks and codebases, is discussed in the new paragraph at the end of Appendix G.3.
At the start of Appendix G.1 (Recursive Self-Improvement through LLM-Development Code), we have added the following sentences.
"The idea of RSI goes back to self-referential meta-learning (Schmidhuber, 1987), in which a system learns how to improve its own learning procedure, and to the Gödel Machine (Schmidhuber, 2003), a self-referential problem solver that rewrites its own code once it can prove that the rewrite is beneficial. RSIBench-Code evaluates a practical prerequisite of such systems, namely whether agents can modify the code that trains and serves models."
(Weakness 2 and Question 2) Whether the evaluators were checked by people, and how often they reject correct solutions or accept wrong ones.
We manually audited tasks end to end, covering the generation prompts, task cases, evaluation environment setup and final scores. To support automatic task generation, we also used the findings of this audit to iterate on the evaluator generation prompt, so that later evaluators need as little human correction as possible. The two controls in Section 3.2 also measure how evaluators treat known solutions. The reference solution is a known correct submission, so its score shows how often an evaluator rejects correct behavior. The unchanged starter code is a known incomplete submission, so its score shows how much credit an evaluator gives without any work on the task. Reference solutions receive 99.95% on average, so the evaluators almost never reject correct solutions, and unchanged starter code receives only 26.2%, so the evaluators do not accept it as a solution. We have added the following subsection, "Evaluator Audit", to Appendix D (Evaluation Implementation).
"A score is meaningful only if the evaluator credits the required behavior and nothing else. Human reviewers therefore audited tasks along their full evaluation path, and we used their findings to iterate on the evaluator generation prompt, so that automatic task generation needs as little human labor as possible. The audit covers four parts of each task. First, for generation prompts, we checked that each task specification and instruction states every behavior the evaluator checks, so that no check depends on unstated requirements. Second, for task cases, we checked that each hidden test case matches the instruction and that its expected output is correct. Third, for the evaluation environment, we checked that the solver and verifier containers build offline with pinned dependencies, and that the evaluator runs within its time and memory limits. Fourth, for final scores, we inspected low-scoring and failed attempts to confirm that lost credit comes from the submission rather than from the evaluator, and that evaluator errors are reported as errors rather than as zero scores. We also run two controls on every task, the reference solution and the unchanged starting code, and report their scores in Section 3. Reference solutions receive 99.95% on average, so the evaluators rarely reject correct behavior, and the unchanged starting code receives 26.2%, so the evaluators give limited credit without work on the task."
(Weakness 2 and Question 3) Figure 8 scores that do not match the released graders.
Thank you for carefully checking this row. The Figure 8 scores were produced with an earlier version of the evaluators. During our audit, we updated the evaluators but did not regenerate Figure 8, so the figure does not match the released graders. For prompt-rollout-allocation-study-easy, the earlier version has 14 checks and the released version has 13. This mismatch also shows that we did audit the evaluators and update them where needed. We have regenerated Figure 8 and all other figures with the released graders from the new runs.
(Weakness 2 and Question 4) Uneven model comparison across attempts and harnesses.
We have completed two extra runs for Hy4, GLM-5.2, DeepSeek V4.1 Flash and GPT 5.6, which had one run each, so every model now has three runs per task. We have added the mean score of each run next to the bootstrap intervals in Table 2. The three run means of each model differ by at most 6.2 points, less than the width of the bootstrap intervals. Adjacent models still have overlapping intervals, so Section 3.2 now states that small gaps, such as the 0.7 points between Hy4 and DeepSeek V4.1 Flash, should not be read as a strict ranking.
For harnesses, we aim to evaluate models the way people actually use them, not the model alone. A model and its official harness are designed together and complement each other, so we treat the harness as part of the evaluated system. GPT 5.6 therefore runs with its official harness, Codex CLI, and the models without an official harness supported in Harbor run with Terminus-2, Harbor's reference agent. This follows common practice. Terminal-Bench 2.0 (Merrill et al., 2026) runs in Harbor and reports each model with its own harness. SkillsBench (2026), also built on Harbor, evaluates Claude Code, Codex CLI and Gemini CLI, each with its own model. PostTrainBench (Rank et al., 2026) evaluates models in their official harnesses and finds that the official harness usually performs better with the same model.
Thank you for your careful and detailed review. You read our paper and released tasks very closely, down to checking individual scores against the released graders, and your advice is constructive throughout. We appreciate your recognition of the relevance of our target and the openness of our released tasks and graders. Your comments have helped us improve our work substantially.
Several reviewers suggested evaluating every model with three runs. We have completed two extra runs for each model that had one run, so every model now has three runs per task. We have uploaded the revised PDF with the updated results and all changes. We address each point below.
With our benchmark, a higher score means that fewer of the failures that would break or degrade a real training or serving run remain in the agent's change.
Each task is a change that a real run needed. Our tasks are derived from real tasks in enterprise LLM development, where each change was required for training or serving runs to work correctly or to improve. These tasks affect model training in practice, which gives a prior justification for their relevance.
Each evaluator matches the acceptance criteria of its task. The evaluator checks the behaviors that the change should have, and we manually audited the evaluators (Weakness 2 and Question 2 below). Passing an evaluator therefore indicates that the task is completed and that normal model training can resume.
Each check is a failure mode of a run. Scores are partial credit over checks, and each failed check corresponds to a concrete way a run goes wrong. For example, if a resume restores the model but not the optimizer or RNG state (checkpoint-resume-state-repair), training continues with reset optimizer moments and a different sampling stream, so the resumed run silently diverges from the original one. If in-flight rollout requests are served with stale weights after a refit (atomic-weight-refit-feature), samples from an old policy are trained on as if they came from the current one, which biases the policy update. If PPO advantages change sign when the same responses are regrouped or padded (sequence-population-normalization-repair), the same response is reinforced in one batch layout and penalized in another, so the update depends on batching rather than on response quality. If the top-k distillation gradient is wrong under tensor parallelism (topk-gradient-contract-tests), the student learns from a distorted teacher signal, and the error appears only in multi-GPU runs, so single-device tests do not catch it. A higher score therefore means fewer of these failures remain.
The checked code is the code real runs execute. Reviewer QfF2 also notes that "some tasks presented in Appendix B connects to the current RSI".
Our design aims at efficient evaluation and avoids excessive evaluation costs. We therefore check the most essential observable outcomes rather than every aspect of the development process. As the examples above show, this does not mean that our tasks are disconnected from RSI. Compared with existing work, our approach sources tasks from real-world development while avoiding the complexity and overhead of end-to-end evaluation.