acceptodds
Under review as a conference paper at ICLR 2027

RSIBench-Code: Can LLM Agents Handle Enterprise-Level LLM Development Code for Self-Improvement?

Abstract

Recursive self-improvement (RSI) aims to enable AI systems to contribute to the processes that produce stronger future models. A practical route is model self-training, but the code behind training, evaluation, serving, and infrastructure is still largely designed, maintained, and debugged by human engineers. This raises a central question: **can LLM agents handle the enterprise-level LLM-development code needed for self-improving AI**? We introduce **RSIBench-Code**, a benchmark for evaluating LLM agents on coding tasks reconstructed from practical LLM development problems in an enterprise. The benchmark contains 105 tasks across post-training, training infrastructure, serving, evaluation, and agent systems. These tasks cover implementation, repair, test writing, controlled experiments, and execution-path alignment. In each task, agents must locate and modify relevant code within repositories containing approximately 1.8 million source lines, with changing up to thousands of lines. RSIBench-Code supports long-horizon coding attempts with up to 150 agent turns and a 7,200-second solving budget. Experiments show that existing agents make meaningful but incomplete progress, suggesting that enterprise-level LLM-development code remains a key bottleneck for recursive self-improvement. Furthermore, RSIBench-Code provides a continuing source of executable evaluation tasks and potential training data for RSI in future work.

Primary area:
datasets and benchmarks
Keywords:
Recursive Self-Improvement, Large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_crvsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RSIBench-Code, 105 coding tasks taken from LLM development work such as post-training, training infrastructure and serving. The tasks are built by an LLM (GPT 5.6 Sol) on top of the public verl and SGLang codebases, each with a hidden grader and a reference solution. Current agents score around 36–44%, and the authors argue that this kind of code is a bottleneck for recursive self-improvement (RSI).

Critical strengths
  1. A relevant and under-served target. The tasks live inside real LLM training and serving code and go beyond bug fixing, for example writing tests and running small experiments.

  2. Open and transparent. All tasks and graders are released, together with starter and reference baselines, which makes the benchmark easy to inspect and reuse.

Critical weaknesses
  1. The RSI claim is not supported, and the novelty is unclear. The paper shows that the tasks are hard, but not that solving them leads to better models or agents, so the link to RSI is a hypothesis rather than a result. The construction method (mining tasks from coding sessions and having LLMs write tasks and graders) is close to existing benchmarks such as SWE-smith and ProdCodeBench, and the most similar recent benchmarks are barely discussed. The related work also does not mention the original key RSI algorithms: the Gödel Machine (Schmidhuber, 2003) and recursive meta-evolution (Schmidhuber, 1987).

  2. The scores are not yet reliable. The graders are written by an LLM and no human check is reported. Some reported scores also do not match what the released graders can produce, which needs an explanation. In addition, the model comparison is uneven (different numbers of attempts and different harnesses), and the gaps between the top models are within the noise, so the ranking is not established.

References

[1] J. Schmidhuber. "Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements." arXiv:cs/0309048, 2003.

[2] J. Schmidhuber. "Evolutionary Principles in Self-Referential Learning, or On Learning How to Learn: The Meta-Meta-... Hook." Diploma thesis, Technische Universität München, 1987.

[3] J. Yang, K. Lieret, C. E. Jimenez, et al. "SWE-smith: Scaling Data for Software Engineering Agents." NeurIPS 2025.

[4] "ProdCodeBench: A Production-Derived Benchmark for Evaluating AI Coding Agents." arXiv:2604.01527, 2026. Later version titled "REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage".

[5] L. Ding, et al. "Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?" arXiv:2609.10226, 2026.

[6] W. Chen, et al. "Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?" arXiv:2604.10547, 2026.

[7] A. Nangia, et al. "ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?" arXiv:2602.19594, 2026.

Questions for authors
  1. (Clarification) Is there evidence that solving these tasks leads to better models or agents downstream? For example, does a higher score predict that an agent's code changes improve a real training or serving run? If not, could you present the RSI link as a hypothesis?
  2. (Clarification) Were the graders checked by people, and how often do they reject correct solutions or accept wrong ones?
  3. (Clarification) Can you explain the Figure 8 scores that appear inconsistent with the released graders, such as the 14% starter score for prompt-rollout-allocation-study-easy? The grader version and score breakdown for this row would help.
  4. (Suggestion) Repeated runs for all models, plus a comparison in the same harness, would make the ranking more convincing.
  5. (Suggestion) Please discuss how RSIBench-Code differs from Φ-Bench, Agent² RL-Bench and ISO-Bench, for example in its coverage of test writing and controlled experiments (Table 1). The related work should also discuss the original key RSI algorithms, the Gödel Machine (2003) and recursive meta-evolution (1987).
8 comments

Thank you for your careful and detailed review. You read our paper and released tasks very closely, down to checking individual scores against the released graders, and your advice is constructive throughout. We appreciate your recognition of the relevance of our target and the openness of our released tasks and graders. Your comments have helped us improve our work substantially.

Several reviewers suggested evaluating every model with three runs. We have completed two extra runs for each model that had one run, so every model now has three runs per task. We have uploaded the revised PDF with the updated results and all changes. We address each point below.

(Weakness 1 and Question 1) Evidence that solving the tasks leads to better models, or presenting the RSI link as a hypothesis.

With our benchmark, a higher score means that fewer of the failures that would break or degrade a real training or serving run remain in the agent's change.

  • Each task is a change that a real run needed. Our tasks are derived from real tasks in enterprise LLM development, where each change was required for training or serving runs to work correctly or to improve. These tasks affect model training in practice, which gives a prior justification for their relevance.

  • Each evaluator matches the acceptance criteria of its task. The evaluator checks the behaviors that the change should have, and we manually audited the evaluators (Weakness 2 and Question 2 below). Passing an evaluator therefore indicates that the task is completed and that normal model training can resume.

  • Each check is a failure mode of a run. Scores are partial credit over checks, and each failed check corresponds to a concrete way a run goes wrong. For example, if a resume restores the model but not the optimizer or RNG state (checkpoint-resume-state-repair), training continues with reset optimizer moments and a different sampling stream, so the resumed run silently diverges from the original one. If in-flight rollout requests are served with stale weights after a refit (atomic-weight-refit-feature), samples from an old policy are trained on as if they came from the current one, which biases the policy update. If PPO advantages change sign when the same responses are regrouped or padded (sequence-population-normalization-repair), the same response is reinforced in one batch layout and penalized in another, so the update depends on batching rather than on response quality. If the top-k distillation gradient is wrong under tensor parallelism (topk-gradient-contract-tests), the student learns from a distorted teacher signal, and the error appears only in multi-GPU runs, so single-device tests do not catch it. A higher score therefore means fewer of these failures remain.

  • The checked code is the code real runs execute. Reviewer QfF2 also notes that "some tasks presented in Appendix B connects to the current RSI".

  • Our design aims at efficient evaluation and avoids excessive evaluation costs. We therefore check the most essential observable outcomes rather than every aspect of the development process. As the examples above show, this does not mean that our tasks are disconnected from RSI. Compared with existing work, our approach sources tasks from real-world development while avoiding the complexity and overhead of end-to-end evaluation.

(Weakness 1 and Question 5) Novelty relative to SWE-smith and ProdCodeBench, discussion of Φ-Bench, Agent² RL-Bench and ISO-Bench, and citation of the original RSI algorithms.

Our task source and evaluation differ from existing work. SWE-smith synthesizes bugs in public repositories. REAP, the later version of ProdCodeBench, keeps only single-turn conversations and excludes code changes that come from more than one conversation. Unlike regular debugging, one LLM development task may be scattered across several sessions, so a complete task is hard to collect, extract and filter, and we reconstruct each complete task from this scattered context using curated prompts and our generation pipeline. REAP also keeps only tasks that existing fail-to-pass tests can verify. LLM development code often has no tests for the target behavior, so our framework writes and validates hidden evaluators, including evaluators for test writing and controlled experiments that existing tests cannot check. This is why we built a complete and curated task generation framework rather than filtering existing changes and tests.

Φ-Bench, Agent² RL-Bench and ISO-Bench are also close to our setting. Compared with them, RSIBench-Code tasks come from real enterprise LLM development work rather than from public pull requests, masked code or existing benchmarks. We sampled tasks and found no related tasks or code available online, so there is no task leakage into the training data of the evaluated models. Without public tasks or solutions to start from, task generation becomes much harder at every stage, from collecting complete tasks to filtering useful ones and generating reliable evaluation tasks. To address this, we provide the full task generation pipeline, which also lets companies build private benchmarks from their own development tasks and codebases.

Thank you for pointing out the missing works. We have compared RSIBench-Code with all the works you list and added the missing ones. In the first paragraph of Appendix G.2 (Generating Executable Tasks for Self-Improvement), we have added the following sentence after CursorBench, which also derives tasks from real coding sessions.

"ProdCodeBench, later extended as REAP, curates tasks from single-turn conversations with a production coding assistant and keeps only tasks that existing fail-to-pass tests can verify."

At the end of the same paragraph, we have added the following contrast.

"[RSIBench-Code] reconstructs complete development tasks that may span several sessions, and writes and validates hidden evaluators for behavior that existing tests do not cover."

At the end of the first paragraph of Appendix G.3 (Evaluating Agents on LLM-Development Tasks), we have added the following sentences.

"Closest to our setting, Φ-Bench evaluates kernel completion, feature implementation, and end-to-end optimization of LLM infrastructure, with tasks built from public pull requests, issues, and masked repository code. Agent² RL-Bench evaluates whether agents can build post-training pipelines that improve a base model on six public benchmarks, and ISO-Bench evaluates whether agents can reproduce inference optimizations from merged pull requests in vLLM and SGLang. Unlike these benchmarks, RSIBench-Code draws its tasks from enterprise LLM-development work rather than from public pull requests, masked code, or existing benchmarks, and a check of sampled tasks found no related tasks or code available online (Section 4)."

The full task generation pipeline, which lets companies build private benchmarks from their own development tasks and codebases, is discussed in the new paragraph at the end of Appendix G.3.

At the start of Appendix G.1 (Recursive Self-Improvement through LLM-Development Code), we have added the following sentences.

"The idea of RSI goes back to self-referential meta-learning (Schmidhuber, 1987), in which a system learns how to improve its own learning procedure, and to the Gödel Machine (Schmidhuber, 2003), a self-referential problem solver that rewrites its own code once it can prove that the rewrite is beneficial. RSIBench-Code evaluates a practical prerequisite of such systems, namely whether agents can modify the code that trains and serves models."

(Weakness 2 and Question 2) Whether the evaluators were checked by people, and how often they reject correct solutions or accept wrong ones.

We manually audited tasks end to end, covering the generation prompts, task cases, evaluation environment setup and final scores. To support automatic task generation, we also used the findings of this audit to iterate on the evaluator generation prompt, so that later evaluators need as little human correction as possible. The two controls in Section 3.2 also measure how evaluators treat known solutions. The reference solution is a known correct submission, so its score shows how often an evaluator rejects correct behavior. The unchanged starter code is a known incomplete submission, so its score shows how much credit an evaluator gives without any work on the task. Reference solutions receive 99.95% on average, so the evaluators almost never reject correct solutions, and unchanged starter code receives only 26.2%, so the evaluators do not accept it as a solution. We have added the following subsection, "Evaluator Audit", to Appendix D (Evaluation Implementation).

"A score is meaningful only if the evaluator credits the required behavior and nothing else. Human reviewers therefore audited tasks along their full evaluation path, and we used their findings to iterate on the evaluator generation prompt, so that automatic task generation needs as little human labor as possible. The audit covers four parts of each task. First, for generation prompts, we checked that each task specification and instruction states every behavior the evaluator checks, so that no check depends on unstated requirements. Second, for task cases, we checked that each hidden test case matches the instruction and that its expected output is correct. Third, for the evaluation environment, we checked that the solver and verifier containers build offline with pinned dependencies, and that the evaluator runs within its time and memory limits. Fourth, for final scores, we inspected low-scoring and failed attempts to confirm that lost credit comes from the submission rather than from the evaluator, and that evaluator errors are reported as errors rather than as zero scores. We also run two controls on every task, the reference solution and the unchanged starting code, and report their scores in Section 3. Reference solutions receive 99.95% on average, so the evaluators rarely reject correct behavior, and the unchanged starting code receives 26.2%, so the evaluators give limited credit without work on the task."

(Weakness 2 and Question 3) Figure 8 scores that do not match the released graders.

Thank you for carefully checking this row. The Figure 8 scores were produced with an earlier version of the evaluators. During our audit, we updated the evaluators but did not regenerate Figure 8, so the figure does not match the released graders. For prompt-rollout-allocation-study-easy, the earlier version has 14 checks and the released version has 13. This mismatch also shows that we did audit the evaluators and update them where needed. We have regenerated Figure 8 and all other figures with the released graders from the new runs.

(Weakness 2 and Question 4) Uneven model comparison across attempts and harnesses.

We have completed two extra runs for Hy4, GLM-5.2, DeepSeek V4.1 Flash and GPT 5.6, which had one run each, so every model now has three runs per task. We have added the mean score of each run next to the bootstrap intervals in Table 2. The three run means of each model differ by at most 6.2 points, less than the width of the bootstrap intervals. Adjacent models still have overlapping intervals, so Section 3.2 now states that small gaps, such as the 0.7 points between Hy4 and DeepSeek V4.1 Flash, should not be read as a strict ranking.

For harnesses, we aim to evaluate models the way people actually use them, not the model alone. A model and its official harness are designed together and complement each other, so we treat the harness as part of the evaluated system. GPT 5.6 therefore runs with its official harness, Codex CLI, and the models without an official harness supported in Harbor run with Terminus-2, Harbor's reference agent. This follows common practice. Terminal-Bench 2.0 (Merrill et al., 2026) runs in Harbor and reports each model with its own harness. SkillsBench (2026), also built on Harbor, evaluates Claude Code, Codex CLI and Gemini CLI, each with its own model. PostTrainBench (Rank et al., 2026) evaluates models in their official harnesses and finds that the official harness usually performs better with the same model.

Reviewer_QfF2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RSIBench-Code, a benchmark to evaluate LLM agents on coding tasks reconstructed from practical LLM development in enterprise. Deepseek v4.1 flash on the benchmark results in 39.3 mean score, showing that such benchmark is reasonably hard to task the agentic coding capability of the model.

Critical strengths
  1. This benchmark contains broad task types, including implementation, repair, tests, experiments and execution-path alignment.
  2. I like the part where the benchmark covers parts of the LLM-development pipeline, including post-training, training-infrastructure, etc. I do think some tasks presented in Appendix B connects to the current RSI, for instance, actor-critic-reward-debug-hard, etc.
  3. Good sources of problem selection, for instance, VERL as a popular RL framework.
Critical weaknesses
  1. The results of GPT 5.6 do not reveal which gpt 5.6 is, and what's the setting of the reasoning efforts for all the model. I can not find the details of reasoning effort and temperature (I assume 1.0). Even for gpt 5.6 luna, the performance between reasoning effort medium and max can be pretty different, without talking about the model capability different between gpt 5.6 luna and gpt 5.6 sol. The results presented in the paper makes me tend to believe it's gpt 5.6 Terra or Luna instead of Sol.
  2. For task difficulty, it's great to have three trials, but only limited to Deepseek V4 Pro can introduce model bias. I would suggest including more models for the task difficulty labeling.
  3. All the environments are CPU only is understandable, but this constraint makes the evaluation not close to real RSI evaluation. The gap between the benchmark setting and the RSI is pretty huge. The cost is understandable, but even enable some dummy GPU environments would significantly close the gap between current benchmark settings and real-world RSI setting.
Questions for authors
  1. I am not sure why we need limit the max token length to 32k, maybe because the harness adopted can not compact the conversation?

  2. If GPT 5.6 refers to Terra or Luna, can the authors add the results of GPT 5.6 SOL with reasoning effort high? This would be good enough to tell how long this benchmark would be able to consider as a frontier model capability estimation.

3 comments

Thank you very much for your thoughtful and encouraging review. We truly appreciate that you recognize the breadth of our task types, the coverage of the LLM-development pipeline, the connection of tasks such as actor-critic-reward-debug-hard to current RSI, and our choice of problem sources such as verl. Your detailed questions on model settings and difficulty labeling help us make our work more complete and more reproducible.

Several reviewers suggested evaluating every model with three runs. We have completed two extra runs for each model that had one run, so every model now has three runs per task. We have uploaded the revised PDF with the updated results and all changes. We address each point below.

(Weakness 1 and Question 2) GPT 5.6 variant, reasoning effort and temperature, and results for GPT 5.6 Sol with high reasoning effort.

GPT 5.6 in the paper refers to GPT 5.6 Sol with high reasoning effort, so the reported results already use the setting you suggest. We have added the settings of every model to the "Execution Settings and Attempt Coverage" subsection of Appendix D with the following sentences.

"GPT 5.6 refers to GPT 5.6 Sol with high reasoning effort. Hy4 uses high reasoning effort, and GLM-5.2, DeepSeek V4 Pro, and DeepSeek V4.1 Flash use max reasoning effort. Neither Terminus-2 nor Codex CLI sets a sampling temperature, so every model uses the default temperature."

(Weakness 2) Difficulty labels from DeepSeek V4 Pro alone may be biased.

DeepSeek V4 Pro is used only in task generation, to filter out tasks that are too easy. Because the generation pipeline is fully automatic, filtering is a major cost. Each candidate task needs several full agent runs, repeated over further rounds of refinement for tasks that are too easy. Using a single model keeps this cost manageable. It is a trade-off between construction efficiency and filtering quality.

Difficulty labeling, however, serves a different purpose from task filtering. Filtering only decides which generated tasks are hard enough to keep, while the difficulty labels describe the released benchmark. For labeling, we agree that a single model can introduce bias, so we relabeled difficulty with all models. Each task's difficulty is now computed from three runs of all five models, and the same labels are used for every model. The cross-model labels make the difficulty levels better in three ways.

  • Balanced levels. The tasks now split into 34 Hard, 36 Medium and 35 Easy tasks, while the DeepSeek V4 Pro labels put 57 of the 105 tasks in Hard.

  • Clear separation for every model. Each model's mean score falls steadily from Easy to Medium to Hard, for example from 74.9% to 51.9% to 24.3% for GPT 5.6 and from 70.7% to 41.8% to 15.4% for GLM-5.2.

  • Robustness to the choice of models. Leaving out any single model, including DeepSeek V4 Pro, changes at most 10 of the 105 labels.

We have rewritten Section 2.3 (Task Difficulty) with the new labels, updated Figure 3 to show them, and updated the labels in Appendix B (Task Catalog). The labeling sentences in Section 2.3 now read as follows.

"To assign difficulty levels, we summarize each task by one score that reflects all five models. The all-model score of a task is the mean of its 15 attempt scores, three from each model. [...] We then split the tasks into three levels by their all-model scores. Hard tasks score below 0.34, Medium tasks score from 0.34 to below 0.567, and Easy tasks score at least 0.567. We chose these thresholds so that each level holds about a third of the tasks, which keeps every group large enough to compare models. Because the labels combine all five models, no single model determines them. This matters because DeepSeek V4 Pro also filters tasks during construction."

(Weakness 3) CPU-only environments are far from real RSI settings, and dummy GPU environments would help.

Given our compute budget, we evaluate on CPU, which also keeps the benchmark cheap to run and reproducible for groups without GPU clusters. Within this setting, we still include GPU code. In 23 of the 105 tasks, across 13 task families, agents change production code that runs in GPU training and serving jobs, such as FSDP checkpointing, weight refit from the trainer to the rollout engine and Megatron tensor-parallel distillation. The evaluators check this code on CPU by replacing only the device, distributed and serving layers with CPU stand-ins, which is close to the dummy GPU environment you suggest. The submitted change is therefore the same code that would run on GPUs. Tasks that must execute on GPUs require sandboxes with GPU support, which our current evaluation infrastructure does not provide, but our task generation framework is not limited to CPU environments. We have added the following sentences at the end of the "Limited computational resources" paragraph in Appendix H (Limitations).

(Question 1) Why the maximum token length is limited to 32k.

As stated in Section 3.1 of the submitted version ("at most 32,768 output tokens per model call"), the 32,768 limit is the maximum number of output tokens per model call, not the context length, and both harnesses manage the conversation context themselves. This limit is enough for each model call in our experiments.

"CPU-only evaluation does not exclude GPU code. In 23 of the 105 tasks, agents change production code that runs in GPU training and serving jobs, such as FSDP checkpointing, weight refit to the rollout engine, and Megatron tensor-parallel distillation, and the evaluators check this code on CPU by replacing only the device, distributed, and serving layers. Tasks that must execute on GPUs require sandboxes with GPU support, such as Docker containers with GPU access. The task generation framework does not depend on CPU environments, so institutions with such infrastructure can use it to generate GPU tasks on their own codebases."

Thank you again for your reviews. We would like to briefly report the evaluations that we added during the discussion. We have added GPT 6 Astra with max reasoning effort, with three runs on every task, and we have rerun DeepSeek V4 Pro three times with its default reasoning effort under the same evaluation settings as the other models. Table 2 and all result figures now compare six models under the same protocol. GPT 6 Astra scores 48.4% and DeepSeek V4 Pro scores 42.8%, while GPT 5.6 Sol remains the highest at 52.4%. The difficulty labels now combine all six models and split the tasks into 33 Hard, 33 Medium and 34 Easy tasks. Each model's mean score still falls from Easy to Medium to Hard, and leaving out any single model changes at most 12 of the 100 labels.

GPT 6 Astra played no role in task generation. In contrast, GPT 5.6 Sol wrote the tasks, although its solver attempts were not used to filter them. This gives a comparison that is independent of task construction.

During the discussion, we also dropped some tasks whose evaluators we found to be incorrect, so all results now use the remaining 100 tasks.

Reviewer_HR1AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RSIBench-Code, an executable benchmark comprising 105 tasks designed to evaluate LLM agents' software engineering capabilities across training, post-training, model serving, and related infrastructure. While the work addresses a relevant problem, concerns remain regarding its evaluation methodology, novelty, and connection to RSI.

Critical strengths
  1. Relevant research problem. The benchmark targets complex engineering tasks in LLM development, extending the scope of traditional code-generation benchmarks.

  2. Comprehensive evaluation pipeline. The authors employ hidden evaluators, reference implementations, and containerized environments, and analyze task scores, resource consumption, and agent trajectories.

  3. Potential for reuse. The automated task construction and validation pipeline could be adapted to other machine learning engineering scenarios.

Critical weaknesses
  1. Model selection bias. DeepSeek V4 Pro is used for task selection, difficulty categorization, and final evaluation. This may systematically underestimate its performance and compromise the fairness of cross-model comparisons.

  2. Insufficient evaluator validation. The paper lacks independent audits and systematic validation of its evaluators. It is therefore unclear whether low scores consistently reflect model limitations rather than issues with evaluator design or task specifications.

  3. Overstated claims. The benchmark primarily covers two public repositories and evaluates agents in CPU-based environments. These results are insufficient to support broad claims about enterprise-level LLM development capabilities or RSI bottlenecks.

  4. Insufficient substantive connection to RSI. The paper frames RSI as its central motivation, yet its tasks closely resemble those in general-purpose coding benchmarks such as SWE-Bench, primarily evaluating agents' ability to complete engineering tasks in existing codebases. The benchmark does not directly evaluate recursive self-improvement processes or the capability gains resulting from such processes. Consequently, RSI appears to serve more as a motivating label than as a substantive component of the benchmark's design and evaluation objectives. The authors should clarify how the benchmark captures the distinctive requirements of RSI.

  5. Potential data contamination. The benchmark relies on public repositories such as verl and sglang, whose code or solutions may have appeared in the training data of the evaluated models. The authors should audit potential contamination and validate their results on independent repositories or previously undisclosed tasks.

Questions for authors

See Weaknesses

5 comments

Thank you for your careful review. We appreciate your recognition of the research problem, the evaluation pipeline and the reusability of our construction pipeline. Your comments help us make our work more complete.

Several reviewers suggested evaluating every model with three runs. We have completed two extra runs for each model that had one run, so every model now has three runs per task. We have uploaded the revised PDF with the updated results and all changes. We address each point below.

(Weakness 1) Model selection bias from using DeepSeek V4 Pro for task selection, difficulty labeling and evaluation.

DeepSeek V4 Pro is used only in task generation, to filter out tasks that are too easy. Because the generation pipeline is fully automatic, filtering is a major cost. Each candidate task needs several full agent runs, repeated over further rounds of refinement for tasks that are too easy. Using a single model keeps this cost manageable. It is a trade-off between construction efficiency and label quality.

To make sure the tasks are hard beyond DeepSeek V4 Pro, we also evaluated four state-of-the-art models not used in task generation, GPT 5.6, Hy4, GLM-5.2 and DeepSeek V4.1 Flash. The low scores are not specific to DeepSeek V4 Pro. Over three runs, these four models score from 42.9% to 50.6%, far below the 99.95% of the reference solutions (Figure 1, Table 2 and Figure 4). We have updated these results in Section 3.2 and Appendix E with the new runs. Difficulty labels also no longer depend on DeepSeek V4 Pro. We have relabeled difficulty with all five models (Section 2.3). Each task's difficulty is now computed from three runs of all five models, and the same labels are used for every model. The tasks now split into 34 Hard, 36 Medium and 35 Easy tasks. Each model's mean score falls steadily from Easy to Medium to Hard, for example from 74.9% to 51.9% to 24.3% for GPT 5.6. Leaving out any single model, including DeepSeek V4 Pro, changes at most 10 of the 105 labels.

(Weakness 2) No independent audit or validation of the evaluators.

We manually audited tasks end to end, covering the generation prompts, task cases, evaluation environment setup and final scores. To support automatic task generation, we also used the findings of this audit to iterate on the evaluator generation prompt, so that later evaluators need as little human correction as possible. We have added the following subsection, "Evaluator Audit", to Appendix D (Evaluation Implementation).

"A score is meaningful only if the evaluator credits the required behavior and nothing else. Human reviewers therefore audited tasks along their full evaluation path, and we used their findings to iterate on the evaluator generation prompt, so that automatic task generation needs as little human labor as possible. The audit covers four parts of each task. First, for generation prompts, we checked that each task specification and instruction states every behavior the evaluator checks, so that no check depends on unstated requirements. Second, for task cases, we checked that each hidden test case matches the instruction and that its expected output is correct. Third, for the evaluation environment, we checked that the solver and verifier containers build offline with pinned dependencies, and that the evaluator runs within its time and memory limits. Fourth, for final scores, we inspected low-scoring and failed attempts to confirm that lost credit comes from the submission rather than from the evaluator, and that evaluator errors are reported as errors rather than as zero scores. We also run two controls on every task, the reference solution and the unchanged starting code, and report their scores in Section 3. Reference solutions receive 99.95% on average, so the evaluators rarely reject correct behavior, and the unchanged starting code receives 26.2%, so the evaluators give limited credit without work on the task."

(Weakness 3) Claims about enterprise-level LLM development and RSI are too broad for two public repositories and CPU-only environments.

Our tasks include both CPU and GPU code. In 23 of the 105 tasks, across 13 task families, agents change production code that runs in GPU training and serving jobs. Examples include FSDP checkpointing in checkpoint-resume-state-repair, weight refit from the trainer to the rollout engine with KV-cache release in atomic-weight-refit-feature, and Megatron tensor-parallel distillation in topk-gradient-contract-tests. Evaluating these tasks by running GPU jobs would take massive compute and time for every attempt. We therefore evaluate the code directly. The evaluators check the behavior of this code on CPU by replacing only the device, distributed and serving layers with CPU stand-ins, so the submitted change is the same code that would run on GPUs. This greatly reduces evaluation time and difficulty, and keeps the benchmark cheap to run and reproducible for groups without GPU clusters. Tasks that must execute on GPUs need sandboxes with GPU support, and our framework can generate them wherever such infrastructure is available.

We have added the following sentences at the end of the "Limited computational resources" paragraph in Appendix H (Limitations).

"CPU-only evaluation does not exclude GPU code. In 23 of the 105 tasks, agents change production code that runs in GPU training and serving jobs, such as FSDP checkpointing, weight refit to the rollout engine, and Megatron tensor-parallel distillation, and the evaluators check this code on CPU by replacing only the device, distributed, and serving layers. Tasks that must execute on GPUs require sandboxes with GPU support, such as Docker containers with GPU access. The task generation framework does not depend on CPU environments, so institutions with such infrastructure can use it to generate GPU tasks on their own codebases."

(Weakness 4) Weak connection to RSI, with tasks that resemble SWE-bench.

RSI covers many aspects, and RSIBench-Code focuses on the coding tasks that support it. RSIBench-Code captures the distinctive requirements of these tasks in the following ways.

  • Goal. SWE-bench-style benchmarks evaluate how well models can be used for general software engineering. RSIBench-Code evaluates whether models can improve models. Each task is one step of how models are improved in practice, such as repairing a reward computation or comparing training choices in a controlled experiment, so an agent that solves these tasks does the improvement work that RSI requires, on the code that trains models like itself.

  • Required abilities. Improving models requires more than issue resolution. Beyond implementation and repair, our tasks require test writing, controlled experiments and execution-path alignment across many categories of LLM development (Figure 2), and many of them are harder end-to-end tasks. For example, atomic-weight-refit-feature-hard asks the agent to complete the weight refit contract from trainer transfer to rollout activation, so that requests already in flight stay on their admitted weight version and a failure at any stage rolls back cleanly. This requires coordinated changes across the trainer, the checkpoint engine and the rollout workers.

  • Task source. SWE-bench builds tasks from public GitHub issues and pull requests. Model improvement work has no such public source, and even paper re-implementation and post-training benchmarks, such as PaperBench and PostTrainBench, build their tasks from public papers, models and datasets. Our tasks come from real tasks in enterprise LLM development. These tasks were needed to keep model training working in practice, which gives a prior motivation for their relevance. One such task may be scattered across several agent sessions, so we reconstruct each complete task from this context with curated framework.

  • Evaluation. We adopt a SWE-style evaluation for the efficiency discussed under Weakness 3. SWE-bench verifies tasks with the tests from the original pull request. LLM development code often has no tests for the target behavior, so our framework writes and validates hidden evaluators, including evaluators for test writing and controlled experiments. We manually audited these evaluators (Weakness 2), passing them indicates that the task is completed and that normal model training can resume.

  • Supporting references. Appendix G (Related Work) already compares RSIBench-Code with SWE-bench-style benchmarks, including SWE-bench, SWT-Bench and SWE-smith (Appendix G.2 and G.3), and Reviewer QfF2 also highlights our enterprise-level task sources as a strength.

Our design keeps evaluation efficient and avoids excessive evaluation time. We therefore check the most essential observable outcomes rather than every aspect of the development process. This does not mean that our tasks are disconnected from RSI.

To illustrate this better in the paper, we have added the following paragraph at the end of Appendix G.3 (Evaluating Agents on LLM-Development Tasks).

"SWE-bench and its variants build tasks from public GitHub issues and pull requests in general-purpose repositories and mainly evaluate issue resolution. RSIBench-Code instead targets the code that trains and serves LLMs, and solving its tasks often requires knowledge beyond general software engineering. For example, repairing checkpoint evaluation during training requires reasoning about asynchronous scheduling, bounded concurrency, and checkpoint identity. Its tasks are reconstructed from enterprise-level LLM-development work, and each task adds a mechanism that does not exist in the public codebase. Beyond the released tasks, companies can use the full task generation pipeline to build private benchmarks from their own development tasks and codebases, which reflect their workloads more closely than benchmarks built from public repositories."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.