- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors present RSIBench-Infra, a benchmark for end-to-end "production-level" training and inference optimization. Agents submit their changes as an OCI image layer while a separate processes performs the end-to-end evaluation. The authors find analyze common failure modes across tasks and agents under a RSI (recursive self-improvement) framing.
Critical strengths
The authors present a wide range of tasks and settings and the evaluation design is quite clean (especially the OCI layer evaluation).
Critical weaknesses
With creating benchmarks for RSI and RSI-adjacent tasks becoming crowded rapidly, it is difficult to confidently say that this benchmark does something that past benchmarks have not explored already.
The paper fundamentally makes bold claims about RSI and production-level work that leaves much to be desired beyond the scale of the tasks. For instance, there are many possible confounders that may affect results such as agent harnesses, operational compliance, the task-specific skills that seem to be given to the agents (which opens its own can of worms), along with inherent stochastic of agent models. While the hardware and models used are "production-level", the workload and overall evaluation itself would be difficult to be considered to be "production-level", which generally introduce many more obstacles beyond the seemingly static and fixed nature of the benchmark.
Questions for authors
-
[Clarification] The leaderboard compares configurations of models and agent harnesses (e.g., Codex + GPT-6 Astra, Claude Code + Opus 5, and OpenCode + etc). How should readers interpret such differences between configurations? In other words, what exactly is the benchmark measuring and to what extent would it actually support conclusions about model capability as opposed to the other confounders?
-
[Clarification] The paper motivates RSIBench-Infra as evaluating a component of recursive self-improvement, but the experiments themselves do not seem to close the loop. Could the authors clarify what makes this benchmark measure recursive self-improvement, as opposed to infrastructure engineering? To be clear, I would find both benchmarks to be valuable, there just seems to be a slight mismatch in what is being presented and what is being evaluated.
-
[Clarification] Although the paper boasts a "production scale", it is still difficult to separate the contributions of this paper from its surrounding works regarding automating AI research and development using AI agents. In particular, 2607.20468 and 2603.08640 respectively focus on benchmarking AI agents' ability to perform end-to-end inference optimization and post-training, making it unclear where RSIBench-Infra makes its mark in the literature besides being a combination of two existing benchmarks. Note that there are many other adjacent works, as the paper itself states, which makes this question even more pressing. While the breadth of tasks and the sizes of models used make the findings still somewhat valuable, it is not clear what this paper finds that other papers have not already found out, which is something I consider to be a key criteria for benchmark papers.
5 comments
Author Response to Reviewer s5d6 (Part 2)
Q1: What does the leaderboard measure: model capability or the whole configuration?
Thank you for this thoughtful clarification question. The leaderboard measures a complete agent configuration's ability to diagnose, implement, validate, and package a working system improvement within eight hours. Matching Terminal-Bench [9] and InferenceBench [10], this lets native harnesses (Codex, Claude Code) use their intended tool interfaces. At the same time, our experiments explicitly disentangle all four confounders you noted:
- Model vs. harness: Four of the six Table 2 entries share the exact same OpenCode 1.18.18 harness yet span Faster Rates from 6/20 (DeepSeek V4 Flash) to 11/20 (Qwen3.8-Max), isolating model capability within a fixed scaffold. Conversely, holding GPT-6 Astra and the skill fixed on the five-task Lite Set, Codex reached 1.5423 ± 0.2172× on Qwen3.8-27B (1.1263 ± 0.0602× median) whereas OpenCode failed the quality gate (0.0000×; 1.0952 ± 0.0351× median), quantifying harness impact on valid delivery.
- Guidance ("skill") ablation: As noted in W2, removing the skill on Codex + GPT-6 Astra yields 1.2054 ± 0.0377× vs. 1.1263 ± 0.0602× with it.
- Inherent stochasticity: Three full 20-task runs (60 trials each) for Codex + GPT-6 Astra and OpenCode + Qwen3.8-Max both average 10.67 Faster tasks (±1.15 and ±0.58; Appendix G, Table 10); we now state next to Table 2 (Section 4.2, in blue) that small single-run gaps are not a strict ranking.
- Operational compliance vs. optimization depth: Clean OCI packaging and multi-rank launch compliance are intrinsic to real systems delivery; to make outcomes transparent, our rubric-based LLM-judge audit (Appendix F.1, Table 8) explicitly separates code/kernel implementations (62/120 trials), flag-only tuning (35/120), and zero-score deployment/quality failures (39/120).
Q2: Does this benchmark demonstrate recursive self-improvement?
Thank you for highlighting this framing distinction—we completely agree. Our experiments evaluate infrastructure engineering as an enabling component of RSI, rather than a closed multi-generation loop. To align presentation with evaluation, we removed "Their Own" from the title and revised Section 1 (in blue) to state that each trial evaluates a single bounded optimization episode and leaves iterative loop-closing to future work. While API-hosted models (GPT-6 Astra, Opus 5) run on provider-managed stacks that cannot be swapped mid-evaluation, our OCI protocol makes closing the loop direct for self-hosted open-weights models—our suite already evaluates Kimi K3 optimizing Kimi K3 serving and GLM-5.3 optimizing GLM-5.3 serving, where accepted OCI layers can directly serve subsequent rounds.
Author Response to Reviewer s5d6 (Part 3)
Q3: What distinguishes RSIBench-Infra from InferenceBench and PostTrainBench, and what new findings emerge?
Thank you for pointing out these closely related benchmarks and emphasizing empirical findings as a key criterion. RSIBench-Infra differs in both optimization scope and the cross-layer systems findings that emerge only under full-stack, multi-accelerator evaluation:
| Dimension | InferenceBench [10] | PostTrainBench [11] | RSIBench-Infra |
|---|---|---|---|
| Target & workloads | Single-GPU text-LLM serving (4 scenarios on Mistral-7B + 2 transfers) | Post-training recipes/data to raise downstream benchmark scores (7 benchmarks × 4 models) | Quality-preserving infrastructure efficiency across 6 inference modalities (16 tasks) + SFT, GRPO, OPD training (4 tasks) |
| Software & hardware | Serving repo on 1× H100 | Training run on 1× H100 | 7 engine families (SGLang, vLLM, vLLM-Omni, TRT-LLM, SkyRL, verl) on 1–32 H20 GPUs & Ascend 910B2 NPUs |
| Deliverable | Single-node server | Model checkpoint | Sandbox-wide OCI layer deployed across clean ranks under isolated semantic/numerical gates |
Unlike PostTrainBench, which alters training algorithms or data to improve downstream accuracy, our training tasks hold the objective and pointwise loss/grad-norm curves (<2% / <15%) fixed to evaluate systems execution efficiency. Evaluating this broader surface reveals four empirical findings not captured by prior kernel or single-GPU benchmarks:
- Code-level edits—not just flag tuning—drive most validated wins: Nearly two-thirds (36/57 = 63.2%) of Faster outcomes require modifying kernels, fusion, residency, or engine scheduling paths (58.1% Faster rate across 62 implementation trials), compared with 31.6% (18/57) from configuration flags alone (Appendix F.1, Table 8).
- A wide gap between profiling diagnosis and end-to-end speedup: Agents run accelerator profilers in 64.2% (77/120) of trials and complete closed-loop profile–diagnose–edit–validate cycles in 70.8% (85/120), yet only 47.5% (57/120) exceed 1.10×: local operator wins are frequently absorbed by CUDA-graph boundaries, host dispatch overhead, or unaddressed cross-role bottlenecks (e.g., reallocating idle GPUs to the teacher pool in 16-GPU OPD yields 1.26×, whereas local training-side fusion yields 1.01×).
- Long-horizon stateful and multi-rank failure modes invisible to unit tests: Locally faster edits break under full-system execution—such as FP8 attention (+13.7% speed) passing single-step checks but drifting across 16 autoregressive world-model chunks (MAD 1.96 → 53.27), recycled Python object IDs corrupting static CUDA-graph KV caches (π₀.₅, Appendix D), and four 1.27×–1.59× inference runs rejected by independent quality gates alongside multi-rank packaging/port failures (39/120 zero-score runs).
- Quantified distance to hardware Speed-of-Light: Under the anchored SOL Score of SOL-ExecBench (scoring baseline = 0.500000, SOL bound = 1.000000), even the strongest agent reaches 0.554755 (with Lite Set baselines starting at 35.2%–67.2% of SOL; Appendix I, Tables 12–13), establishing substantial remaining cross-layer headroom.
References
- [1] Lin et al., SOL-ExecBench, arXiv:2603.19173, 2026 (url).
- [2] Xing et al., FlashInfer-Bench, arXiv:2601.00227, 2026 (url).
- [3] Meng et al., RSIBench-Data, arXiv:2607.25886, 2026 (url).
- [4] RSI-Exam Team, RSI-Exam v0.1, 2026 (url).
- [5] RSI-Exam Team, Paged Ragged GQA Decode, 2026 (url).
- [6] Red Hat, AI Inference 3.5: llm-d Patterns, Ch. 10 (url).
- [7] Khalil et al., Private LLM Server for SMBs, arXiv:2512.23029, 2025 (url).
- [8] Inferact, Reflection & NVIDIA, Reflection Beam on vLLM, 2026 (url).
- [9] Merrill et al., Terminal-Bench, arXiv:2601.11868, 2026 (url).
- [10] Yeon et al., InferenceBench, arXiv:2607.20468, 2026 (url).
- [11] Rank et al., PostTrainBench, arXiv:2603.08640, 2026 (url).
Author Response to Reviewer s5d6 (Part 1)
We thank Reviewer s5d6 for recognizing the breadth of our tasks and our clean OCI layer evaluation design. Below we address all concerns and questions across three parts (manuscript updates are highlighted in blue).
Thank you for raising this question on positioning within a rapidly growing area. We distinguish RSIBench-Infra along two axes:
Thank you for pointing out these potential confounders and scope boundaries. We address workload operating points and task guidance directly: