acceptodds
Under review as a conference paper at ICLR 2027

RSIBench-Infra: Can Agents Optimize Their Own Training and Inference Infrastructure at Production Scale?

Abstract

Improving the efficiency of training and inference infrastructure is a practical component of recursive self-improvement. Existing benchmarks for agent-driven optimization largely focus on individual software layers, leaving unclear whether agents can deliver end-to-end performance gains under production workloads. We introduce RSIBench-Infra, a benchmark of 20 training and inference tasks spanning diverse model families and modalities, with deployments of up to 32 accelerators on NVIDIA GPUs and Ascend NPUs. Agents start from runnable baselines derived from official recipes and can modify the software stack across engine, kernel, and instruction levels. An OCI image-layer submission protocol captures these changes for reproducible deployment, while independent evaluation measures end-to-end performance under task-specific quality constraints. Across six agent–model configurations and 120 trials, 57 trials achieve validated speedups above 1.10×, with the best configuration exceeding this threshold on 60% of tasks. These results demonstrate that current agents can improve production- scale training and inference performance, but identifying effective optimization strategies and consistently translating local improvements into end-to-end gains remain challenging.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Coding Agent, Infrasturcture Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_s5d6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors present RSIBench-Infra, a benchmark for end-to-end "production-level" training and inference optimization. Agents submit their changes as an OCI image layer while a separate processes performs the end-to-end evaluation. The authors find analyze common failure modes across tasks and agents under a RSI (recursive self-improvement) framing.

Critical strengths

The authors present a wide range of tasks and settings and the evaluation design is quite clean (especially the OCI layer evaluation).

Critical weaknesses

With creating benchmarks for RSI and RSI-adjacent tasks becoming crowded rapidly, it is difficult to confidently say that this benchmark does something that past benchmarks have not explored already.

The paper fundamentally makes bold claims about RSI and production-level work that leaves much to be desired beyond the scale of the tasks. For instance, there are many possible confounders that may affect results such as agent harnesses, operational compliance, the task-specific skills that seem to be given to the agents (which opens its own can of worms), along with inherent stochastic of agent models. While the hardware and models used are "production-level", the workload and overall evaluation itself would be difficult to be considered to be "production-level", which generally introduce many more obstacles beyond the seemingly static and fixed nature of the benchmark.

Questions for authors
  1. [Clarification] The leaderboard compares configurations of models and agent harnesses (e.g., Codex + GPT-6 Astra, Claude Code + Opus 5, and OpenCode + etc). How should readers interpret such differences between configurations? In other words, what exactly is the benchmark measuring and to what extent would it actually support conclusions about model capability as opposed to the other confounders?

  2. [Clarification] The paper motivates RSIBench-Infra as evaluating a component of recursive self-improvement, but the experiments themselves do not seem to close the loop. Could the authors clarify what makes this benchmark measure recursive self-improvement, as opposed to infrastructure engineering? To be clear, I would find both benchmarks to be valuable, there just seems to be a slight mismatch in what is being presented and what is being evaluated.

  3. [Clarification] Although the paper boasts a "production scale", it is still difficult to separate the contributions of this paper from its surrounding works regarding automating AI research and development using AI agents. In particular, 2607.20468 and 2603.08640 respectively focus on benchmarking AI agents' ability to perform end-to-end inference optimization and post-training, making it unclear where RSIBench-Infra makes its mark in the literature besides being a combination of two existing benchmarks. Note that there are many other adjacent works, as the paper itself states, which makes this question even more pressing. While the breadth of tasks and the sizes of models used make the findings still somewhat valuable, it is not clear what this paper finds that other papers have not already found out, which is something I consider to be a key criteria for benchmark papers.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer s5d6 (Part 1)

We thank Reviewer s5d6 for recognizing the breadth of our tasks and our clean OCI layer evaluation design. Below we address all concerns and questions across three parts (manuscript updates are highlighted in blue).

W1: Novelty relative to existing RSI and infrastructure benchmarks

Thank you for raising this question on positioning within a rapidly growing area. We distinguish RSIBench-Infra along two axes:

  • Relative to infrastructure benchmarks: Kernel suites such as SOL-ExecBench [1] and FlashInfer-Bench [2] evaluate isolated operators against hardware limits or fixed operator bindings, while single-GPU serving benchmarks modify one repository on a single device. Our core contribution is an end-to-end multi-accelerator system delivery protocol: agents modify any layer of the sandbox—engine Python/C++, system libraries, and compiled CUDA/Triton/Ascend artifacts—and package them as a single OCI filesystem layer, deployed identically across clean multi-GPU/NPU ranks and gated on task-level semantic or numerical correctness before throughput counts.
  • Relative to RSI benchmarks: Existing RSI suites target orthogonal axes or isolated kernels: RSIBench-Data [3] varies training data under a fixed stack, and RSI-Exam's ML-infra tasks [4, 5] (e.g., paged-GQA decode) score single-operator latency behind a fixed function signature. RSIBench-Infra tests whether an agent can profile a complete distributed serving or post-training system, coordinate cross-layer changes, and deliver a deployable stack with validated end-to-end gains.

W2: Production relevance, workload scope, and skill non-leakage

Thank you for pointing out these potential confounders and scope boundaries. We address workload operating points and task guidance directly:

  • Workload diversity and low-concurrency relevance: Spanning six stacks (SGLang, vLLM, vLLM-Omni, TensorRT-LLM, SkyRL, verl), six inference modalities, and three post-training objectives across 1.7B–2.8T models on 1–32 GPUs/NPUs, our suite exercises diverse scheduling, memory, and collective bottlenecks. While we explicitly distinguish our static benchmark from live multi-tenant traffic management, dedicated single-node and low-concurrency serving are well-established production operating points (Red Hat llm-d [6]; SMB private Qwen3-30B serving [7]; vLLM's batch-1 TP8 optimization cutting Reflection Beam decode from 8.95 ms to 3.30 ms [8]). Furthermore, auditing nine matched baseline-vs.-candidate E2E p99 latency records across four tasks confirms that every Faster submission simultaneously reduced p99 tail latency by 6.3%–16.1% rather than trading latency for throughput.
  • Task guidance ("skills") does not leak solutions: The supplied skill provides only a generic profile–diagnose–validate methodology, and our task authoring rules strictly forbid hotspot rankings or optimization hints in prompts. Empirically, ablating the skill on our five-task single-H20 Lite Set (Appendix I, Table 12) shows Codex + GPT-6 Astra achieves 1.2054 ± 0.0377× without the skill vs. 1.1263 ± 0.0602× with it, confirming that agent gains do not depend on hand-crafted guidance.
AuthorsOpenReview ↗

Author Response to Reviewer s5d6 (Part 2)

Q1: What does the leaderboard measure: model capability or the whole configuration?

Thank you for this thoughtful clarification question. The leaderboard measures a complete agent configuration's ability to diagnose, implement, validate, and package a working system improvement within eight hours. Matching Terminal-Bench [9] and InferenceBench [10], this lets native harnesses (Codex, Claude Code) use their intended tool interfaces. At the same time, our experiments explicitly disentangle all four confounders you noted:

  • Model vs. harness: Four of the six Table 2 entries share the exact same OpenCode 1.18.18 harness yet span Faster Rates from 6/20 (DeepSeek V4 Flash) to 11/20 (Qwen3.8-Max), isolating model capability within a fixed scaffold. Conversely, holding GPT-6 Astra and the skill fixed on the five-task Lite Set, Codex reached 1.5423 ± 0.2172× on Qwen3.8-27B (1.1263 ± 0.0602× median) whereas OpenCode failed the quality gate (0.0000×; 1.0952 ± 0.0351× median), quantifying harness impact on valid delivery.
  • Guidance ("skill") ablation: As noted in W2, removing the skill on Codex + GPT-6 Astra yields 1.2054 ± 0.0377× vs. 1.1263 ± 0.0602× with it.
  • Inherent stochasticity: Three full 20-task runs (60 trials each) for Codex + GPT-6 Astra and OpenCode + Qwen3.8-Max both average 10.67 Faster tasks (±1.15 and ±0.58; Appendix G, Table 10); we now state next to Table 2 (Section 4.2, in blue) that small single-run gaps are not a strict ranking.
  • Operational compliance vs. optimization depth: Clean OCI packaging and multi-rank launch compliance are intrinsic to real systems delivery; to make outcomes transparent, our rubric-based LLM-judge audit (Appendix F.1, Table 8) explicitly separates code/kernel implementations (62/120 trials), flag-only tuning (35/120), and zero-score deployment/quality failures (39/120).

Q2: Does this benchmark demonstrate recursive self-improvement?

Thank you for highlighting this framing distinction—we completely agree. Our experiments evaluate infrastructure engineering as an enabling component of RSI, rather than a closed multi-generation loop. To align presentation with evaluation, we removed "Their Own" from the title and revised Section 1 (in blue) to state that each trial evaluates a single bounded optimization episode and leaves iterative loop-closing to future work. While API-hosted models (GPT-6 Astra, Opus 5) run on provider-managed stacks that cannot be swapped mid-evaluation, our OCI protocol makes closing the loop direct for self-hosted open-weights models—our suite already evaluates Kimi K3 optimizing Kimi K3 serving and GLM-5.3 optimizing GLM-5.3 serving, where accepted OCI layers can directly serve subsequent rounds.

AuthorsOpenReview ↗

Author Response to Reviewer s5d6 (Part 3)

Q3: What distinguishes RSIBench-Infra from InferenceBench and PostTrainBench, and what new findings emerge?

Thank you for pointing out these closely related benchmarks and emphasizing empirical findings as a key criterion. RSIBench-Infra differs in both optimization scope and the cross-layer systems findings that emerge only under full-stack, multi-accelerator evaluation:

DimensionInferenceBench [10]PostTrainBench [11]RSIBench-Infra
Target & workloadsSingle-GPU text-LLM serving (4 scenarios on Mistral-7B + 2 transfers)Post-training recipes/data to raise downstream benchmark scores (7 benchmarks × 4 models)Quality-preserving infrastructure efficiency across 6 inference modalities (16 tasks) + SFT, GRPO, OPD training (4 tasks)
Software & hardwareServing repo on 1× H100Training run on 1× H1007 engine families (SGLang, vLLM, vLLM-Omni, TRT-LLM, SkyRL, verl) on 1–32 H20 GPUs & Ascend 910B2 NPUs
DeliverableSingle-node serverModel checkpointSandbox-wide OCI layer deployed across clean ranks under isolated semantic/numerical gates

Unlike PostTrainBench, which alters training algorithms or data to improve downstream accuracy, our training tasks hold the objective and pointwise loss/grad-norm curves (<2% / <15%) fixed to evaluate systems execution efficiency. Evaluating this broader surface reveals four empirical findings not captured by prior kernel or single-GPU benchmarks:

  • Code-level edits—not just flag tuning—drive most validated wins: Nearly two-thirds (36/57 = 63.2%) of Faster outcomes require modifying kernels, fusion, residency, or engine scheduling paths (58.1% Faster rate across 62 implementation trials), compared with 31.6% (18/57) from configuration flags alone (Appendix F.1, Table 8).
  • A wide gap between profiling diagnosis and end-to-end speedup: Agents run accelerator profilers in 64.2% (77/120) of trials and complete closed-loop profile–diagnose–edit–validate cycles in 70.8% (85/120), yet only 47.5% (57/120) exceed 1.10×: local operator wins are frequently absorbed by CUDA-graph boundaries, host dispatch overhead, or unaddressed cross-role bottlenecks (e.g., reallocating idle GPUs to the teacher pool in 16-GPU OPD yields 1.26×, whereas local training-side fusion yields 1.01×).
  • Long-horizon stateful and multi-rank failure modes invisible to unit tests: Locally faster edits break under full-system execution—such as FP8 attention (+13.7% speed) passing single-step checks but drifting across 16 autoregressive world-model chunks (MAD 1.96 → 53.27), recycled Python object IDs corrupting static CUDA-graph KV caches (π₀.₅, Appendix D), and four 1.27×–1.59× inference runs rejected by independent quality gates alongside multi-rank packaging/port failures (39/120 zero-score runs).
  • Quantified distance to hardware Speed-of-Light: Under the anchored SOL Score of SOL-ExecBench (scoring baseline = 0.500000, SOL bound = 1.000000), even the strongest agent reaches 0.554755 (with Lite Set baselines starting at 35.2%–67.2% of SOL; Appendix I, Tables 12–13), establishing substantial remaining cross-layer headroom.

References

  • [1] Lin et al., SOL-ExecBench, arXiv:2603.19173, 2026 (url).
  • [2] Xing et al., FlashInfer-Bench, arXiv:2601.00227, 2026 (url).
  • [3] Meng et al., RSIBench-Data, arXiv:2607.25886, 2026 (url).
  • [4] RSI-Exam Team, RSI-Exam v0.1, 2026 (url).
  • [5] RSI-Exam Team, Paged Ragged GQA Decode, 2026 (url).
  • [6] Red Hat, AI Inference 3.5: llm-d Patterns, Ch. 10 (url).
  • [7] Khalil et al., Private LLM Server for SMBs, arXiv:2512.23029, 2025 (url).
  • [8] Inferact, Reflection & NVIDIA, Reflection Beam on vLLM, 2026 (url).
  • [9] Merrill et al., Terminal-Bench, arXiv:2601.11868, 2026 (url).
  • [10] Yeon et al., InferenceBench, arXiv:2607.20468, 2026 (url).
  • [11] Rank et al., PostTrainBench, arXiv:2603.08640, 2026 (url).
Reviewer_d2TvOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper presents RSIBench-Infra, measuring the ability of agents to self-improve their own training and inference stack. It combines reproducible deployment of agent changes and independent quality checks to measure end-to-end performance. Experiments highlight the difficulty of turning local improvements into reliable system gains.

Critical strengths
  1. Unlike prior works that target a single level of the underlying software stack, such as GPU kernels in the serving traces, this benchmark evaluates agent-driven optimization with the whole training/inference stack in mind, which is a much harder task to address.
  2. By grounding tasks in real workloads and existing software stacks, the benchmarked improvements, with proper validation in place, can potentially be deployed into real-world settings that directly cut the training and inference costs, beyond model ranking alone.
Critical weaknesses

W1: My first concern is on the training tasks construction. I believe the four training tasks are not sufficiently representative of training-stack optimization. They cover a pretty narrow range of workloads.

  • For instance, from Table 3, even the largest prompt / response limit is 2048 / 8192 tokens. Long-context RL training is therefore unevaluated, despite its importance in production.
  • Besides that, all four tasks train in BF16, leaving lower-precision approaches such as FP8 training unexplored, even though the H20 GPUs support FP8 computation.
  • Furthermore, training is tested only on 8 or 16 H20 GPUs, leaving many optimizations relevant to larger-scale training untested. This limits how representative the benchmark is of production-level training infrastructure.

W2: My other concern is that speedup alone as the metrci provides limited insights into how well agents optimize an end-to-end software stack. It measures the final outcome but does not reveal which underlying optimization capabilities are improving.

  • The evaluation should also assess how well agents use profilers to diagnose bottlenecks, monitor training runs, design validation checks, and implement kernels. Aggregate speedups do not reveal these individual behaviors.
  • On the other hand, Megatron backend in SkyRL and verl offer many optimization flags. Enabling one flag vs. developing that optimization can yield similar speedups but require very different capabilities. Such flags can benefit general workloads and thus are not inherently benchmark hacking, yet we can't reliably distinguish the two.
Questions for authors
  1. What are the feedback signals to the agents? In the traces, did any agent leverage profiling tools like nsys to investigate opportunities for improvement? In other words, did the agents use information beyond step-time alone to further improve?
  2. How are the optimization categories in Figure 3 defined? For instance, what distinguishes "Training async / overlap" from "Communication overlap", and which operations are being overlapped in each?
  3. What evidence from the agent traces supports the abstract’s claim that translating local improvements into end-to-end gains remains challenging? Shouldn't improvements at each stack layer (i.e. faster kernels + better overlap) gradually build up into end-to-end gains?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer d2Tv (Part 1)

We thank Reviewer d2Tv for recognizing the difficulty and practical value of whole-stack optimization. We address all concerns below across three parts (manuscript additions are highlighted in blue).

W1: Representativeness of the training tasks

Thank you for highlighting these important post-training regimes. We address context length, cluster scale, and FP8 precision below:

  • Regarding longer contexts and larger deployments (W1a & W1c)

    1. Substantial optimization headroom remains at the current scale. Even at 8–16 GPUs, top agents achieve >1.10×>1.10\times speedup on at most 2/4 training tasks. The best SOL Scores per task are 0.5517 for text GRPO, 0.5422 for multimodal GRPO, and 0.7394 for OPD, where the baseline anchors at 0.5 and the analytical SOL reference at 1.0. These results indicate substantial remaining headroom, particularly for RL, while SFT is closer to its reference at 0.9584.

    2. Larger deployments introduce substantial reproducibility costs. Evaluating ≥64\ge 64-GPU multi-node topologies would extend coverage to large-scale PP/CP and cross-switch collective contention. However, a single 256-GPU trial would require over 2,000 GPU-hours, limiting reproducibility across labs. We now explicitly bound our training claims to 8–16 GPU single- and two-node regimes in Section 5 (Limitations).

    3. A new long-context RL problem is ongoing. We are extending our existing text GRPO task using Qwen3-4B on 16 H20 GPUs, targeting a 32k sequence budget while retaining the same end-to-end timing and correctness gates. We will share results with the reviewers as soon as they become available.

  • Regarding lower-precision FP8 training (W1b): To evaluate lower-precision training on H20 Tensor Cores, we have added a MiMo-7B-RL FP8 SFT benchmark on 8 NVIDIA H20 GPUs, measuring end-to-end forward, backward, and optimizer throughput on fixed GSM8K data (batch size 64, max sequence length 1,024, 10 warmup + 106 timed steps) while preserving MTP auxiliary training and prescribed training semantics. Experimental evaluations on this new FP8 task are currently underway—the results are on the way, and we will update this thread and the manuscript as soon as the runs finish.

AuthorsOpenReview ↗

Author Response to Reviewer d2Tv (Part 2)

W2: Distinguishing optimization capabilities beyond final speedup

Thank you for this insightful suggestion. To move beyond outcome speedup alone, we have added three full-suite analyses across all 120 trials (6 configurations × 20 tasks) in Table 2, Section 4.2, Appendix F.1 (Tables 8–9), and Appendix I (Table 13):

  • SOL Score to quantify optimization headroom: Combining recorded execution traces with compute, memory, and communication models, we estimate task-level Speed-of-Light (SOL) bounds (e.g., Lite Set Oracle baselines achieve b/s=0.3520–0.6720b/s = 0.3520\text{–}0.6720, or 35.2%–67.2% of SOL; Appendix I, Table 12). Following the SOL Score definition on the SOL-ExecBench leaderboard [1], S(t)=1/[1+(t−tSOL)/(tb−tSOL)]S(t) = 1 / [1 + (t - t_{\mathrm{SOL}})/(t_b - t_{\mathrm{SOL}})] anchors the scoring baseline at 0.500000 and the SOL bound at 1.000000 (capped at 1.0, with failed/missing trials contributing 0 to the 20-task arithmetic mean; uncapped scores where edits like speculative decoding alter the reference graph are retained in Appendix I):
Agent modelGPT-6 AstraClaude Opus 5Qwen3.8-MaxKimi K3DeepSeek V4 FlashGLM-5.3
SOL Score0.5547550.4998440.4410880.4333190.4257660.373219
  • Separating code/kernel implementation from configuration-only tuning: Using GPT-6 Astra as an LLM judge with explicit operational rubrics over trajectories, OCI layers, and source diffs (Appendix F.1, Table 8), we separate submissions that modify execution code (custom Triton/CUDA kernels, operator fusion, layout/residency logic, or engine batching/sync paths) from those that only toggle existing runtime/Megatron flags:
Agent modelImplementation changesConfiguration-onlyNo confirmed opt.Insufficient / incompleteFull profiling loop
GPT-6 Astra1235012
Claude Opus 51451018
Qwen3.8-Max1162113
Kimi K31081116
GLM-5.31163014
DeepSeek V4 Flash475412
Total (/120)62 (51.7%)35 (29.2%)17 (14.2%)6 (5.0%)85 (70.8%)
Faster (>1.10×>1.10\times)36/62 (58.06%)18/35 (51.43%)——44/85 (51.76%)

Nearly two-thirds of all Faster wins across the benchmark (36/57 = 63.2%) come from substantive code or kernel implementations, whereas configuration-only tuning accounts for less than one-third (18/57 = 31.6%; 3 Faster runs have no confirmed new method). Including this explicit decomposition in Appendix F.1 ensures our benchmark directly distinguishes deep implementation capability (e.g., Opus 5 and Astra with 14 and 12 implementation runs) from flag selection (e.g., DeepSeek V4 Flash with 4).

  • Auditing closed-loop profiling and validation: Requiring concrete measurement output → bottleneck diagnosis → targeted edit → end-to-end performance/quality validation (including candidates properly reverted upon regression), 85/120 trials (70.8%) complete this full engineering loop and achieve a 51.76% (44/85) Faster rate versus 37.14% (13/35) otherwise.
AuthorsOpenReview ↗

Author Response to Reviewer d2Tv (Part 3)

Q1: What feedback do agents receive, and do they actually use profilers?

Thank you for this question—we find inspecting actual profiler feedback particularly informative. Agents routinely collect operator-level, phase-level, and host-side profiles beyond step time (Appendix F.1, Table 8; non-exclusive across 120 trajectories):

Tool / feedback typeConcrete tools in trajectoriesTrajectories (/120)Share
Accelerator profilingtorch.profiler, torch_npu, /start_profile, profile_sglang.py, nsys, msprof7764.2%
Phase timingDriver timing_sec, stage logs, perf_counter, CUDA events2924.2%
MicrobenchmarksIsolated GEMM/GEMV, attention, CUDA-graph, NCCL, preprocessing tests2924.2%
CPU profilingpy-spy, cProfile1210.0%

Specifically, we verified six nsys profile collection attempts across three trajectories. Concrete examples from Opus 5 illustrate how fine-grained signals guided edits (while respecting evaluation isolation, e.g., OPD self-tests are marked not_for_scoring and cannot access frozen gate assets):

  • Qwen3-VL: profile_sglang.py (report.json) showed the top four MLP FP8 GEMMs took 59.6% of device time (25.9% + 13.2% + 13.2% + 7.3%, within ~75% total FP8 GEMMs in Appendix F), motivating small-MM GEMM, fusion, and residency edits.
  • π₀.₅: profile_pi05.py and timings.py showed 22.62 ms device vs. 32.80 ms wall time (486 launches) in the prefix encoder plus 8,999 denoise launches (28.58 ms), isolating host launch overhead and ada-RMSNorm fusion targets.
  • OPD: train_profile_workload.py (selftest-summary.json) reported 17.53 / 28.37 / 20.34 s for sampling, teacher scoring, and optimization; reallocating teacher replicas cut local self-test teacher time to 15.1–16.4 s (final verification failed on a port conflict, scoring 0).

Q2: How are the optimization categories defined?

Thank you for asking for this clarification. All 21 categories in Figure 3 and our trajectory audits are assigned by an LLM judge (GPT-6 Astra) using explicit non-exclusive operational rubrics over OCI layers and trajectories (full table in Appendix F.1, Table 9). For the categories you highlighted:

CategoryOperational definitionTrajectory example
Training async / overlapConcurrent or asynchronous execution of distinct logical training stages (rollout, teacher scoring, weight sync, CPU datum prep, GPU shards)Moving CPU tensor conversion for forward_backward into a thread pool overlapped with the preceding GPU shard
Communication overlapOverlapping collective/transfer ops (all-gather, all-reduce, reduce-scatter, all-to-all, IPC) with compute/transfers across streams, or fusing collectivesFSDP all-gather overlapped with layer compute, multi-stream shared experts, or FlashComm fused collectives
Training update schedulingRestructuring update RPC granularity or stage resource quotas without cross-stage concurrencyReallocating teacher-pool GPU replicas or coalescing forward_backward calls

Q3: Why do local gains not necessarily accumulate into end-to-end gains?

Thank you for raising this central systems question. Agent trajectories expose three concrete barriers preventing isolated kernel or overlap gains from summing linearly:

  • Graph boundaries, launch overhead, and bandwidth ceilings: In Opus 5's π₀.₅ run (Appendix D), optimizing prefix K/V concatenation lowered isolated kernel time but gave zero gain in end-to-end A/B serving and was removed. Likewise, GLM-5.3's Qwen3-TTS run (Appendix F) stacked four optimization categories yet reached only 1.0690× due to enclosing predictor CUDA-graph boundaries and HBM bandwidth (Kimi K3 similarly stacked three edits for 1.0256×).
  • Cross-stage critical-path mismatch: On OPD (Appendix F), Astra diagnosed teacher scoring as ~43% of step time, yet its training-side fusion, memory, and wakeup edits yielded only 1.0107× because the teacher pool remained saturated; GLM-5.3 reassigned four idle GPUs within the 16-GPU budget to expand teacher replicas and reached 1.2588×.
  • Stateful numerical drift: In Qwen3.8-Max's LingBot run (Appendix F), FP8 FlashAttention passed single-step checks and raised speed by +13.7% (7.594 → 8.638 frames/s), but pixel MAD drifted across 16 autoregressive chunks from 1.96 to 53.27 (exceeding the 2.0 gate), forcing a revert.

References

  • [1] Lin et al., SOL-ExecBench, arXiv:2603.19173, 2026 (url).
AuthorsOpenReview ↗

Follow-up: Results on Additional Benchmark Tasks

Thank you again for suggesting that we expand our evaluation to cover low-precision training and long-context reinforcement learning. We have added two tasks to RSIBench-Infra—train-mimo-7b-sft-fp8 and train-qwen3-4b-longctxgrpo—and report their setups and the results below.

1. train-mimo-7b-sft-fp8

We evaluate full-parameter SFT of MiMo-7B on GSM8K (batch size 64) across 1x8 NVIDIA H20 GPUs over 116 steps (10 warmup + 106 timed). The optimizer and schedule match the BF16 reference: Adam (lr 1e-5, betas (0.9, 0.95), weight decay 0.1, grad clip 1.0, 10-step warmup with cosine decay), retaining the Multi-Token Prediction (MTP) auxiliary objective (one MTP layer, hard cross-entropy, loss scale 0.1, complete gradient path). We use the Transformer Engine hybrid FP8 recipe with tensorwise/current scaling across supported Transformer linear layers for forward activation, input-gradient, and weight-gradient computation, while embeddings, output head, normalization, softmax, loss, and optimizer states stay in BF16 or higher precision.

Performance is normalized against an independently calibrated FP8 Oracle achieving 1,788 steps/hour (speedup = candidate steps/hour / 1,788). The quality gate requires <= 2% maximum relative error on the main-loss curve and <= 15% on the global gradient-norm curve against an independent FP8 reference, alongside verified full FP8 linear coverage, finite MTP metrics, and valid submission artifacts. Opus 5.5 is excluded because it frequently flags and stops our evaluation session through frontier-llm.

Model (Harness)Throughput (steps/h)SpeedupStatus
Kimi K3 (OpenCode)2,3561.318Accepted
GPT-6 Astra (Codex)2,3201.298Accepted
Qwen3.8-max (OpenCode)N/A0Failed quality gate
DeepSeek V4 Flash (OpenCode)N/A0Failed quality gate
GLM-5.3 (OpenCode)N/AN/AFailed OCI format: duplicate skyrl-workspace/SkyRL path
  • Kimi K3 (+31.8% speedup): Enabled THD variable-length sequence packing (remove_microbatch_padding=true) to eliminate padding compute, paired with an equivalent packed MTP hard-CE implementation masking padded and cross-segment positions, and enabled Megatron gradient-reduction and parameter-gather overlap. Token-based microbatching was tuned for the fixed workload, while full recomputation was retained after direct measurement showed it was faster and lower-memory by enabling larger compute-efficient microbatches and avoiding dynamic-shape allocator stalls.
  • GPT-6 Astra (+29.8% speedup): Improved runtime memory handling via allocator reuse and expandable_segments, freeing headroom to recompute only the first 24 of 36 decoder layers. It alsoreduced service-side waiting via short-interval future polling and explicit CUDA synchronization while preserving the FP8 precision contract.

2. train-qwen3-4b-longctxgrpo

We evaluate full-parameter BF16 GRPO of Qwen/Qwen3-4B on DAPO-Math-17k across 2x8 NVIDIA H20 GPUs (96 GB/GPU) via SkyRL, co-locating Megatron training (baseline TP2xCP4xDP2) and vLLM rollout (eight TP2 engines). Each RL step processes 64 prompts x 8 responses (512 responses; minibatch 128, 1 PPO epoch), giving 4 updates/step and 12 updates over 3 timed steps (no warmup, <= 1,800 s/step). Microbatching and parallelism are tunable under a fixed 32,768-token budget (<= 2,048 prompt + <= 30,720 response/thinking), requiring >= 1 actual sequence of >= 29,492 tokens per step. Optimizer and GRPO semantics are fixed: Adam (lr 1e-5, betas (0.9, 0.999), eps 1e-8, wd 0.01, clip 1.0), group-std advantage norm, token-mean loss, and dual clip (0.8/1.28, coeff 10.0).

Speedup is normalized against the calibrated Oracle (3.257 RL steps/hour), and the admission gate replays three frozen trajectory batches across all 12 updates (requiring <= 2% loss and <= 15% grad-norm error). All four completed runs below passed admission with zero dropped responses (GPT-6 Astra is pending a valid native-Codex run).

Model (Harness)Throughput (RL steps/h)SpeedupMax seq len
Kimi K3 (OpenCode)4.1731.28130,981
GLM-5.3 (OpenCode)3.3781.03731,018
DeepSeek V4 Flash (OpenCode)3.3721.03530,981
Qwen3.8-max (OpenCode)3.1480.966631,018
  • Kimi K3 (+28.1% speedup): Reconfigured rollout from eight TP2 vLLM engines to four TP4 engines on the 16 GPUs—pooling KV-cache memory across 4 GPUs to mitigate 32K preemption bottlenecks—while raising GPU memory utilization (0.60 to 0.85), max concurrent sequences (32 to 96), and batched tokens (8,192 to 16,384), retaining the TP2/CP4 training layout.

We hope these new benchmark problems and analyses thoroughly address your questions, and we sincerely look forward to your reply.

Reviewer_kieVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks a practical question: given a real, working ML system, can a coding agent make it faster end to end without breaking it

The benchmark has 20 jobs (16 serving, 4 training: SFT, GRPO, distillation) across eight engines , covering chat, image, video, speech, world-model, and robot-action models from 1.7B to 2.8T parameters, on 1–32 accelerators (NVIDIA H20 and Ascend 910B2, up to four machines). Each job starts from the official, already-working recipe; the agent gets eight hours and may change anything, from engine code down to low-level instructions.

Grading is kept out of the agent's hands. Its changes are too scattered for a patch, so it submits them as one container layer, which a fresh job installs on a clean system, identically on every machine. A separate grader that never loads the layer checks quality first — long-context answers, closeness to reference outputs, intelligible speech, matching training loss curves — and only then measures speed; an LLM judge (GPT-5.6 sol) also scans the trajectory for cheating

Critical strengths

It tests the real job. Earlier benchmarks test one piece at a time: write a kernel, or build a server from scratch. Real infrastructure work is different: find where time goes in a large, working system, across the scheduler, the cache, the kernels, and the communication, and make the whole thing faster without breaking it. This is, to my knowledge, the first benchmark to test exactly that

submissions and grading are cleanly separated. An agent that can change anything cannot hand in a patch. its changes are scattered across code, libraries, and compiled files. Packing them into one container layer, installing that layer on a fresh system, and using the identical layer on every machine is the right fix

Faster only counts if the system is still correct. Every task has an independent quality check, applied before speed counts. Four submissions that were faster on paper were scored zero for failing it, and in one case an agent itself dropped a locally faster change because its errors accumulated over time

Critical weaknesses

I cannot tell the model apart from the harness. Each top entry pairs a different harness with a different model, and every agent also receives the authors' optimization playbook. So the ranking is about bundles, not models. One swap run (the same model in a second harness) would show how much is which.

I do not know how much room the baselines left. Official recipes are starting points, not tuned deployments, and some wins are switching on known features. The best-documented case, starts from a baseline slowed by its own padding path. Finding such issues is real work, but without even one expert-tuned comparison we cannot tell hard wins from low-hanging fruit

Production scale is mainly about model size. The text workloads run at concurrency 1–4, fire requests as fast as possible, and have no latency requirement — a speedup bought with worse tail latency would still count. Several multimodal jobs are tiny (one video for MiniMax-H3, four for Wan2.2, 32 images for Qwen3-VL). Gains under these conditions may not carry over to a loaded production service.

The side analyses and the title claim a bit too much. The method counts in Fig. 3 are hand labels on finished submissions; methods appear together, so they cannot show which one helped. The 24-hour study is two cases and shows nothing general. And the title's "their own" infrastructure overshoots: in most trials the agent is optimizing a different model's system, in a single pass, with no loop closing. "A component of RSI" — the phrasing the body mostly uses — is the right claim.

Questions for authors

How much of the ranking is the model, and how much is the harness? Could you run even one swap — the same model under a second harness, or a second model under Codex or Claude Code — and report how much the gap moves?

How tuned are the official-recipe baselines? On two or three tasks, what does a human expert (or the best configuration you know of) reach — i.e., how much headroom was there for the agents to find?

Your repeated runs make the top two configurations tie on average (10.67 Faster tasks each). Should Table 2 be read as a ranking at all — and would you be willing to say so in the main text, next to the table?

What are the numeric LongBench v2 acceptance thresholds for the text tasks (App. B.1 gives subset sizes but I could not find the thresholds)? And what should a group budget, in accelerator time, to evaluate one new configuration — is a smaller core subset planned?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer kieV (Part 1)

We thank Reviewer kieV for recognizing that RSIBench-Infra tests complete infrastructure workloads, cleanly separates OCI layer submissions from grading, and enforces task quality before counting speed. Below we address each point across four parts (manuscript updates are highlighted in blue).

W1 & Q1: Separating model, harness, and optimization-guidance effects

Thank you for suggesting this control experiment. Our leaderboard evaluates model–harness configurations, following Terminal-Bench [1], to measure an agent system's end-to-end delivery capability. Notably, four of the six entries in Table 2 already share the exact same OpenCode 1.18.18 harness—spanning Faster Rates from 6/20 (DeepSeek V4 Flash) to 11/20 (Qwen3.8-Max)—providing a controlled within-harness model comparison.

To isolate harness and guidance effects for the top entry as requested, we evaluated GPT-6 Astra on the five-task single-H20 Lite Set: A vs. B compares Codex and OpenCode with the skill enabled; A vs. C compares Codex with and without the skill.

SettingQwen3.8-27BMuse-Glimmer-30BQwen3-VL-32BLLaDA2.1-mini-16BWan2.2-TI2V-5BMedian speedup
A: Codex + skill1.5423 ± 0.21721.0766 ± 0.00581.1484 ± 0.07272.4411 ± 1.15151.0009 ± 0.00141.1263 ± 0.0602
B: OpenCode + skill0.0000 ± 0.00001.1585 ± 0.00551.0952 ± 0.03513.1614 ± 0.58571.0008 ± 0.00041.0952 ± 0.0351
C: Codex without skill1.8682 ± 0.11191.1299 ± 0.00331.2054 ± 0.03772.7550 ± 0.98901.0008 ± 0.00071.2054 ± 0.0377
  • Skill sensitivity: Removing the skill did not hurt performance: Codex reached 1.2054 ± 0.0377× without the skill (~7.0% above 1.1263 ± 0.0602× with it), showing Astra discovers substantive optimizations without our playbook (though weaker models or multi-node tasks may benefit differently).
  • Harness sensitivity: Although five-task medians were close (1.1263 ± 0.0602× vs. 1.0952 ± 0.0351×), the harness strongly affected task-level delivery: OpenCode consistently scored zero on Qwen3.8-27B due to quality-gate failure while Codex achieved 1.5423 ± 0.2172× (whereas OpenCode led on Muse-Glimmer and LLaDA2.1-mini). This confirms that the execution harness materially affects valid delivery and motivates reporting model–harness pairs explicitly.
AuthorsOpenReview ↗

Author Response to Reviewer kieV (Part 2)

W2 & Q2: Baseline headroom and the role of known optimizations

Thank you for raising this important question about baseline strength and available headroom. Our baselines are reproducible deployments derived from official engine cookbooks. We contextualize their optimization headroom through same-task analytical Speed-of-Light (SOL) scores and official framework maintainer reports:

  • Same-task analytical SOL headroom: Combining native execution traces with compute, memory, and communication models, our Oracle baselines on the H20 Lite Set achieve 35.2% (Qwen3-VL-32B, b/s=0.3520b/s = 0.3520) to 67.2% (Muse-Glimmer-30B, b/s=0.6720b/s = 0.6720) of their analytical SOL throughput (Appendix I, Table 12). Using the anchored SOL Score of SOL-ExecBench [2] across all 20 tasks (where the scoring baseline anchors at 0.500000, the SOL bound anchors at 1.000000, and failures score 0), the six configurations achieve SOL Scores of 0.373219–0.554755 (Table 2, Table 13), confirming substantial untapped headroom beyond low-hanging flags.
  • Maintainer-reported improvements in established engines: Screening official vLLM and SGLang engineering blogs yielded 13 articles with 44 same-framework, matched-GPU full-service throughput comparisons (retaining negative results; excluding cross-hardware, kernel-only, and headline-max claims). Taking within-article geometric means and weighting articles equally gives:
FrameworkArticlesGeometric mean speedup
vLLM [3–8]62.016×
SGLang [9–15]71.313×
Combined131.600×

Under our eight-hour budget, agents achieve suite-median speedups of 1.0003×–1.1602× (best median: 1.1602× by Opus 5; best Faster Rate: 12/20 by Astra at 1.1205×). Comparing overlapping model families with our best validated agent result per task (unchanged on our 8-entry live board) further contextualizes headroom:

Task / modelBest agent speedupBlog-reported improvementMain differences
Qwen3-TTS Base1.2203× (Opus 5)1.6151×, 26.55→42.88 audio-s/s [7]Both use vLLM-Omni voice cloning. Ours: 1×H20, c=1, reference-audio numerator; blog: 2×H20, c=64, generated-audio throughput.
Kimi K31.0963× (Astra)2.20×–2.81× [8]Ours: SGLang, 32×H20, 256k trace; blog: vLLM across versions on 8×B300, 8K/1K requests.
GLM-5.3-Flash6.7883× (Qwen3.8-Max)2.11×–4.28× [16]Ours: vLLM-Ascend on 16×NPUs; blog: SGLang 3P1D on 16×H200, 64K/1K under latency SLOs.

Moreover, our rubric-based trajectory audit (Appendix F.1, Table 8) shows that 63.2% (36/57) of Faster wins involve code or kernel implementations rather than flag-only tuning (31.6%, 18/57). Even in π₀.₅-LIBERO (Appendix D), removing the baseline padding fast path uncovered a recycled-address KV-cache bug on the 64-case pool, requiring monotonic context IDs, fused ada-RMSNorm Triton kernels, shape-aware split-KV, and bounded CUDA graphs to reach 3.79×.

W4: Overstated claims in the analyses and title

Thank you for pointing out where our claims needed tighter qualification. We have revised all three points in the manuscript (highlighted in blue):

  • RSI framing: Removed "Their Own" from the title and framed each trial in Section 1 as a bounded optimization episode for a component of RSI, leaving closed-loop reuse to future work.
  • Figure 3: Clarified in Section 1, Section 4.2, and Appendix F that categories are assigned by an LLM judge (GPT-6 Astra) using explicit operational rubrics (Appendix F.1, Table 9), can co-occur, and describe outcome associations rather than isolated causal effects.
  • Extended-budget cases: Reframed Section 4.2 and Appendix H as two exploratory cases, reporting actual execution times (9.62 h early exit; 20.27 h deployment failure) without claiming general time-scaling conclusions.
AuthorsOpenReview ↗

Author Response to Reviewer kieV (Part 3)

W3: Production relevance and latency metrics

Thank you for this thoughtful critique regarding serving concurrency, tail latency, and multimodal sample counts. We address each aspect below:

  • Low-concurrency scope: Single-node and batch-1/low-concurrency serving are established targets for private endpoints and sequential agent loops: Red Hat [17] documents single-node inference for low-traffic production services, and vLLM [18] explicitly optimizes batch-1 decode (8.95 ms → 3.30 ms/step on Reflection Beam). We agree that high-concurrency multi-tenant serving under strict SLOs is a distinct operating regime.
  • Archived E2E p99 tail-latency audit: Auditing archived verifier outputs yielded nine matched baseline-vs.-candidate E2E p99 comparisons across four tasks (all passing quality gates):
Task (protocol, requests, concurrency)ConfigurationBase p99 (s)Cand. p99 (s)p99 diffSpeedup
Qwen3-VL-32B (MM-v1, N=32N=32, c=8c=8)Codex + GPT-6 Astra19.70416.529-16.12%1.1904×
Qwen3-VL-32B (MM-v1, N=32N=32, c=8c=8)OpenCode + Qwen3.8-Max19.70416.707-15.21%1.1802×
Qwen3-VL-32B (MM-v1, N=32N=32, c=8c=8)OpenCode + GLM-5.319.70416.884-14.31%1.1678×
Qwen3-VL-32B (MM-v1, N=32N=32, c=8c=8)OpenCode + Kimi K319.70417.195-12.73%1.1465×
Qwen3-VL-32B (MM-v1, N=32N=32, c=8c=8)Claude Code + Opus 519.70419.639-0.33%1.0022×
Qwen3-VL-32B (MM-v1, N=32N=32, c=8c=8)OpenCode + DeepSeek V4 Flash19.70419.627-0.39%1.0007×
LLaDA2.1-mini-16B (trace, N=140N=140, c=4c=4)OpenCode + DeepSeek V4 Flash77.58772.698-6.30%1.4300×
DeepSeek-V4-Flash (trace, N=193N=193, c=4c=4)OpenCode + DeepSeek V4 Flash250.444185.671-25.86%1.0293×
Qwen3.8-27B (trace, N=193N=193, c=4c=4)OpenCode + DeepSeek V4 Flash232.194234.622+1.05%1.0003×

Every Faster (>1.10×>1.10\times) run simultaneously reduced E2E p99 latency by 6.30%–16.12% rather than trading tail latency for throughput (DeepSeek-V4-Flash also cut p99 by 25.86%, while Qwen3.8-27B shifted +1.05% at 1.0003×). Following your suggestion, future releases will report E2E/TTFT/TPOT p50/p95/p99 across all serving tasks and add SLO-constrained tasks.

  • Multimodal workload sizes: Timing workloads are decoupled from larger quality-gate pools: Qwen3-VL-32B times 32 concurrent 1080p requests (65.3k vision + ~69.3k text tokens) but gates accuracy on 847 MMMU examples; Qwen3-TTS gates on 202 utterances; π₀.₅ gates against a 64-case pool. Meanwhile, Wan2.2-TI2V-5B (four 1280×7041280\times704, 121-frame videos; ~36 min H20 runtime per pass, gated per video on LPIPS/temporal bounds) and MiniMax-H3-64B (672×384672\times384, 107-frame audio-video on 2×H20) execute deterministic fixed-step DiT/VAE graphs whose operator costs repeat identically across denoising steps.

Q3: Should Table 2 be interpreted as a stable ranking?

Thank you for highlighting this point. We agree that Table 2 should be read as a single-run snapshot rather than a statistically stable ranking, and we have stated this explicitly in Table 2's caption and the adjacent text in Section 4.2 (in blue; details in Appendix G, Table 10). Across three complete 20-task runs (60 trials each):

ConfigurationSingle-run Table 23-run mean ± SD (Faster / Median)Task-mean aggregateSOL Score
Codex + GPT-6 Astra12/20 / 1.1205×10.67 ± 1.15 (12, 10, 10) / 1.1097 ± 0.0118×11/20 / 1.1253×0.554755
OpenCode + Qwen3.8-Max11/20 / 1.1483×10.67 ± 0.58 (11, 10, 11) / 1.1694 ± 0.0820×10/20 / 1.0964×0.441088

Both configurations average 10.67 in run-level Faster counts; averaging per-task speedups before thresholding gives 11/20 vs. 10/20 due to indicator nonlinearity, while continuous SOL Scores (0.554755 vs. 0.441088) capture headroom-weighted gains without a hard 1.10× cutoff.

AuthorsOpenReview ↗

Author Response to Reviewer kieV (Part 4)

Q4: LongBench v2 thresholds, evaluation cost, and a smaller core subset

Thank you for these practical reproducibility questions. We detail the threshold calibration, Lite Set, and accelerator budgets below:

  • LongBench v2 thresholds: Evaluating every official Oracle baseline three times on LongBench v2 showed a maximum downward accuracy fluctuation of 4.5 percentage points. We therefore set a uniform 5pp one-sided absolute tolerance across all ten text tasks (acandidate≥areference−0.05a_{\mathrm{candidate}} \geq a_{\mathrm{reference}} - 0.05), matching the exact measured >= expected - 0.05 rule in vLLM's official accuracy regression tests [19, 20].
  • Lite subset: Added in Appendix I (Table 12), our five-task single-H20 Lite Set (Qwen3.8-27B, Muse-Glimmer-30B, Qwen3-VL-32B, LLaDA2.1-mini-16B, Wan2.2-TI2V-5B) spans text, VLM, diffusion-LLM, and video generation while reusing full-suite tasks, gates, and Oracle baselines.
  • Evaluation cost: Across the 120 main-leaderboard trials, empirical optimization-stage usage averaged 960.5 accelerator-hours per configuration on the 20-task suite (635.1 H20h + 325.4 NPUh; range 578.9–1,192.4; 8h/task cap: 1,480) and 26.65 H20-hours on the Lite Set (range 18.06–32.02; cap: 40). Including environment startup and isolated verification (with immediately available accelerators; queue wait without allocation costs zero), the end-to-end planning ceiling is 86.5 H20-hours for the Lite Set (40 opt + 10 Agent/Benchmark startup + 0.5 setup + 36 dual-H20 Serve/Benchmark verification after Agent release, plus transfer/cleanup margin) and ~3,518 accelerator-hours for the 20-task suite (2,600 H20h + 918 NPUh).

References

  • [1] Merrill et al., Terminal-Bench, arXiv:2601.11868, 2026 (url).
  • [2] Lin et al., SOL-ExecBench, arXiv:2603.19173, 2026 (url).
  • [3] vLLM, v0.6.0 Perf Update, 2024 (url).
  • [4] vLLM, Model Runner V2, 2026 (url).
  • [5] vLLM, vLLM x Mooncake, 2026 (url).
  • [6] vLLM, Artificial Analysis Leaderboard, 2026 (url).
  • [7] vLLM, TTS in vLLM-Omni, 2026 (url).
  • [8] vLLM, Kimi K3 Optimizations, 2026 (url).
  • [9] SGLang, SGLang v0.4, 2024 (url).
  • [10] SGLang, DFlash and Spec V2, 2026 (url).
  • [11] SGLang, Waterfill and LPLB, 2026 (url).
  • [12] SGLang, Agent-Assisted SGLang Dev, 2026 (url).
  • [13] SGLang, SGLang Omni SPG, 2026 (url).
  • [14] SGLang, HiCache Distributed Serving, 2026 (url).
  • [15] SGLang, Unified Memory, 2026 (url).
  • [16] Jiang et al., GLM-5.3-Flash on H200, LMSYS, 2026 (url).
  • [17] Red Hat, AI Inference 3.5: llm-d Patterns, Ch. 10 (url).
  • [18] Inferact, Reflection & NVIDIA, Reflection Beam on vLLM, 2026 (url).
  • [19] vLLM, test_gpqa_correctness.py#L170, 0ef2a95 (url).
  • [20] vLLM, test_accuracy.py#L47, 0ef2a95 (url).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their constructive feedback and for recognizing the value of evaluating end-to-end optimization across full serving and training stacks, our clean separation between OCI image-layer submissions and isolated verification, and our strict task-level quality gates. We have uploaded a revised manuscript (all updates highlighted in blue) incorporating the following new experiments, audits, and scope clarifications:

  1. Controlled harness and guidance ablations (kieV W1&Q1, s5d6 W2&Q1). On our five-task single-H20 Lite Set (Appendix I, Table 12), holding GPT-6 Astra fixed while swapping Codex for OpenCode exposes clear task-level harness effects (1.5423 ± 0.2172× vs. 0.0000× on Qwen3.8-27B; medians 1.1263 ± 0.0602× vs. 1.0952 ± 0.0351×), whereas removing the optimization skill yields 1.2054 ± 0.0377×, showing gains do not rely on prompt hints.
  2. Quantifying baseline headroom via SOL Score and maintainer reports (kieV W2&Q2, d2Tv W2a, s5d6 Q3). Following the SOL-ExecBench leaderboard convention, we added SOL Scores across all 120 trials (Table 2, Section 4.2, Appendix I Table 13; scoring baseline = 0.500000, SOL bound = 1.000000, top agent = 0.554755, with Lite Set baselines at 35.2%–67.2% of SOL in Table 12) alongside an audit of 13 official vLLM/SGLang engineering posts (44 matched-GPU throughput comparisons; geometric mean 1.600×).
  3. Process-level and implementation-depth audits (d2Tv W2b–c, Q1–Q2; s5d6 Q1, Q3). Added in Appendix F.1 (Tables 8–9), rubric-based LLM-judge (GPT-6 Astra) audits over all 120 trajectories and OCI layers show that 63.2% (36/57) of Faster wins come from code/kernel implementations (36/62 = 58.1% Faster) versus 31.6% (18/57) from flag tuning (18/35 = 51.4%), 70.8% (85/120) complete full profile–diagnose–edit–validate loops (51.8% vs. 37.1% Faster), and 64.2% (77/120) invoke accelerator profilers (torch.profiler, nsys, msprof).
  4. Tail-latency audit, Lite Set cost accounting, and new FP8 training task (kieV W3, Q4; d2Tv W1). Across nine recovered baseline-vs.-candidate E2E p99 comparisons on four tasks, every Faster run simultaneously lowered p99 latency by 6.3%–16.1%. We report empirical accelerator-hours (960.5 full-suite / 26.65 Lite optimization mean; 86.5 H20-hours total Lite planning ceiling) and added an 8×H20 MiMo-7B-RL FP8 SFT task (experimental runs are on the way and will be updated here and in the manuscript upon completion).
  5. Calibrated RSI, ranking, and scope claims (kieV W4, Q3; d2Tv W1; s5d6 W2, Q2). We removed "Their Own" from the title, framed experiments as evaluating an infrastructure component of RSI (Section 1), clarified next to Table 2 (Section 4.2, Appendix G) that single-run scores are not a statistically stable ranking, unified Figure 3's rubric-based LLM-judge framing (Section 4.2, Appendix F), and scoped the 24-hour cases (Section 4.2, Appendix H) and 8–16 GPU training bounds (Section 5).

For all reviewers: as the discussion period is approaching its end, please join the discussion and put acknowledge on authors' rebuttal, thank you!

AuthorsOpenReview ↗

Summary of Additional Experiments and Discussions During the Rebuttal Phase

We thank the reviewers for their feedback on RSIBench-Infra. We summarize the main results and clarifications below.

Additional experiments and analyses

  1. FP8 training and long-context RL (Reviewer d2Tv). On MiMo-7B FP8 SFT, Kimi K3 and GPT-6 Astra passed the quality checks at 1.318× and 1.298× baseline throughput. On Qwen3-4B 32K-context GRPO, all four completed runs passed verification. Kimi K3 achieved 1.281× by reallocating rollout resources to relieve KV-cache pressure.

  2. Harness and guidance ablations (Reviewers kieV and s5d6). With GPT-6 Astra and the same guide, Codex achieved 1.54× on Qwen3.8-27B, while OpenCode failed verification. Across the five-task Lite Set, Codex achieved a median 1.21× without the guide and 1.13× with it.

  3. Which changes produced speedups? (Reviewers d2Tv and s5d6). Among 57 verified runs exceeding 1.10× baseline throughput, 36 involved code or kernel changes and 18 used configuration tuning alone. On distillation, training-side changes yielded 1.01×, while moving four idle GPUs to the bottleneck stage, teacher scoring, yielded 1.26×.

  4. Tail latency and baseline headroom (Reviewers kieV and d2Tv). Across nine matched comparisons on four serving tasks, every run exceeding 1.10× throughput also reduced p99 latency, by 6.3%–16.1%. Lite Set baselines reached 35.2%–67.2% of their estimated hardware speed-of-light bounds.

Further discussion and clarifications

  • Leaderboard interpretation and reproducibility (Reviewers kieV and s5d6). Across three full-suite runs, Codex + Astra and OpenCode + Qwen3.8-Max each averaged 10.67 of 20 tasks above 1.10×, illustrating the uncertainty in small single-run ranking differences. The five-task Lite Set retains the original quality checks and averaged 26.65 H20-hours per configuration for optimization.

  • Why end-to-end checks matter (Reviewers d2Tv and s5d6). On LingBot, an FP8 attention change increased throughput by 13.7% and passed single-step checks. Over 16 autoregressive chunks, mean absolute pixel error grew from 1.96 to 53.27, exceeding the tolerance of 2.0.

  • Scope of the RSI claim (Reviewers kieV and s5d6). We revised the framing to describe what each trial measures: one agent delivers a deployable infrastructure change under fixed workload and correctness constraints. Evaluating successive rounds of self-improvement remains future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.