acceptodds
Under review as a conference paper at ICLR 2027

AscendInfraBench: Benchmarking Coding Agents for Full-Lifecycle LLM Infrastructure Engineering on Ascend NPUs

Abstract

LLM infrastructure must coordinate model computation, communication, and state management. We introduce a capability-oriented task construction method that specifies required work, permitted changes, and performance objectives while leaving implementation choices open. We use this method to construct AscendInfraBench, with 24 tasks spanning Optimization, Model Adaptation, and Open Build across training, reinforcement learning, and serving infrastructure on Ascend NPUs. We independently rebuild each submission, check numerical correctness and state handling, measure performance, and review the source against task requirements. State checks verify that training resumes from checkpoints, rollout uses updated policies, and serving preserves request-cache isolation. We evaluate 9 coding agents across 216 development runs, each with a 2-hour budget. GPT-6 Astra delivers valid implementations on all 24 tasks, including all 6 Model Adaptation and Open Build tasks, achieving a 1.63× task-weighted geometric-mean speedup over task baselines. Across agents, valid implementations meet the same task requirements through different choices in execution, model integration, and system organization. AscendInfraBench assesses agents’ ability to optimize existing code, integrate model support, and build complete LLM systems.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Coding Agents, LLM Infrastructure, LLM For System, Empirical Study

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6BXdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces AscendInfraBench, a benchmark of 24 LLM-infrastructure tasks on Ascend 910B NPUs spanning training, RL, and serving. Tasks span implementation optimization, joint optimization, model adaptation, and building complete systems. Evaluation independently rebuilds submissions, checks numerical correctness and state continuity, measures performance, and reviews source code. Across 216 two-hour runs by 9 coding agents, GPT-6 Astra solves all 24 tasks with a 1.63× geometric-mean speedup, and scope-level results reveal capability differences that optimization-only scores miss.

Critical strengths

The stage-by-scope design exposes capabilities that optimization-only evaluation would miss. Tasks cover training, RL, and serving, while distinguishing implementation optimization, joint optimization, model adaptation, and complete-system construction. As Table 1 shows, no prior benchmark covers all nine cells

Critical weaknesses
  1. The headline comparisons rest on fragile statistics and baselines of uncertain strength. (1) Each agent–task pair receives only one development run. This materially limits confidence in the comparative findings, especially because Model Adaptation and Open Build contain only three tasks each. (2) Table 14 shows the R03 baseline spends 1012 of 1298 s in initialization, so much of that gain is removing framework startup overhead, not improving the RL loop itself.
  2. Source review is assisted by the top-ranked evaluated model. Appendix D.6 states that GPT-6 Astra assists the source review that decides validity, and Astra is also the agent with a perfect 24/24 record. Authors make final decisions, but the paper reports no count of cases where humans overrode the model, and no indication that Astra’s own submissions were reviewed by a different model.
Questions for authors
  1. Were any independent development reruns conducted beyond the 216 reported runs? If so, how stable are validity counts, performance coverage, and agent ordering? If none exist, a targeted repeated-run study would be an additional experiment needed to resolve this uncertainty
  2. How were the performance-baseline configurations selected and tuned? What's the headroom for speedup?
9 comments
AuthorsOpenReview ↗

Response to Reviewer 6BXd (1/3): Re-W1

Re-W1

The headline comparisons rest on fragile statistics and baselines of uncertain strength. (1) Each agent–task pair receives only one development run. This materially limits confidence in the comparative findings, especially because Model Adaptation and Open Build contain only three tasks each. (2) Table 14 shows the R03 baseline spends 1012 of 1298 s in initialization, so much of that gain is removing framework startup overhead, not improving the RL loop itself.

Thank you for raising both points. Regarding (1), we added 30 independent 2-hour reruns across all 6 Model Adaptation and Open Build tasks (shown in Table R1 and analyzed in Re-Q1), confirming 100% delivery stability on Pre-Training and Serving.

Regarding (2), our task objective is end-to-end job throughput, allowing agents to freely identify and optimize any bottleneck across the entire execution window. The fact that most agents identified and eliminated startup overhead in the recommended verl 2-NPU PPO setup reflects adherence to the end-to-end task specification—and critically, agents also substantially accelerate the core online RL training loop itself. As shown in Table 14b, excluding initialization and finalization, all 4 valid submissions achieve 2.41–14.42× speedups inside the online RL loop alone. Specifically, all 4 valid implementations eliminate weight-transfer bottlenecks by generating rollouts directly from the training actor rather than a separate rollout engine, and GPT-6 Astra (14.42× inside-loop, 16.43× end-to-end) further keeps model and optimizer states resident on NPUs, uses graph decoding, and overlaps actor and critic updates.

Table R1: Repeated-run speedups across all 6 Model Adaptation and Open Build tasks (30 new 2-hour runs).

TaskGPT-6 Astra: original; 2 rerunsGPT-6.1 Sol: 3 runs
T05 · Pre-Training Adaptation1.03×; 1.00×, 1.02×1.00×, 1.03×, 1.00×
R04 · RL Adaptation1.33×; 1.03×, Invalid1.20×, Invalid, Invalid
S12 · Serving Adaptation1.44×; 1.44×, 1.59×1.53×, 1.55×, 1.60×
T06 · Pre-Training Open Build4.82×; 4.07×, 5.09×4.87×, 4.62×, 4.68×
R05 · RL Open Build2.54×; 1.88×, 1.53×1.40×, 2.45×, Invalid
S13 · Serving Open Build1.10×; 0.99×, 1.08×1.06×, 1.07×, 0.85×
AuthorsOpenReview ↗

Response to Reviewer 6BXd (2/3): Re-W2

Re-W2

Source review is assisted by the top-ranked evaluated model. Appendix D.6 states that GPT-6 Astra assists the source review that decides validity, and Astra is also the agent with a perfect 24/2424/24 record. Authors make final decisions, but the paper reports no count of cases where humans overrode the model, and no indication that Astra's own submissions were reviewed by a different model.

Thank you for highlighting this concern. Following LLM-assisted code auditing practices in recent benchmarks (such as InferenceBench [1]), we strip all evaluated model and agent-harness identifiers to anonymize submissions prior to review (while keeping the underlying target frameworks visible for contract verification), manually audit every reject and needs_evidence verdict, and independently cross-audited all 24 of GPT-6 Astra's submissions using both human authors and a separate model.

Specifically, out of 216 total runs, 180 submissions underwent source review (the remaining 36 submissions comprise 23 Correctness failures, 8 Measurement failures, and 5 unchanged Optimization baselines). Authors manually inspected all 9 initial reject cases and all 3 needs_evidence cases (plus ~50% of accept cases at random, finding zero discrepancies). Human review overturned 2 reject verdicts to acceptances (GLM-5.3 on R03 and T01) and resolved all 3 needs_evidence cases as acceptances (Kimi K3 on S13, Claude Fable 5.1 on R03, and GPT-6 Astra on R03), as summarized in Table R2. Furthermore, all 24 anonymized submissions from GPT-6 Astra were independently audited from scratch by GPT-6.1 Sol (High) with Codex and independently verified by the human authors, with both accepting all 24 submissions (Table R3). We will add Tables R2–R3 to Appendix D.6.

Table R2: Source-review accounting and human override breakdown across all 180 reviewed submissions.

Review Stage / OutcomeCountAudit Scope & Human Override Details
Total benchmark runs2169 agents × 24 tasks
Submissions entering source review180Evaluated model and agent-harness identifiers stripped prior to review
Initial accept (review agent)168~50% randomly re-audited by authors; 0 overturned
Initial reject (review agent)9100% manually audited by authors; 2 overturned to accept (GLM-5.3 on R03 and T01)
Initial needs_evidence (review agent)3100% manually audited by authors; all 3 resolved as accept (Kimi K3 on S13; Claude Fable 5.1 & GPT-6 Astra on R03)
Final source-review verdict173 accept / 7 reject2 rejections overturned; 3 needs_evidence cases resolved as accept

Table R3: Independent cross-review of all 24 anonymized submissions from GPT-6 Astra.

Independent Cross-Review of GPT-6 Astra's 24 SubmissionsOutcome
GPT-6.1 Sol (High) with Codex (independent model reviewer)24 / 24 accepted
Human Authors (independent manual audit)24 / 24 accepted

References for this reviewer response are listed in comment 3/3.

AuthorsOpenReview ↗

Response to Reviewer 6BXd (3/3): Re-Q1 + Re-Q2

Re-Q1

Were any independent development reruns conducted beyond the 216 reported runs? If so, how stable are validity counts, performance coverage, and agent ordering? If none exist, a targeted repeated-run study would be an additional experiment needed to resolve this uncertainty

Yes. As shown in Table R1 (in Re-W1), we conducted 30 additional 2-hour runs across all 6 Model Adaptation and Open Build tasks (2 reruns for GPT-6 Astra to reach 3 attempts per task, and 3 runs for GPT-6.1 Sol under Codex). Across these reruns, validity counts, performance coverage, and task-level patterns remain consistent, reinforcing our cross-trajectory findings:

  • Validity counts: GPT-6 Astra achieves 17/18 (94.4%) valid runs and GPT-6.1 Sol achieves 15/18 (83.3%), with both reaching 100% (3/3) validity on every Pre-Training and Serving task and all 4 failures occurring in RL (R04/R05).
  • Performance coverage: Using exact unrounded speedups, runs reaching ≥1.10× are 10/18 for GPT-6 Astra and 9/18 for GPT-6.1 Sol (note that GPT-6 Astra's original S13 ratio is 1.0999×, which rounds to 1.10× in display but is below the ≥1.10× threshold), with tight speedup bands on T06 (4.07–5.09×) and S12 (1.44–1.60×).
  • Relative task comparisons: GPT-6 Astra matches or exceeds GPT-6.1 Sol in validity across all 6 tasks, while each agent leads in speedup on different tasks (e.g., GPT-6 Astra leads on T06 and R05, whereas GPT-6.1 Sol achieves a higher median on S12 at 1.55× vs. 1.44×).

Re-Q2

How were the performance-baseline configurations selected and tuned? What's the headroom for speedup?

Optimization and Open Build baselines use each framework's official default or recommended configuration for the target model, workload, and NPU allocation (representing standard out-of-the-box deployment), while Model Adaptation baselines use minimal working author adaptations.

To measure pure configuration-tuning headroom, we ran a 2-hour random configuration search (sampling predefined configuration spaces without replacement under fixed seeds, counting startup and compilation toward the budget) on all 5 Joint Optimization tasks. As shown in Table R4, the best agent result outperforms random configuration search on all 5 tasks (following the configuration-search comparison approach in InferenceBench [1]), showing that agents can both select better configurations and improve performance through implementation changes. Across the broader 24-task suite, at least one valid submission achieves ≥1.03× speedup on every task, with strong individual headroom (such as 1.50× on S02, 1.26× on T01, and 1.10× on S13).

Table R4: Two-hour random configuration search vs. best agent results across all 5 Joint Optimization tasks.

TaskConfigurations triedBest sampled configuration (excluding baseline)Best reported agent resultMain agent changes
T04 · MindSpeed training101.04×1.27× · GPT-5.6 SolConfig: interleaved pipeline stages. Implementation: reduced startup and shutdown overhead.
R02 · verl GRPO80.75×5.99× · GPT-6 AstraConfig: separate actor/rollout devices. Implementation: overlapped rollout and updates, packed training, direct weight transfer.
R03 · verl PPO81.03×16.43× · GPT-6 AstraImplementation: resident PPO runtime, shared actor/rollout, graph decoding, concurrent actor/critic updates.
S10 · SGLang serving92.63×2.64× · GPT-6 AstraConfig: decode graphs and graph batch sizes. Implementation: paged-attention graph capture and synchronized input updates.
S11 · vLLM-Ascend serving81.00×1.11× · Claude Fable 5.1Config: larger KV cache and prefill batches, expert-weight prefetch, CPU affinity, task queue settings.

References:

[1] Yeon et al., InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents, arXiv:2607.20468, 2026.

Reviewer_dG94OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AscendInfraBench, a benchmark for evaluating coding agents on LLM infrastructure engineering tasks on Ascend NPUs. It contains 24 tasks spanning training, reinforcement learning, and serving, organized into four engineering scopes: Implementation Optimization, Joint Optimization, Model Adaptation, and Open Build. The proposed task construction method specifies required computation, state transitions, and performance objectives while allowing flexibility in implementation. Evaluation combines independent reconstruction, numerical and state correctness checks, paired performance measurements, and source review. The authors evaluate nine coding agents under a two-hour budget per task and report differences across engineering scopes. The strongest evaluated agent produces valid implementations for all 24 tasks and achieves a task-weighted geometric-mean speedup of 1.63× over the baselines.

Critical strengths
  1. The benchmark addresses a meaningful engineering setting and may be benificial for long-horizon coding tasks.

  2. This work conducted comprehensize expriments on LLM infrasturcture engineering tasks including various coding agents, tasks.

  3. They propose three phases to make sure the experiments smooth, including Correctness, Measurement, and Review.

Critical weaknesses
  1. **The representativeness and difficulty of the task collection require further justification. ** The benchmark contains 24 tasks across 12 stage–scope combinations, with only six tasks covering Model Adaptation and Open Build together. Conclusions about these broader capabilities may therefore depend on a small number of tasks. The task selection rationale and coverage of different workloads should be clear. Since the strongest agent produces valid implementations for all 24 tasks, it is also useful to distinguish the benchmark’s remaining difficulty in correctness from its difficulty in performance optimization.

  2. The robustness of the agent comparisons needs clarification. The reported 216 development runs correspond to nine agents evaluated on 24 tasks, which appears to imply one development run per agent–task pair. Paired performance measurements can address timing variability, but they do not establish robustness to variation in the agents’ development trajectories. It is better to show more trajectory behavior analysis to give the insight on the comparisons from different agents and tasks.

  3. The interpretation of the reported task-weighted geometric mean depends on the choice of baselines, task weights, and treatment of unsuccessful submissions. These choices become particularly important when agents complete different subsets of tasks. The evaluation should make clear whether aggregate performance remains comparable across agents with different validity rates.

Questions for authors
  1. How were the 24 tasks selected, and what makes them representative of the capabilities associated with each stage–scope combination?

  2. Do the 216 development runs represent a single attempt per agent–task pair? If so, what evidence supports the stability of the reported comparisons?

  3. What is the exact formula for the task-weighted geometric-mean speedup, including the treatment of invalid or unsuccessful runs? How are task baselines and weights chosen?

6 comments
AuthorsOpenReview ↗

Response to Reviewer dG94 (1/4): Re-W1

Re-W1

The representativeness and difficulty of the task collection require further justification. The benchmark contains 24 tasks across 12 stage–scope combinations, with only six tasks covering Model Adaptation and Open Build together. Conclusions about these broader capabilities may therefore depend on a small number of tasks. The task selection rationale and coverage of different workloads should be clear. Since the strongest agent produces valid implementations for all 24 tasks, it is also useful to distinguish the benchmark's remaining difficulty in correctness from its difficulty in performance optimization.

Thank you for this thoughtful point. Because AscendInfraBench evaluates end-to-end systems engineering, each task opens a broad design space where agents can freely explore system architectures, execution plans, and kernels. Consequently, a single task can deeply probe real infrastructure capability—while also incurring substantial evaluation cost. Across the 24 tasks, we systematically cover all 12 stage–scope combinations and representative production workloads, including DP, TP, and packed-sequence training; GRPO and PPO reinforcement learning; and offline, saturation, mixed-interactivity, repeated-context, and SLO-constrained serving.

More importantly, consistent with prior performance benchmarks such as KernelBench [1], InferenceBench [2], and Φ-Bench [3], passing correctness checks is a necessary baseline requirement, whereas performance optimization (captured by our Performance Coverage curve in Figure 4) is the primary differentiator. As detailed in Table R1, while strong agents already achieve high validity, substantial optimization headroom remains and difficulty manifests differently across agents: even though GPT-6 Astra solves 24/24 tasks validly, its performance coverage at ≥1.10× is 12/24—trailing both GPT-5.6 Sol and Claude Fable 5.1 (13/24 each, Figure 4). Moreover, across all 5 Pre-Training Optimization and Adaptation tasks (T01–T05), GPT-6 Astra achieves 1.03–1.09×, whereas GLM-5.3 (1.26× on T01), DeepSeek-V4-Pro (1.36× on T03), and Claude Opus 5 (1.34× on T05) deliver significantly larger speedups.

Table R1: Validity and performance coverage (≥1.10×) by scope and stage across the 216 benchmark runs.

ScopeStageTasksValid submissionsPerformance coverage ≥1.10×
OptimizationPre-TrainingT01–T0435/3617/36
OptimizationRL TrainingR01–R0319/2718/27
OptimizationServingS01–S1189/9927/99
Model AdaptationPre-TrainingT058/96/9
Model AdaptationRL TrainingR043/92/9
Model AdaptationServingS129/99/9
Open BuildPre-TrainingT065/95/9
Open BuildRL TrainingR055/94/9
Open BuildServingS135/90/9

References for this reviewer response are listed in comment 4/4.

AuthorsOpenReview ↗

Response to Reviewer dG94 (2/4): Re-W2

Re-W2

The robustness of the agent comparisons needs clarification. The reported 216 development runs correspond to nine agents evaluated on 24 tasks, which appears to imply one development run per agent–task pair. Paired performance measurements can address timing variability, but they do not establish robustness to variation in the agents' development trajectories. It is better to show more trajectory behavior analysis to give the insight on the comparisons from different agents and tasks.

Thank you for the suggestion. In the current submission, Section 4.5 and Appendix G.1–G.3 (Table 15) already examine several development trajectories in detail—covering operator reuse, input-size crossover thresholds, and checkpoint-restart debugging—and we also release all 216 de-identified trajectories and final artifacts in our anonymous repository and project website (Appendix A). Alongside our 30 new repeated runs (shown in Table R2), we will expand Section 4.5 with three additional comparative trajectory insights:

  1. Configuration tuning vs. kernel/graph engineering: On S10, 5 of the 7 valid submissions only tune deployment flags, whereas GPT-6 Astra and GPT-5.6 Sol also modify attention and graph capture, enabling Astra's top 2.64× speedup.
  2. Error-driven platform adaptation: On R05, Claude Fable 5.1's smoke test failed because HCCL rejected float64 all-reduce. The agent parsed the error trace, switched communication to FP32, and passed all 17 checks.
  3. Empirical profiling overturning intuitive designs: On T06, GPT-5.6 Sol initially implemented checkpoint compression (cutting data volume by ~23%), measured that CPU compression overhead exceeded I/O savings, reverted it, benchmarked host memory transfers, and adopted reusable pinned staging buffers.

Table R2: Repeated-run speedups across all 6 Model Adaptation and Open Build tasks (30 new 2-hour runs).

TaskGPT-6 Astra: original; 2 rerunsGPT-6.1 Sol: 3 runs
T05 · Pre-Training Adaptation1.03×; 1.00×, 1.02×1.00×, 1.03×, 1.00×
R04 · RL Adaptation1.33×; 1.03×, Invalid1.20×, Invalid, Invalid
S12 · Serving Adaptation1.44×; 1.44×, 1.59×1.53×, 1.55×, 1.60×
T06 · Pre-Training Open Build4.82×; 4.07×, 5.09×4.87×, 4.62×, 4.68×
R05 · RL Open Build2.54×; 1.88×, 1.53×1.40×, 2.45×, Invalid
S13 · Serving Open Build1.10×; 0.99×, 1.08×1.06×, 1.07×, 0.85×
AuthorsOpenReview ↗

Response to Reviewer dG94 (3/4): Re-W3 + Re-Q1

Re-W3

The interpretation of the reported task-weighted geometric mean depends on the choice of baselines, task weights, and treatment of unsuccessful submissions. These choices become particularly important when agents complete different subsets of tasks. The evaluation should make clear whether aggregate performance remains comparable across agents with different validity rates.

Thank you for raising this important point. Following KernelBench's fast_p metric [1], Figure 4 evaluates Performance Coverage over the exact same 24 tasks for every agent, counting failed or invalid submissions as zero. Thus, Performance Coverage compares all agents on a fixed 24-task denominator, while Task GM reports the average speedup on each agent's valid implementations. As an additional check, we recomputed equal-weighted GM on the 9 tasks where all 9 agents produced valid submissions (see Table R3): GPT-6 Astra (1.24×) and GPT-5.6 Sol (1.18×) remain first and second on this shared subset, while some middle rankings shift (e.g., Kimi K3 moves from 5th to 3rd at 1.17×). We will include Table R3 and clarify the distinct roles of both metrics in Section 4.1.

Table R3: Aggregate performance comparison across valid subsets and the common 9-task subset solved by all 9 agents.

AgentValid tasks / 24Performance coverage at 1.10× / 24GM over each agent's valid tasksGM over the same 9 tasks
GPT-6 Astra24121.63×1.24×
GPT-5.6 Sol23131.39×1.18×
Claude Fable 5.122131.37×1.12×
GLM-5.31991.34×1.12×
Kimi K320111.31×1.17×
Claude Opus 521111.22×1.10×
DeepSeek-V4-Pro20111.21×1.11×
Gemini 3.7 Flash1441.13×1.10×
MiniMax M31541.12×1.10×

Re-Q1

How were the 24 tasks selected, and what makes them representative of the capabilities associated with each stage–scope combination?

As noted in Re-W1, we selected the 24 tasks from real engineering workflows across MindSpeed-LLM, verl, SGLang, and vLLM-Ascend to reflect the three routine infrastructure jobs—optimizing existing pipelines, integrating new model architectures, and building complete systems—with most tasks using 1 or 2 NPUs (and T04 using 4 NPUs) to keep evaluation reproducible and cost-tractable. To isolate the target capability in each scope, we paired models intentionally: Optimization and Open Build tasks use models with mature NPU support (such as Qwen3 dense and MoE models) so evaluation focuses on execution efficiency and system design, whereas Model Adaptation uses LFM2.5—whose hybrid attention–convolution architecture lacked native framework support—to directly test weight mapping, custom operator integration, and hybrid state management across Pre-Training (T05), RL (R04), and Serving (S12).

References for this reviewer response are listed in comment 4/4.

Reviewer_Eyc3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present AscendInfraBench, a benchmark of 24 tasks spanning training, reinforcement learning, and inference serving on Ascend NPUs. The authors evaluate nine models on their ability to optimize, adapt, and build LLM infrastructure systems.

Critical strengths
  • The authors provide a wide variety of tasks revolving around inference infrastructure.
Critical weaknesses
  • Currently there seems to be a limited novelty as to what new capability of agents this benchmark covers compared to existing works. Although the authors present a wide variety of tasks, it is unclear whether this meaningfully measures a new capability not captured by existing benchmarks.
  • In addition, it is not sufficiently explored what the results mean beyond performance metrics, making it difficult to understand what readers of the benchmark should take away from it.
Questions for authors
  • Many of the optimization strategies discussed in the paper seem to also apply for other hardware specifications. Does the Ascend NPU introduce a meaningfully different failure mode compared to Nvidia-based infrastructure?
  • What new failure modes or capabilities not captured by existing works is captured by this benchmark?
  • There is a wide variety of tasks, but could authors elaborate more about the task design - how were the tasks derived and how independent are the capabilities the different tasks are measuring?
  • The paper evaluates each agent on each task once. While I do realize that these agent runs can be quite expensive, previous works have stated that the stochastic nature of agents leads to non-trivial variability. How confident are authors in the robustness of results presented?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Eyc3 (1/3): Re-W1 + Re-W2

Re-W1

Currently there seems to be a limited novelty as to what new capability of agents this benchmark covers compared to existing works. Although the authors present a wide variety of tasks, it is unclear whether this meaningfully measures a new capability not captured by existing benchmarks.

Thank you for raising this question. While existing benchmarks evaluate repository patches, accelerator kernels, or inference serving deployment and bespoke serving systems (such as InferenceBench [1] and VibeServe [2]), AscendInfraBench is the first benchmark to evaluate coding agents across Pre-Training, RL Training, and Serving spanning Optimization, Model Adaptation, and Open Build on Ascend NPUs (Section 2, Table 1).

Rather than assembling synthetic tasks just for variety, we distilled all 24 tasks from real engineering bottlenecks encountered in MindSpeed-LLM, verl, SGLang, and vLLM-Ascend. Expanding evaluation to full-lifecycle infrastructure reveals capability gaps that narrower evaluations miss:

On one hand, agents that look similar on localized optimization tasks can differ sharply when building a system from scratch: GPT-5.6 Sol and DeepSeek-V4-Pro both deliver valid solutions on 17/18 Optimization tasks, yet in Open Build, GPT-5.6 Sol completes all 3 tasks while DeepSeek-V4-Pro completes only 1.

On the other hand, our newly introduced RL infrastructure tasks and Pre-Training Adaptation tasks expose widespread difficulties in cross-component state coordination and policy synchronization, where only 3/9 agents succeed on R04. Pinpointing these failure modes across stages and scopes also provides concrete diagnostic feedback for curating training data, designing agentic RL rewards, and supporting recursive self-improvement (RSI) on real systems.


Re-W2

In addition, it is not sufficiently explored what the results mean beyond performance metrics, making it difficult to understand what readers of the benchmark should take away from it.

Thank you for pointing this out. We agree that the broader takeaways of our findings deserve more prominence, so we will add a dedicated subsection in Section 4.5 and expand the Conclusion around three practical insights:

  1. Breaking rigid framework abstractions unlocks massive speedups, pointing to clear bottlenecks for framework developers. Default out-of-the-box framework configurations suffer from heavy process initialization overhead (e.g., 1,012s out of 1,298s in R03 verl PPO) and costly cross-device weight transfers between training and rollout engines. The largest speedups—4.82× on Pre-Training Open Build (T06) and 16.43× on PPO (R03)—come from replacing default abstractions with resident actor/rollout sharing, overlapped execution, and sharded optimizer states.
  2. Practitioners can tune infrastructure once with agents to save NPU hours, and should pick models based on the task scope. Modern coding agents are already mature enough to serve as a one-time upfront optimizer: tuning a framework once for a target model and hardware setup before recurring production runs can save substantial NPU hours. Alongside broader engineering benchmarks (such as Artificial Analysis [3], FrontierSWE [4], and Terminal-Bench [5]), our stage-by-scope breakdown helps match models to specific engineering needs: GPT-6 Astra delivers the strongest overall reliability (24/24 valid), DeepSeek-V4-Pro (1.36× on T03) and GLM-5.3 (1.26× on T01) achieve higher peak speedups on Pre-Training optimization, and GPT-6 Astra or GPT-5.6 Sol are much stronger when building complete systems from scratch.
  3. Training next-generation infrastructure agents should prioritize multi-component RL systems and execution-guided feedback. Across our evaluation, RL Training exhibits the lowest overall validity rate (27/45, and only 3/9 on R04 Model Adaptation). Improving future coding agents therefore requires expanding training tasks and environments around tightly coupled multi-component RL systems to target the failure modes detailed in Re-Q1 and Re-Q2—such as hardware-stack constraints (e.g., HCCL datatype restrictions and NPU graph synchronization), packed-sequence boundary alignment, and cross-process state continuity (e.g., actor-to-rollout weight synchronization and checkpoint recovery). Incorporating execution-guided verification and profiling feedback into training rewards—teaching agents to revert measured regressions (as in S07) and select faster system architectures (as in T06 and R05)—offers a direct path toward stronger systems engineering agents.

References for this reviewer response are listed in comment 3/3.

AuthorsOpenReview ↗

Response to Reviewer Eyc3 (2/3): Re-Q1 + Re-Q2

Re-Q1

Many of the optimization strategies discussed in the paper seem to also apply for other hardware specifications. Does the Ascend NPU introduce a meaningfully different failure mode compared to Nvidia-based infrastructure?

Yes. While high-level optimization strategies transfer across accelerators, the Ascend software stack (CANN, HCCL, and NPU graph replay) enforces distinct operator, datatype, and stream-synchronization constraints. Agents that apply NVIDIA-centric assumptions without checking runtime behavior hit Ascend-specific failures.

In R05, Claude Fable 5.1 wrote a float64 all-reduce—which works under NVIDIA NCCL—only for the deployed PyTorch/HCCL backend to reject it at runtime; after reading the HCCL error trace, the agent switched communication to FP32 and delivered a valid RL system.

In S09, both GPT-5.6 Sol and GPT-6 Astra replaced a device-to-host sequence-length read with CPU metadata, causing serving to stall because on the tested Ascend execution path, that read also acted as an implicit synchronization barrier waiting for the previous graph replay to finish before updating shared inputs. In S02, GPT-6 Astra's full implementation preserved this necessary synchronization and reduced redundant graph updates, achieving a 1.50× end-to-end speedup.


Re-Q2

What new failure modes or capabilities not captured by existing works is captured by this benchmark?

Beyond the Ascend-specific backend constraints discussed in Re-Q1, AscendInfraBench tests cross-process and cross-stage state continuity. Specifically, it is our newly introduced Pre-Training Model Adaptation and RL Training tasks (spanning Optimization, Model Adaptation, and Open Build)—which existing benchmarks do not cover—that expose system-level failures where local computation passes, yet end-to-end system state breaks down.

As summarized in Section 4.3 and Appendix H.2, existing single-kernel or single-turn serving benchmarks do not exercise multi-component lifecycle transitions. Across our 23 Correctness failures (largely arising in RL, Model Adaptation, Open Build, and Joint Optimization tasks), the primary failure modes include:

  • State-handoff and checkpoint-recovery failures (8/23 cases): Agents pass steady-state steps, yet fail PPO checkpoint-restart checks (Kimi K3 and MiniMax M3 on R03), fail to propagate updated actor weights to the RL rollout engine (R04/R05), or mismatch short-convolution parameter keys during weight loading (Gemini 3.7 Flash on R04);
  • Numerical consistency failures (5/23 cases): Including packed-document boundary log-probability and gradient mismatches in RL (3 agents on R04), as well as loss/norm and generated-token scoring mismatches (MiniMax M3 on T05 and S10);

Separately, source review rejected 7 submissions by catching semantic contract violations that pass initial runtime probes, such as passing BF16 parameters directly into AdamW instead of maintaining FP32 master and moment states (Gemini 3.7 Flash on R05). We will highlight this failure taxonomy in Section 4.3 to contrast these system-level failures with single-operator evaluations.

AuthorsOpenReview ↗

Response to Reviewer Eyc3 (3/3): Re-Q3 + Re-Q4

Re-Q3

There is a wide variety of tasks, but could authors elaborate more about the task design - how were the tasks derived and how independent are the capabilities the different tasks are measuring?

We derived the 24 tasks from three routine infrastructure jobs—optimizing an existing framework, integrating a new model architecture, and building a system from scratch—spanning 4 engineering scopes × 3 infrastructure stages (12 combinations) across MindSpeed-LLM, verl, SGLang, and vLLM-Ascend with diverse production workloads. Most tasks use 1 or 2 NPUs (with T04 using 4 NPUs) to keep evaluation reproducible and cost-tractable.

Each scope defines distinct engineering requirements (improving existing code, adding model support, or constructing a system from an empty tree). Across stages, Training and Serving are relatively decoupled, whereas our newly introduced RL tasks tightly couple Training (policy updates), Serving (rollout generation), and cross-engine weight synchronization, making RL the hardest stage. Holding the model or workload constant across tasks illustrates how outcomes vary across these requirements. On the serving side, S01–S06 isolate 6 individual workloads whereas S09 requires a single implementation to balance all 6 at once—for example, Kimi K3 achieves 2.31× on S09's offline workload, while its ratios on the other 5 regimes stay near 1.00× (yielding 1.15× overall while passing all performance requirements), showing that gains vary substantially by request regime. Across stages, adapting the same hybrid attention–convolution model LFM2.5 becomes progressively harder as state coupling increases: 9/9 agents succeed in Serving (S12: prefill/decode cache state), 8/9 in Pre-Training (T05: backward pass and checkpoint recovery), and only 3/9 in RL (R04: packed boundaries plus actor-to-rollout policy refresh).


Re-Q4

The paper evaluates each agent on each task once. While I do realize that these agent runs can be quite expensive, previous works have stated that the stochastic nature of agents leads to non-trivial variability. How confident are authors in the robustness of results presented?

To directly evaluate run-to-run variability, we added 30 independent 2-hour runs across all 6 Model Adaptation and Open Build tasks (see Table R1): 2 reruns per task for GPT-6 Astra (giving 3 total attempts alongside its original run) and 3 new runs per task for GPT-6.1 Sol, both using Codex.

The repeated runs show that Pre-Training and Serving results are highly reproducible, whereas run-to-run variance is concentrated in RL. Both agents achieve 100% validity (3/3) on every Pre-Training and Serving task across all attempts, with tightly clustered speedups—all 6 implementations on T06 reach 4.07–5.09×, and all 6 on S12 reach 1.44–1.60×. By contrast, RL systems coordinate multiple components and policy synchronization, resulting in higher variance, with all 4 invalid attempts occurring in R04 and R05. In production engineering, teams expect agents to deliver working systems reliably in a single shot, making cross-run delivery consistency an essential dimension of agent capability (17/18 valid overall for GPT-6 Astra vs. 15/18 for GPT-6.1 Sol).

Table R1: Repeated-run speedups across all 6 Model Adaptation and Open Build tasks (30 new 2-hour runs).

TaskGPT-6 Astra: original; 2 rerunsGPT-6.1 Sol: 3 runs
T05 · Pre-Training Adaptation1.03×; 1.00×, 1.02×1.00×, 1.03×, 1.00×
R04 · RL Adaptation1.33×; 1.03×, Invalid1.20×, Invalid, Invalid
S12 · Serving Adaptation1.44×; 1.44×, 1.59×1.53×, 1.55×, 1.60×
T06 · Pre-Training Open Build4.82×; 4.07×, 5.09×4.87×, 4.62×, 4.68×
R05 · RL Open Build2.54×; 1.88×, 1.53×1.40×, 2.45×, Invalid
S13 · Serving Open Build1.10×; 0.99×, 1.08×1.06×, 1.07×, 0.85×

References:

[1] Yeon et al., InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents, arXiv:2607.20468, 2026.

[2] Kamahori et al., VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?, arXiv:2605.06068, 2026.

[3] Artificial Analysis, Artificial Analysis Coding Agent Benchmarks, 2026.

[4] Kondra et al., FrontierSWE v2, Proximal Blog, 2026.

[5] Merrill et al., Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces, arXiv:2601.11868, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chair and Reviewers: Summary of Revisions and Discussion Outcomes

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers (Reviewers Eyc3, dG94, and 6BXd) for their constructive feedback throughout the review and discussion period. Their valuable insights have helped us substantially strengthen the paper, and we are grateful for their recognition of our efforts and added experiments.

Within the practical compute constraints of end-to-end multi-NPU benchmarking, we have worked to resolve every actionable technical, statistical, and methodological question raised—adding 45 new independent 2-hour agent runs, a 2-hour random configuration search across all Joint Optimization tasks, a full source-review audit and 24/24 independent cross-review, and controlled capability-vs-difficulty analyses in our revised manuscript. We briefly summarize these updates below:

1. Concrete Additions in the Revised Manuscript

  • Targeted 54-Attempt Repeatability Study (Section 4.1, Figure 5, Appendix I.1, Tables 24–25): To verify run-to-run stability beyond single-shot runs, we conducted 45 new independent 2-hour runs alongside 9 original GPT-6 Astra runs across 9 representative tasks spanning Implementation Optimization (T03, R01, S02), Model Adaptation (T05, R04, S12), and Open Build (T06, R05, S13) for GPT-6 Astra and GPT-6.1 Sol with Codex. Both achieve 100% valid delivery (36/36) across all tested Pre-Training and Serving tasks, while all 4 failures occur in RL Training (R04/R05, 14/18 valid), confirming that RL integration brittleness persists across independent runs.
  • Baseline Headroom & Core RL Loop Profiling (Section 4.2, Tables 3–4, Appendices C.5 & F.5, Table 20): We added a 2-hour random configuration search across all 5 Joint Optimization tasks (T04, R02, R03, S10, S11), showing top agent implementations outperform pure configuration tuning on every task (1.27×, 5.99×, 16.43×, 2.64×, 1.11× vs. 1.04×, 0.75×, 1.03×, 2.63×, 1.00×). Phase-by-phase PPO (R03) profiling also confirms that all 4 valid submissions accelerate the core online RL loop by 2.41–14.42× (excluding initialization and finalization).
  • Source-Review Audit & Independent Cross-Review (Section 3.3, Appendix D.6, Tables 16–17): We added the complete human-audit breakdown across all 180 reviewed submissions (168 initial acceptances, 9 initial rejections with 2 overturned by authors, and 3 needs_evidence cases resolved as acceptances, yielding 173 accepted and 7 rejected), plus an independent 24/24 anonymized cross-review of GPT-6 Astra's submissions by GPT-6.1 Sol (High) with Codex and the authors.
  • Fixed-Subset Comparisons, Failure Diagnoses & Trajectory Insights (Sections 4.1–4.5, Table 10, Appendices C.4, F.1, G): We added Common-9 GM (Table 10) comparing all 9 agents on the exact same 9 tasks solved by all agents, expanded code-level failure diagnoses and runtime-feedback trajectory analyses (Table 21), and updated Limitations (Section 5) to document our repeatability scope and compute trade-offs.

2. Perspective on Remaining Discussion Points

  • Practical Compute Trade-offs in Multi-Run Coverage (Reviewer 6BXd): Reviewer 6BXd noted that repeating all 9 original agents would further verify cross-agent rankings. Because each end-to-end run requires up to 2 hours on dedicated Ascend 910B NPUs plus heavy LLM token usage (18M–173M median input tokens per run, Table 2), repeating the full 9×249 \times 24 matrix multiple times is impractical. Focusing 54 attempts across all 3 stages and 3 scopes bounds variance where architectural freedom is highest, and we explicitly note this scope boundary in Section 5.
  • Distinct Capabilities & Capability-vs-Difficulty Disentanglement (Reviewer Eyc3): Unlike localized patch, kernel, or standalone serving benchmarks, AscendInfraBench tests stateful cross-component integration under open architectural choices—whether agents can reorganize layouts, residency, and schedules without violating state invariants across restarts, policy weight handoffs, and packed-sequence boundaries. Rather than a uniform difficulty scalar, results show non-monotonic rank crossovers (e.g., agents with identical 17/18 Optimization completion rates diverging to 3/3, 2/3, and 1/3 in Open Build; 5 of the 8 agents passing LFM2.5 in both Pre-Training T05 and Serving S12 failing RL R04, while GLM-5.3 succeeds in R04 despite fewer overall tasks) paired with code-level root-cause isolation (packed-document short-convolution state leaks and actor-to-rollout weight-mapping errors).

We thank the Area Chair and all reviewers again for their time, rigor, and helpful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.