- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces AscendInfraBench, a benchmark of 24 LLM-infrastructure tasks on Ascend 910B NPUs spanning training, RL, and serving. Tasks span implementation optimization, joint optimization, model adaptation, and building complete systems. Evaluation independently rebuilds submissions, checks numerical correctness and state continuity, measures performance, and reviews source code. Across 216 two-hour runs by 9 coding agents, GPT-6 Astra solves all 24 tasks with a 1.63× geometric-mean speedup, and scope-level results reveal capability differences that optimization-only scores miss.
Critical strengths
The stage-by-scope design exposes capabilities that optimization-only evaluation would miss. Tasks cover training, RL, and serving, while distinguishing implementation optimization, joint optimization, model adaptation, and complete-system construction. As Table 1 shows, no prior benchmark covers all nine cells
Critical weaknesses
- The headline comparisons rest on fragile statistics and baselines of uncertain strength. (1) Each agent–task pair receives only one development run. This materially limits confidence in the comparative findings, especially because Model Adaptation and Open Build contain only three tasks each. (2) Table 14 shows the R03 baseline spends 1012 of 1298 s in initialization, so much of that gain is removing framework startup overhead, not improving the RL loop itself.
- Source review is assisted by the top-ranked evaluated model. Appendix D.6 states that GPT-6 Astra assists the source review that decides validity, and Astra is also the agent with a perfect 24/24 record. Authors make final decisions, but the paper reports no count of cases where humans overrode the model, and no indication that Astra’s own submissions were reviewed by a different model.
Questions for authors
- Were any independent development reruns conducted beyond the 216 reported runs? If so, how stable are validity counts, performance coverage, and agent ordering? If none exist, a targeted repeated-run study would be an additional experiment needed to resolve this uncertainty
- How were the performance-baseline configurations selected and tuned? What's the headroom for speedup?
9 comments
Response to Reviewer 6BXd (2/3): Re-W2
Re-W2
Source review is assisted by the top-ranked evaluated model. Appendix D.6 states that GPT-6 Astra assists the source review that decides validity, and Astra is also the agent with a perfect record. Authors make final decisions, but the paper reports no count of cases where humans overrode the model, and no indication that Astra's own submissions were reviewed by a different model.
Thank you for highlighting this concern. Following LLM-assisted code auditing practices in recent benchmarks (such as InferenceBench [1]), we strip all evaluated model and agent-harness identifiers to anonymize submissions prior to review (while keeping the underlying target frameworks visible for contract verification), manually audit every reject and needs_evidence verdict, and independently cross-audited all 24 of GPT-6 Astra's submissions using both human authors and a separate model.
Specifically, out of 216 total runs, 180 submissions underwent source review (the remaining 36 submissions comprise 23 Correctness failures, 8 Measurement failures, and 5 unchanged Optimization baselines). Authors manually inspected all 9 initial reject cases and all 3 needs_evidence cases (plus ~50% of accept cases at random, finding zero discrepancies). Human review overturned 2 reject verdicts to acceptances (GLM-5.3 on R03 and T01) and resolved all 3 needs_evidence cases as acceptances (Kimi K3 on S13, Claude Fable 5.1 on R03, and GPT-6 Astra on R03), as summarized in Table R2. Furthermore, all 24 anonymized submissions from GPT-6 Astra were independently audited from scratch by GPT-6.1 Sol (High) with Codex and independently verified by the human authors, with both accepting all 24 submissions (Table R3). We will add Tables R2–R3 to Appendix D.6.
Table R2: Source-review accounting and human override breakdown across all 180 reviewed submissions.
| Review Stage / Outcome | Count | Audit Scope & Human Override Details |
|---|---|---|
| Total benchmark runs | 216 | 9 agents × 24 tasks |
| Submissions entering source review | 180 | Evaluated model and agent-harness identifiers stripped prior to review |
Initial accept (review agent) | 168 | ~50% randomly re-audited by authors; 0 overturned |
Initial reject (review agent) | 9 | 100% manually audited by authors; 2 overturned to accept (GLM-5.3 on R03 and T01) |
Initial needs_evidence (review agent) | 3 | 100% manually audited by authors; all 3 resolved as accept (Kimi K3 on S13; Claude Fable 5.1 & GPT-6 Astra on R03) |
| Final source-review verdict | 173 accept / 7 reject | 2 rejections overturned; 3 needs_evidence cases resolved as accept |
Table R3: Independent cross-review of all 24 anonymized submissions from GPT-6 Astra.
| Independent Cross-Review of GPT-6 Astra's 24 Submissions | Outcome |
|---|---|
| GPT-6.1 Sol (High) with Codex (independent model reviewer) | 24 / 24 accepted |
| Human Authors (independent manual audit) | 24 / 24 accepted |
References for this reviewer response are listed in comment 3/3.
Response to Reviewer 6BXd (3/3): Re-Q1 + Re-Q2
Re-Q1
Were any independent development reruns conducted beyond the 216 reported runs? If so, how stable are validity counts, performance coverage, and agent ordering? If none exist, a targeted repeated-run study would be an additional experiment needed to resolve this uncertainty
Yes. As shown in Table R1 (in Re-W1), we conducted 30 additional 2-hour runs across all 6 Model Adaptation and Open Build tasks (2 reruns for GPT-6 Astra to reach 3 attempts per task, and 3 runs for GPT-6.1 Sol under Codex). Across these reruns, validity counts, performance coverage, and task-level patterns remain consistent, reinforcing our cross-trajectory findings:
- Validity counts: GPT-6 Astra achieves 17/18 (94.4%) valid runs and GPT-6.1 Sol achieves 15/18 (83.3%), with both reaching 100% (3/3) validity on every Pre-Training and Serving task and all 4 failures occurring in RL (R04/R05).
- Performance coverage: Using exact unrounded speedups, runs reaching ≥1.10× are 10/18 for GPT-6 Astra and 9/18 for GPT-6.1 Sol (note that GPT-6 Astra's original S13 ratio is 1.0999×, which rounds to 1.10× in display but is below the ≥1.10× threshold), with tight speedup bands on T06 (4.07–5.09×) and S12 (1.44–1.60×).
- Relative task comparisons: GPT-6 Astra matches or exceeds GPT-6.1 Sol in validity across all 6 tasks, while each agent leads in speedup on different tasks (e.g., GPT-6 Astra leads on T06 and R05, whereas GPT-6.1 Sol achieves a higher median on S12 at 1.55× vs. 1.44×).
Re-Q2
How were the performance-baseline configurations selected and tuned? What's the headroom for speedup?
Optimization and Open Build baselines use each framework's official default or recommended configuration for the target model, workload, and NPU allocation (representing standard out-of-the-box deployment), while Model Adaptation baselines use minimal working author adaptations.
To measure pure configuration-tuning headroom, we ran a 2-hour random configuration search (sampling predefined configuration spaces without replacement under fixed seeds, counting startup and compilation toward the budget) on all 5 Joint Optimization tasks. As shown in Table R4, the best agent result outperforms random configuration search on all 5 tasks (following the configuration-search comparison approach in InferenceBench [1]), showing that agents can both select better configurations and improve performance through implementation changes. Across the broader 24-task suite, at least one valid submission achieves ≥1.03× speedup on every task, with strong individual headroom (such as 1.50× on S02, 1.26× on T01, and 1.10× on S13).
Table R4: Two-hour random configuration search vs. best agent results across all 5 Joint Optimization tasks.
| Task | Configurations tried | Best sampled configuration (excluding baseline) | Best reported agent result | Main agent changes |
|---|---|---|---|---|
| T04 · MindSpeed training | 10 | 1.04× | 1.27× · GPT-5.6 Sol | Config: interleaved pipeline stages. Implementation: reduced startup and shutdown overhead. |
| R02 · verl GRPO | 8 | 0.75× | 5.99× · GPT-6 Astra | Config: separate actor/rollout devices. Implementation: overlapped rollout and updates, packed training, direct weight transfer. |
| R03 · verl PPO | 8 | 1.03× | 16.43× · GPT-6 Astra | Implementation: resident PPO runtime, shared actor/rollout, graph decoding, concurrent actor/critic updates. |
| S10 · SGLang serving | 9 | 2.63× | 2.64× · GPT-6 Astra | Config: decode graphs and graph batch sizes. Implementation: paged-attention graph capture and synchronized input updates. |
| S11 · vLLM-Ascend serving | 8 | 1.00× | 1.11× · Claude Fable 5.1 | Config: larger KV cache and prefill batches, expert-weight prefetch, CPU affinity, task queue settings. |
References:
[1] Yeon et al., InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents, arXiv:2607.20468, 2026.
Response to Reviewer 6BXd (1/3): Re-W1
Re-W1
Thank you for raising both points. Regarding (1), we added 30 independent 2-hour reruns across all 6 Model Adaptation and Open Build tasks (shown in Table R1 and analyzed in Re-Q1), confirming 100% delivery stability on Pre-Training and Serving.
Regarding (2), our task objective is end-to-end job throughput, allowing agents to freely identify and optimize any bottleneck across the entire execution window. The fact that most agents identified and eliminated startup overhead in the recommended verl 2-NPU PPO setup reflects adherence to the end-to-end task specification—and critically, agents also substantially accelerate the core online RL training loop itself. As shown in Table 14b, excluding initialization and finalization, all 4 valid submissions achieve 2.41–14.42× speedups inside the online RL loop alone. Specifically, all 4 valid implementations eliminate weight-transfer bottlenecks by generating rollouts directly from the training actor rather than a separate rollout engine, and GPT-6 Astra (14.42× inside-loop, 16.43× end-to-end) further keeps model and optimizer states resident on NPUs, uses graph decoding, and overlaps actor and critic updates.
Table R1: Repeated-run speedups across all 6 Model Adaptation and Open Build tasks (30 new 2-hour runs).