- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces InfraDev, a benchmark for evaluating coding agents on real-world AI infrastructure tasks from Megatron-LM, vLLM, and verl. It constructs 36 multi-file, often multi-GPU tasks from merged pull requests, strengthens evaluation by jointly synthesizing hidden probes and task prompts, and introduces a shared execution environment to reduce evaluation cost. Experiments on six frontier coding agents show that the best model resolves only 38.9% of tasks.
Critical strengths
- The joint design of hidden probes and task prompts is well motivated and appears to be a sensible solution to a real benchmark construction problem: stronger tests are only useful if the prompt also specifies the behaviors and interfaces those tests expect.
- The paper studies an important and timely problem, as coding agents are increasingly being used for large-scale AI infrastructure development, while existing benchmarks still focus largely on kernels, single-GPU tasks, or narrower software engineering settings.
- The benchmark is grounded in real-world repositories and merged pull requests from Megatron-LM, vLLM, and verl, making the tasks substantially more realistic than synthetic or isolated optimization benchmarks.
Critical weaknesses
This paper presents a timely and practically relevant benchmark for evaluating coding agents on real-world AI infrastructure development. However, I have several concerns about the benchmark characterization and the conclusions drawn from its failure analysis.
(1) The failure taxonomy is somewhat coarse and its decision boundaries are not sufficiently clear. The paper defines structural, numerical, and behavioral errors, but in practice the distinction between categories such as state/lifecycle errors, condition-specific behavioral errors, and execution/API errors can be ambiguous. Since the failure analysis is one of the paper’s main empirical contributions, a more operational definition of each category, together with inter-annotator agreement or representative borderline examples, would make the conclusions more convincing.
(2) The task-type coverage is highly imbalanced. Among the 36 tasks, 19 are new features and 15 are performance optimizations, while there is only one refactor and one bug-fix task. This makes it difficult to know whether the conclusions generalize to other common forms of infrastructure development, especially debugging, maintenance, and large-scale refactoring.
(3) The paper does not sufficiently characterize the tasks across different layers of the AI infrastructure stack. Modern AI infrastructure spans multiple layers, from high-level distributed training/serving systems to operators, kernels, runtimes, and hardware-facing components. Although the benchmark draws from Megatron-LM, vLLM, and verl, it is unclear how the selected PRs are distributed across these layers, or whether the benchmark predominantly exercises one particular level of the stack.
Questions for authors
(1) Could the authors clarify the operational definitions and decision boundaries of the failure taxonomy? In particular, how are ambiguous cases distinguished between execution/API, state/lifecycle, and condition-specific behavioral errors, and how consistent are these labels across annotators or repeated labeling?
(2) Could the authors provide a clearer characterization of InfraDev across the AI infrastructure stack? For example, how many selected PRs primarily target high-level training/serving logic, distributed runtime and communication, operators/kernels, or lower-level hardware-facing components, and does agent performance differ across these layers?
2 comments
Reply to Reviewer cU7a (Part 2/2: W2, W3 & Q2)
Continuing from Part 1, we clarify our focus on feature and optimization workloads (W2). We also present task coverage alongside agent performance across the AI infrastructure stack (W3 & Q2).
W2: Intentional Focus on Cross-Module Features and Performance Optimizations
Thank you for raising this question on task-type distribution. Concentrating on substantial feature development (19 tasks) and performance optimizations (15 tasks), alongside refactoring (1) plus bug-fix work (1; Table 1, Table 9), is an intentional strength of InfraDev's design:
- Targeting critical, under-evaluated workloads: Prior repository-level benchmarks focus heavily on localized bug fixes, whereas production AI infrastructure engineering relies critically on cross-module feature builds and verifiable performance optimizations—where existing benchmarks rarely test actual memory, throughput, or communication savings.
- Complementary benchmark scope: Selecting PRs with >=1,000 changed lines and shipped tests captures these complex multi-file workloads directly, complementing benchmarks designed for localized debugging or routine maintenance.
Paper revision. Section 3.1.1 links to the complete 36-task inventory (Table 9), and Section 5 notes how InfraDev complements benchmarks targeting smaller maintenance edits.
W3 & Q2: Task Distribution and Agent Performance Across the AI Infrastructure Stack
Thank you for suggesting an explicit stack-layer characterization. InfraDev already spans the full AI infrastructure stack. Assigning each PR to the layer where it adds the most non-test code illustrates both task distribution and per-layer agent performance:
| Layer & Primary component | Tasks | Official | + Synthesis | + Control |
|---|---|---|---|---|
| L1: High-level training, RL, and serving logic | 14 | 56.0% | 34.5% | 35.7% |
| L2: Distributed runtime and communication | 9 | 46.3% | 31.5% | 37.0% |
| L3: Operators, attention, and fused kernels | 6 | 41.2% | 35.3% | 41.2% |
| L4: Hardware-facing memory and CUDA-graph runtime | 7 | 36.6% | 22.0% | 34.1% |
| All evaluated tasks | 36 | 47.4% | 31.5% | 36.6% |
Two findings directly answer Q2:
- Balanced full-stack coverage: Rather than concentrating on a single level, InfraDev covers high-level training/RL/serving logic (14 tasks), distributed runtime (9), custom operators/kernels (6), and memory/CUDA-graph runtime (7).
- Consistent difficulty once hidden probes are applied: Under official tests alone, high-level logic appears substantially easier than low-level runtime code (56.0% on L1 vs. 36.6% on L4), because high-level PR tests more often miss edge and resource checks. With our synthesized probes under Over-specificity control (+ Control), resolved rates span a narrow band of 34.1% to 41.2%, showing that every layer of the stack poses comparable challenge to current agents.
Paper revision. Section 4.2 highlights this cross-layer finding, Table 5 in Appendix C gives the full breakdown, and Table 9 records each task's primary stack layer (all highlighted in blue).
Reply to Reviewer cU7a (Part 1/2: W1 & Q1)
We thank you for your constructive review and helpful suggestions. We organize our response into two short parts: Part 1 addresses W1 & Q1, and Part 2 covers W2, W3 & Q2.
W1 & Q1: Explicit Operational Boundaries, Matching Order, and Blind Agreement
Thank you for asking us to detail the operational boundaries among execution/API, state/lifecycle, and condition-specific behavioral errors, as well as inter-annotator consistency. Our failure taxonomy in Section 4.4 assigns each case from concrete execution traces and code inspection. To make these decision rules completely explicit and reproducible when a failure exhibits multiple symptoms, we expanded Appendix D to spell out the exact five-step matching order (assigning the first category that matches):
To verify that independent annotators reproduce our labels consistently from these written definitions alone, two external blind LLM judges from different model families (neither among the six evaluated models) labelled a stratified sample of 120 failure groups (20 per error type) from scratch without seeing our labels:
This strong independent agreement (91.9%, kappa = 0.865 against our corpus labels) confirms that our failure categories in Section 4.4 are objective and reproducible.
Paper revision. Appendix D (alongside Table 10, highlighted in blue) details these operational boundaries, the five-step order, and the blind agreement validation.