acceptodds
Under review as a conference paper at ICLR 2027

InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?

Abstract

Coding agents are beginning to develop the production-grade AI infrastructure that trains and serves large models, yet existing benchmarks cover only single kernels or single-GPU tasks, and grade with tests and prompts that misjudge both correct and incorrect patches. We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning. Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs. To grade such changes, we jointly synthesize hidden probes, which enforce the requirements the original tests leave unchecked, and a prompt that states every interface the probes call without revealing the solution. We also design a shared execution environment that reuses dependencies and artifacts across tasks, cutting environment build time by 30-50% and storage by 42-46% on Megatron-LM and vLLM. Across six frontier models, the best resolves only 38.9% of the tasks; the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task. Failure analysis shows that structural and state errors dominate and that agents' own tests rarely check memory, time, or traffic. Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.

Primary area:
datasets and benchmarks
Keywords:
AI Infrastructure, Benchmark, Agent, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cU7aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InfraDev, a benchmark for evaluating coding agents on real-world AI infrastructure tasks from Megatron-LM, vLLM, and verl. It constructs 36 multi-file, often multi-GPU tasks from merged pull requests, strengthens evaluation by jointly synthesizing hidden probes and task prompts, and introduces a shared execution environment to reduce evaluation cost. Experiments on six frontier coding agents show that the best model resolves only 38.9% of tasks.

Critical strengths
  • The joint design of hidden probes and task prompts is well motivated and appears to be a sensible solution to a real benchmark construction problem: stronger tests are only useful if the prompt also specifies the behaviors and interfaces those tests expect.
  • The paper studies an important and timely problem, as coding agents are increasingly being used for large-scale AI infrastructure development, while existing benchmarks still focus largely on kernels, single-GPU tasks, or narrower software engineering settings.
  • The benchmark is grounded in real-world repositories and merged pull requests from Megatron-LM, vLLM, and verl, making the tasks substantially more realistic than synthetic or isolated optimization benchmarks.
Critical weaknesses

This paper presents a timely and practically relevant benchmark for evaluating coding agents on real-world AI infrastructure development. However, I have several concerns about the benchmark characterization and the conclusions drawn from its failure analysis.

(1) The failure taxonomy is somewhat coarse and its decision boundaries are not sufficiently clear. The paper defines structural, numerical, and behavioral errors, but in practice the distinction between categories such as state/lifecycle errors, condition-specific behavioral errors, and execution/API errors can be ambiguous. Since the failure analysis is one of the paper’s main empirical contributions, a more operational definition of each category, together with inter-annotator agreement or representative borderline examples, would make the conclusions more convincing.

(2) The task-type coverage is highly imbalanced. Among the 36 tasks, 19 are new features and 15 are performance optimizations, while there is only one refactor and one bug-fix task. This makes it difficult to know whether the conclusions generalize to other common forms of infrastructure development, especially debugging, maintenance, and large-scale refactoring.

(3) The paper does not sufficiently characterize the tasks across different layers of the AI infrastructure stack. Modern AI infrastructure spans multiple layers, from high-level distributed training/serving systems to operators, kernels, runtimes, and hardware-facing components. Although the benchmark draws from Megatron-LM, vLLM, and verl, it is unclear how the selected PRs are distributed across these layers, or whether the benchmark predominantly exercises one particular level of the stack.

Questions for authors

(1) Could the authors clarify the operational definitions and decision boundaries of the failure taxonomy? In particular, how are ambiguous cases distinguished between execution/API, state/lifecycle, and condition-specific behavioral errors, and how consistent are these labels across annotators or repeated labeling?

(2) Could the authors provide a clearer characterization of InfraDev across the AI infrastructure stack? For example, how many selected PRs primarily target high-level training/serving logic, distributed runtime and communication, operators/kernels, or lower-level hardware-facing components, and does agent performance differ across these layers?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer cU7a (Part 1/2: W1 & Q1)

We thank you for your constructive review and helpful suggestions. We organize our response into two short parts: Part 1 addresses W1 & Q1, and Part 2 covers W2, W3 & Q2.

W1 & Q1: Explicit Operational Boundaries, Matching Order, and Blind Agreement

Thank you for asking us to detail the operational boundaries among execution/API, state/lifecycle, and condition-specific behavioral errors, as well as inter-annotator consistency. Our failure taxonomy in Section 4.4 assigns each case from concrete execution traces and code inspection. To make these decision rules completely explicit and reproducible when a failure exhibits multiple symptoms, we expanded Appendix D to spell out the exact five-step matching order (assigning the first category that matches):

  • 1. Execution or API (evaluated before State/lifecycle): The test fails before producing a checkable output because an import fails, a required symbol is missing, or execution crashes. A run reaches subsequent categories only after executing and producing outputs.
  • 2. Shape, layout, or dtype: A tensor has an incorrect dimension, stride, contiguity, or data type.
  • 3. Resource: Execution and tensor shapes are valid, yet the run misses a quantitative memory, speed, communication, or time bound set by the task.
  • 4. State or lifecycle vs. Condition-specific behavior: State or lifecycle applies when persistent state across calls (such as a cache, reference count, checkpoint, or static CUDA-graph buffer) is corrupted even on the ordinary stateful path. Condition-specific behavior applies when the ordinary path succeeds and failure appears only under a specific flag, boundary input, validation guard, or CUDA-graph replay.
  • 5. Numerical: Execution, shapes, and state succeed, yet output values are wrong. We classify the case as precision-only when the formula is exact while floating-point precision falls short, or algorithm otherwise.

To verify that independent annotators reproduce our labels consistently from these written definitions alone, two external blind LLM judges from different model families (neither among the six evaluated models) labelled a stratified sample of 120 failure groups (20 per error type) from scratch without seeing our labels:

ComparisonSampleAgreementCohen's kappa
Between the two blind judges (3 error classes)120 sampled groups86.7%0.771
Between the two blind judges (7 error types)120 sampled groups82.5%0.788
Blind judges vs. our corpus labels (3 error classes)Groups where judges agree91.9%0.865

This strong independent agreement (91.9%, kappa = 0.865 against our corpus labels) confirms that our failure categories in Section 4.4 are objective and reproducible.

Paper revision. Appendix D (alongside Table 10, highlighted in blue) details these operational boundaries, the five-step order, and the blind agreement validation.

AuthorsOpenReview ↗

Reply to Reviewer cU7a (Part 2/2: W2, W3 & Q2)

Continuing from Part 1, we clarify our focus on feature and optimization workloads (W2). We also present task coverage alongside agent performance across the AI infrastructure stack (W3 & Q2).

W2: Intentional Focus on Cross-Module Features and Performance Optimizations

Thank you for raising this question on task-type distribution. Concentrating on substantial feature development (19 tasks) and performance optimizations (15 tasks), alongside refactoring (1) plus bug-fix work (1; Table 1, Table 9), is an intentional strength of InfraDev's design:

  • Targeting critical, under-evaluated workloads: Prior repository-level benchmarks focus heavily on localized bug fixes, whereas production AI infrastructure engineering relies critically on cross-module feature builds and verifiable performance optimizations—where existing benchmarks rarely test actual memory, throughput, or communication savings.
  • Complementary benchmark scope: Selecting PRs with >=1,000 changed lines and shipped tests captures these complex multi-file workloads directly, complementing benchmarks designed for localized debugging or routine maintenance.

Paper revision. Section 3.1.1 links to the complete 36-task inventory (Table 9), and Section 5 notes how InfraDev complements benchmarks targeting smaller maintenance edits.

W3 & Q2: Task Distribution and Agent Performance Across the AI Infrastructure Stack

Thank you for suggesting an explicit stack-layer characterization. InfraDev already spans the full AI infrastructure stack. Assigning each PR to the layer where it adds the most non-test code illustrates both task distribution and per-layer agent performance:

Layer & Primary componentTasksOfficial+ Synthesis+ Control
L1: High-level training, RL, and serving logic1456.0%34.5%35.7%
L2: Distributed runtime and communication946.3%31.5%37.0%
L3: Operators, attention, and fused kernels641.2%35.3%41.2%
L4: Hardware-facing memory and CUDA-graph runtime736.6%22.0%34.1%
All evaluated tasks3647.4%31.5%36.6%

Two findings directly answer Q2:

  • Balanced full-stack coverage: Rather than concentrating on a single level, InfraDev covers high-level training/RL/serving logic (14 tasks), distributed runtime (9), custom operators/kernels (6), and memory/CUDA-graph runtime (7).
  • Consistent difficulty once hidden probes are applied: Under official tests alone, high-level logic appears substantially easier than low-level runtime code (56.0% on L1 vs. 36.6% on L4), because high-level PR tests more often miss edge and resource checks. With our synthesized probes under Over-specificity control (+ Control), resolved rates span a narrow band of 34.1% to 41.2%, showing that every layer of the stack poses comparable challenge to current agents.

Paper revision. Section 4.2 highlights this cross-layer finding, Table 5 in Appendix C gives the full breakdown, and Table 9 records each task's primary stack layer (all highlighted in blue).

Reviewer_wYdZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

InfraDev is a benchmark of 36 tasks from GPU-heavy open-source projects (Megatron-LM, vLLM, and verl). They source these from PRs and create hidden probes to determine if tasks pass. The best agent achieves ~39% and the hidden probes reject ~32% of passes accepted by tests.

Critical strengths

GPU kernel engineering is an increasingly important task.

Critical weaknesses

At 36 tasks, I would expect them to be listed. They were not. Additionally, there was no human judgement or statistical tests (e.g., measuring Cohen's kappa) on the agreement of the probes vs humans. Given the probes were generated by AI, I have no confidence that the probes are sufficient nor necessary to determine if patches match.

There are also many key papers on how to build good agentic benchmarks, including a lot of work showing issues in agentic benchmarks. This paper does not make an effort to show that they tried to ensure that issues in prior agentic benchmarks do not appear in this benchmark.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Reply to Reviewer wYdZ (Part 1/2: W1 & W2)

We thank you for your constructive feedback. To address your questions thoroughly, we supplemented our paper with the complete 36-task inventory, external blind statistical validation (Cohen's kappa) confirming our execution-gated probes, and an expanded discussion connecting prior benchmark studies to InfraDev's built-in safeguards. Part 1 covers W1 & W2; Part 2 covers W3.

W1: Complete Inventory of All 36 Tasks

Thank you for requesting the full task-level breakdown. Actually, we have already put all the tasks in supplementary materials in our original submission. To further complement the repository summary in Table 1, we now added Table 9 in Appendix C (cited in Section 3.1.1, highlighted in blue), listing all 36 tasks in full with their repository, PR number, PR title, task type, stack layer, changed files, +/- lines, probe files, and verified adversarial patches.

W2: Execution-Gated Probes and Statistical Validation (Cohen's kappa)

In InfraDev, candidate probes are never admitted on subjective AI opinion: from the start, every probe is gated by deterministic, executable checks in the grading environment. To supplement these execution guarantees with the independent statistical validation you requested, we conducted two external blind LLM-judge studies (using models outside the six evaluated agents, with the golden patch withheld):

Validation studyScopeKey result
Per-assertion blind auditAll 34 runs rejected by + Synthesis>90% judge agreement, Cohen's kappa = 0.775 (retest 0.966)
Over-specificity control (+ Control)All 6 tasks with unstated-detail checks11/11 valid alternatives accepted; 20/20 real-error rejections kept
Probe vs. judge agreementAll 101 official-passing runs81.8% agreement (Cohen's kappa = 0.494 under + Control)

Together, our execution gates alongside these blind statistical audits confirm necessity, sufficiency, and strong agreement with independent judgement:

  • Executable necessity and compatibility (Algorithms 2-3): Under CheckProbes (Algorithm 2), every probe must pass 3 consecutive runs on the golden patch. Meanwhile, it must fail on the pre-PR snapshot, on a hollow patch where PR-added functions raise exceptions, and on all 114 adversarial patches that fool the official tests. Over-specificity control (Algorithm 3, RelaxProbes) further ensures probes check only specified behaviors and accept valid alternative patches (positive controls).
  • High blind agreement on probe rejections (Cohen's kappa = 0.775): Two external blind LLM judges auditing every failing assertion across all 34 runs rejected by + Synthesis reach Cohen's kappa = 0.775 (retest 0.966). They confirm 20 genuine semantic, numerical, or resource errors missed by official tests, while + Control accepts all 11/11 valid alternatives and retains all 20/20 real-error rejections (Figure 2c).
  • Sufficiency and agreement across all official-passing runs: Following DeepSWE (Huang et al., 2026), a second pair of external blind judges (kappa = 0.558) evaluated every run without the golden patch. Across all 101 official-passing runs, our probes with Over-specificity control achieve 81.8% probe-vs-judge agreement (kappa = 0.494), cutting false positives across the benchmark from 24.1% (Official) to 14.8% with the lowest overall disagreement (15.9%; Figure 2b).

Paper revision. Table 9 (Appendix C) lists all 36 tasks, Algorithm 3 (Appendix A.1) details RelaxProbes, and Section 4.3 with Figure 2 presents the blind Cohen's kappa validations (highlighted in blue).

AuthorsOpenReview ↗

Reply to Reviewer wYdZ (Part 2/2: W3)

Continuing from Part 1, we connect recent literature on agentic-benchmark pitfalls directly to InfraDev's design (W3).

W3: Built-In Safeguards Addressing Known Agentic-Benchmark Pitfalls

Thank you for highlighting this body of work. InfraDev was designed from the ground up to prevent the core evaluation pitfalls identified in recent PR-benchmark studies. We expanded Section 2 to cite these works and map each issue explicitly to our safeguards in Sections 3.1-3.2:

  • Inherited PR tests that are too weak or too strict: Recent studies show that shipped PR tests often accept broken shortcut patches (Yu et al., 2025 [UTBoost]; Yu et al., 2026 [SWE-ABS]; Wang et al., 2026), yet can also reject valid alternatives by checking unstated details (OpenAI, 2026; Huang et al., 2026 [DeepSWE]). InfraDev directly counters weak tests via verified adversarial patches plus hidden probes. At the same time, it prevents over-strict checks via joint prompt-probe alignment and Over-specificity control (Sections 3.1.2, 4.3).
  • Underspecified prompts vs. solution disclosure: Prior benchmarks curate prompts manually or append interface lists (Chowdhury et al., 2024 [SWE-bench Verified]; Deng et al., 2025 [SWE-Bench Pro]; Badertdinov et al., 2026 [SWE-rebench V2]). InfraDev synthesizes each prompt jointly with the probes using a structured template (Appendix B), enforcing symbol completeness via a static linter and solution non-disclosure via an independent checker.
  • Environment stability vs. test tampering: Recent audits highlight broken grading environments alongside agents modifying weak or visible tests (Zhu et al., 2025 [ABC]; Baker et al., 2025). InfraDev validates every task in the actual grading container via CheckProbes (Algorithm 2), restores a clean post-PR test directory during grading, and keeps hidden probes entirely outside the agent container (Section 3.2).
  • PR lookup or memorization: Prompts prohibit looking up the PR on pre-PR snapshots (Figure 8). Moreover, our ablation replacing synthesized prompts with original GitHub PR descriptions drops Megatron-LM resolution to 5.3% (Official) and 0.0% (+ Synthesis / + Control; Figure 2d), confirming agents cannot pass via PR recall.

Paper revision. Section 2 expands the discussion of repository-level benchmark studies, while Sections 3.1.2, 3.2, and 4.3 connect each safeguard (highlighted in blue).

Reviewer_AK9HOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

InfraDev is a benchmark for coding agents working on AI infrastructure. It includes 36 tasks drawn from merged pull requests in Megatron-LM, vLLM, and verl, spanning model training, inference, and reinforcement learning. The tasks require changes across repositories rather than isolated code edits, and 20 involve multiple GPUs.

The paper’s central idea is to develop hidden tests and task descriptions together. The tests check numerical and behavioral requirements that the projects’ existing suites may miss; the prompts are then written to spell out the relevant interfaces and expected behavior. The authors also provide a shared execution environment to reduce the effort spent rebuilding dependencies and downloading artifacts.

The benchmark evaluates six coding agents, with the best resolving 38.9% of tasks. The hidden tests fail about 32% of submissions that pass the official tests. The authors find recurring problems with structural integration, state handling, and execution modes. They also turn these observations into a reusable skill, which raises one model’s success rate on the 19 Megatron-LM tasks from 42.1% to 52.6%.

Critical strengths
  • The paper addresses an important blind spot in coding-agent benchmarks. AI infrastructure work often requires coordinating APIs, numerical behavior, persistent state, and resource use—not just implementing a self-contained kernel.
  • Developing task descriptions alongside tests is a sensible approach. More demanding tests are only fair when the requirements they enforce are communicated clearly; the paper considers both underspecification and false acceptance.
Critical weaknesses
  • Showing that probes accept the golden patch and reject selected incorrect patches does not establish that they also accept other valid implementations. Since the golden patch informs both prompt and probe construction, the risk of encoding reference-specific assumptions remains.
  • The paper appears to use two different definitions of task resolution. Section 3.2 compares a submission’s test count with the golden patch’s count, while tables and later sections sometimes require every test to pass. These rules can yield different results.
  • The benchmark is relatively small and favors large changes. The 1,000-line-change threshold may omit difficult but representative infrastructure fixes that do not involve a substantial feature addition.
Questions for authors

Thank you for a thoughtful submission. I especially appreciate the focus on repository-level infrastructure work and the recognition that numerical correctness alone is not enough to evaluate systems changes. Pairing task specification with hidden tests is a useful direction, and the failure analysis could inform future agent development. I lean toward acceptance.

My main concern is how reliably the generated grading suite separates incomplete solutions from valid alternatives.

The authors show that the probes accept the reference patch and reject a set of intentionally flawed patches. That is a useful check, but it does not rule out tests that are overly tied to the reference implementation. Likewise, spelling out the interfaces makes tasks more precise, but some of those interfaces may be choices made by the original contributor rather than requirements of the requested feature.

Could the authors explain how they distinguish required compatibility constraints from implementation-specific choices, and how they look for false negatives among rejected submissions? This is especially relevant to the finding that the hidden tests reject 32% of submissions that pass the official tests. Having independent reviewers examine a sample of those additional failures would help show how often they correspond to real semantic or resource errors. I also wonder whether specification and fixture failures excluded from the appendix’s failure analysis are still counted in the headline scores.

I have several further questions:

  • What is the precise rule for declaring a task resolved? Comparing total passed-test counts with the golden patch could allow a submission to fail a test the reference passes while offsetting that failure elsewhere. Would it be clearer to require all tests passed by the reference to pass, while reporting reference failures and skips separately?
  • For how many tasks did probe construction succeed? How many synthesis attempts failed, and how many adversarial patches were produced per task? These figures would help readers judge the process’s coverage and reliability.
3 comments
AuthorsOpenReview ↗

Reply to Reviewer AK9H (Part 1/3: W1 & Q1)

We thank you for your thoughtful review and encouraging assessment. We organize our reply into three short parts: Part 1 addresses W1 & Q1, Part 2 covers W2, Q3 & Q2, and Part 3 covers Q4 & W3.

W1 & Q1: Separating Compatibility Constraints from Implementation Choices

Thank you for asking how our framework ensures compatibility with valid alternative implementations. From the outset, our joint test-and-prompt synthesis is designed to test promised functionality rather than private details of the reference patch. To make this mechanism completely explicit and provide executable guarantees on open choices alongside CheckProbes (Algorithm 2), we detailed Over-specificity control (+ Control; Section 3.1.2, Algorithm 3):

  • Public interface contract vs. open choices: A static analyzer restricts the interface contract strictly to public symbols introduced by the golden patch that the tests import. Prompt synthesis then states each behavior explicitly either as a requirement or as a decision left to the agent, verified by a symbol linter and a non-disclosure checker.
  • Prompt-probe alignment (RelaxProbes, Algorithm 3): RelaxProbes ensures that open choices in the prompt are compatible with official tests. Conversely, it relaxes or removes any probe assertion on an unstated internal detail unless stating that detail reveals no part of the solution.
  • Executable positive controls: Wherever the prompt leaves an open choice, RelaxProbes constructs valid alternative implementations that differ from the golden patch (positive controls), requiring every probe to accept all of them while still passing CheckProbes against shortcuts.

Quantitative Validation of Probe Precision and Valid Alternatives

To quantify probe precision and verify how synthesized probes handle alternative implementations in practice, we conducted two external blind LLM-judge studies (using models outside the six evaluated agents, with the golden patch withheld):

Metric across all runsOfficial+ Synthesis+ Control
Resolved rate (all 6 models)47.4%31.5%36.6%
Resolved rate (GPT-6 Sol)58.3% (21/36)36.1% (13/36)44.4% (16/36)
Probe rejections: real error / valid alt. / split---20 / 11 / 320 / 0 / 3
False positive / False negative / Total (Figure 2b)24.1 / 14.0 / 18.8%13.6 / 19.8 / 17.6%14.8 / 16.5 / 15.9%

These audits answer both parts of Q1 directly:

  • Strong precision of synthesized probes (+ Synthesis): Two blind external judges evaluating every failing assertion across all 34 official-passing runs rejected by + Synthesis reach >90% agreement (Cohen's kappa = 0.775; retest 0.966). Across 31 of the 36 tasks (including all verl tasks), + Synthesis produces zero over-specific rejections, and its settled rejections capture 20 genuine semantic, numerical, or resource defects missed by official PR tests.
  • Full acceptance of valid alternatives under + Control: The remaining 11 cases involved valid alternatives on 5 tasks that differed on unstated internal details—such as tracking free memory pages via a different variable than the golden patch (Megatron-LM #2690) or raising the required ValueError earlier (#1989). Applying Over-specificity control removes those unstated-detail checks across the 6 relevant tasks, accepting all 11/11 valid alternatives upon re-evaluation while keeping all 20/20 real-error rejections (22.8% additional rejection rate; Figure 2c).
  • Lowest overall grading disagreement: Under the DeepSWE blind verifier audit (Huang et al., 2026; kappa = 0.558), + Synthesis already cuts false positives from 24.1% (Official) to 13.6%, and + Control brings false negatives down from 19.8% to 16.5%—achieving the lowest combined disagreement across all settings (15.9% vs. 18.8% Official; Figure 2b) with identical model rankings.

Paper revision. Section 3.1.2 alongside Algorithm 3 (Appendix A.1) details Over-specificity control. Section 4.3, Figure 2, and Table 6 present the ablation results alongside the blind validation audits (highlighted in blue).

AuthorsOpenReview ↗

Reply to Reviewer AK9H (Part 2/3: W2, Q3 & Q2)

Continuing from Part 1, we address our task-resolution criterion (W2 & Q3) and score accounting (Q2).

W2 & Q3: Explicit Per-Node Task Resolution and Reference Outcomes

Thank you for asking us to clarify the exact resolution rule and reference test outcomes. In InfraDev, our evaluation harness tracks individual pytest nodes against the reference run. Section 3.2 now states this exact per-node criterion explicitly across all tables and figures: a task is resolved only when the submission passes every test node that the golden patch passes.

Clarifying both points on offsetting and reference skips or failures:

  • Zero offsetting, with reference outcomes recorded separately: Requiring every golden-passing test node to pass ensures a submission can never offset a failed test by passing another. Across all 36 tasks, the golden patch passes 100% of hidden-probe nodes (534/534, zero failures or skips); on the official suites, environment-gated skips or reference failures on our setup are recorded separately rather than required of submissions.
  • High stability across definitions: Comparing strict per-node matching against simple passed-test count comparisons confirms identical outcomes on 99.1% of runs (211 of 213), with zero change to model rankings or conclusions.

Paper revision. Section 3.2 states this explicit per-node criterion uniformly across all reported tables and figures (highlighted in blue).

Q2: Full Inclusion of All Failing Tests in Benchmark Scores

Thank you for checking this detail. Yes—our benchmark scoring already counts every failing test in full toward % Resolved and % Passed across every reported table, with zero score adjustment.

Setting aside failures caused by prompt wording or test-environment setup applies strictly to the error-type breakdown in Section 4.4 (Table 10). This ensures our failure analysis attributes only patch-induced bugs while leaving every model's benchmark score completely unadjusted.

Paper revision. Appendix D adds an explicit note alongside Table 10 clarifying that benchmark scores are unadjusted (highlighted in blue).

AuthorsOpenReview ↗

Reply to Reviewer AK9H (Part 3/3: Q4 & W3)

Continuing from Part 2, we provide the complete probe-synthesis statistics (Q4) and clarify our benchmark curation focus (W3).

Q4: Probe Construction Success Rate, Revision Rounds, and Adversarial Patches

Thank you for requesting these statistics. Our automated probe-synthesis pipeline succeeded and passed CheckProbes (Algorithm 2) on all 36 of 36 tasks (100%), demonstrating consistent reliability across all three repositories:

RepositoryTasksProbe filesRevised (tasks / failed rounds)Adversarial patches (range / med. / total)
Megatron-LM19216 / 71-5 / 3.0 / 51
verl7140 / 04-11 / 6.0 / 45
vLLM10124 / 41-4 / 1.5 / 18
Total364710 / 111-11 / 3.0 / 114

Answering each part of Q4 directly:

  • 100% convergence with minimal revisions: All 36 tasks passed CheckProbes. Notably, 26 of 36 tasks passed on the very first attempt (0 failed rounds; median 0 per task), while the remaining 10 tasks converged within 1 to 2 revision rounds (11 failed rounds in total).
  • Verified adversarial patches per task: Synthesis produced 1 to 11 adversarial patches per task (median 3.0; 114 in total), every one passing the official tests while rejected by our hidden probes.

Paper revision. Table 4 in Appendix A.1 (cited in Sections 3.1.2, 4.3) reports repository-level statistics, while Table 9 in Appendix C gives per-task counts (highlighted in blue).

W3: Deliberate Focus on Cross-Module Infrastructure Engineering

Thank you for highlighting how our curation criteria shape the benchmark. Selecting PRs with shipped tests and >=1,000 changed lines is an intentional design choice targeting substantial, cross-module AI infrastructure engineering:

  • Complementing localized benchmarks: Existing coding benchmarks already cover small, localized bug fixes extensively. Real-world infrastructure development instead centers on complex feature builds and performance optimizations requiring distributed execution across multiple modules.
  • High-depth evaluation per task: Each of our 36 tasks represents a major production PR (19 features, 15 performance optimizations, 1 refactor, 1 bug-fix PR; Table 1, Table 9), backed by reusable containerized environments, 534 hidden-probe nodes, and 114 verified adversarial patches.

Paper revision. Section 3.1.1 links to the full 36-task inventory (Table 9), and Section 5 notes how InfraDev complements benchmarks targeting smaller localized maintenance tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.