acceptodds
Under review as a conference paper at ICLR 2027

-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

Abstract

Large language models (LLMs) now show strong reasoning and coding abilities, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. Existing benchmarks, however, mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and cannot evaluate open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present -Bench, a benchmark of 85 tasks grounded in real LLM training and inference codebases. The tasks cover nine categories of LLM infrastructure and come in three formats, ranging from kernel function completion to long-horizon implementation and repository-wide end-to-end optimization. They are built by a construction pipeline that turns real pull requests, issues, and repository code into executable tasks with validated tests, and every reward is gated by correctness and anti-hacking checks. Across eight frontier models, the strongest model, Claude Opus 5, scores only 36.53%, and every model performs worse on long-horizon, repository-level implementation than on kernel-level tasks. Trajectory analysis further suggests that stronger models treat optimization as controlled experimentation, validating ideas with cheap local runs and ruling out confounders before drawing conclusions. We will open-source -Bench to facilitate future research on LLM agents' ability to develop and optimize the LLM infrastructure that powers themselves.

Primary area:
datasets and benchmarks
Keywords:
LLM infrastructure, Benchmark, Performance Optimization, Long-Horizon Tasks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zWP6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether LLMs can engineer and optimize the infrastructure that powers them. The authors argue that existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore cannot evaluate the long-horizon and open-ended nature of real-world LLM infrastructure engineering. To address this gap, they build Φ-Bench, which contains 85 tasks in three formats: 55 Kernel Function Completion (KFC) tasks, 20 Long-Horizon Implementation (LHI) tasks, and 10 repository-wide End-to-End Optimization (E2EO) tasks. They analyze 2,260 papers and 1,852 repositories to construct a three-level taxonomy consisting of 9 domains, 62 topics, and 410 tags. The benchmark covers all 9 domains and 52 of the 62 topics. Guided by this taxonomy, the tasks are constructed through PR/issue rollback, curated code masking, and expert curation. Each task has either a functional implementation objective or a performance optimization objective. All task materials, including test cases, are visible to the evaluated models except the evaluation harness. During task validation, a performance task is retained only when its reference solution achieves at least 1.1 times the baseline and the starter or no-op solution does not outperform the baseline. For a functional implementation task, the reference solution must pass all tests, while the starter solution must fail at least one. Functional tasks receive a binary score of 0 or 1, whereas performance tasks receive a log-normalized score between 0 and 1. Both metrics are gated by checks for buildability, functional correctness, edit constraints, and reward hacking. The authors evaluate eight frontier models on Φ-Bench. Claude Opus 5 performs best but achieves an overall score of only 36.53%, and every evaluated model scores lower on LHI than on KFC. The trajectory analysis, which the authors acknowledge is limited to one E2EO task, suggests that stronger models use inexpensive local validation, control variables and measurement noise, and check possible confounding factors more carefully.

Critical strengths
  1. The first critical strength is that Φ-Bench extends evaluation from local kernels and pre-specified optimization targets to repository-level infrastructure engineering. Unlike ISO-Bench and CUDAHercules, which typically identify the target component or performance bottleneck in advance, Φ-Bench leaves bottleneck identification and the choice of optimization strategy to the agent, particularly in E2EO tasks. This extension is supported by more than the task formats themselves. The taxonomy is built from 2,260 papers and 1,852 repositories; the benchmark covers 52 of the 62 topics; and the tasks are constructed through PR/issue rollback, curated masking of repository code, and expert curation of research problems. Together, these provide checkable evidence that Φ-Bench expands the scope and open-endedness of evaluation rather than merely increasing the number of tasks. This supports acceptance because the benchmark targets realistic infrastructure-engineering abilities that are largely excluded by evaluations centered on local code completion or predefined bottlenecks.

  2. The second critical strength is the concrete and executable evaluation protocol. As discussed under Informativeness, it combines visible task materials and test cases with a hidden evaluation harness, objective-specific task-validation criteria, a single formal submission for KFC, and both rule-based and agent-based anti-hacking checks. The rule-based monitor flagged three requests, while the proctor agent found no confirmed hacking. These safeguards matter because infrastructure-performance evaluations can be distorted by incorrect implementations, measurement loopholes, or reward hacking. Without them, a higher score may not reflect stronger engineering ability. Although the protocol cannot address every validity concern, it ensures that positive rewards are gated by checks for buildability, functional correctness, edit constraints, and reward hacking, and provides a reusable basis for future benchmarks.

Critical weaknesses
  1. The first critical weakness is that score differences across task formats cannot be clearly interpreted as differences in the intrinsic difficulty of the formats. As discussed under Soundness, the reward type follows the task objective rather than the format, and the paper does not report the objective composition within each format. Although both rewards are bounded in [0,1][0,1], a binary pass/fail outcome and a continuous reward that increases after exceeding the reference solution have different interpretations. Therefore, while Table 2 shows that every model scores lower on LHI than on KFC, this difference cannot be directly attributed to the intrinsic difficulty of LHI because it may also be affected by objective composition and reward type. This issue is consequential because both the abstract and conclusion emphasize that models perform worse on long-horizon, repository-level implementation than on kernel-level tasks. The authors could address it by reporting the format-by-objective task composition, providing results stratified by objective, and conducting sensitivity analyses under unified or alternative scoring schemes.

  2. The second critical weakness is that the cross-model rankings and behavioral analysis lack sufficient comparability controls, uncertainty quantification, and breadth of evidence. First, GPT-5.6 Sol is evaluated with the Codex scaffold, whereas the other seven models use Claude Code. The paper does not explain whether this choice is constrained by model–scaffold compatibility or assess how scaffold differences may affect the rankings in Table 1. Second, the introduction claims “low run-to-run variability,” but the paper does not report the number of repeated runs, repeated-run means, variances, or confidence intervals. Finally, the trajectory conclusions about inexpensive local validation, variable control, and cautious attribution are based on only three models performing a single E2EO task, and the analysis is mainly qualitative. These limitations do not invalidate the scores observed in Table 1, but they mean that the cross-model rankings and the behavioral conclusion that experimental discipline distinguishes stronger from weaker models should be treated as preliminary. The scaffold and variability concerns could be mitigated by explaining the scaffold-selection constraints, comparing the same model under different scaffolds, and reporting repeated-run statistics. The trajectory conclusions require validation across more tasks and models.

Questions for authors
  1. Sections 3.1 and 3.5 state that the reward type is determined by the task objective rather than by whether the task is KFC, LHI, or E2EO. Please report the numbers of functional implementation and performance optimization tasks within each format, together with model results stratified by objective. Given that binary and continuous rewards have different interpretations, to what extent do the format-level averages in Table 2 reflect differences in the intrinsic difficulty of the task formats? Does the observed LHI–KFC performance gap remain when results are stratified by objective or aggregated using another clearly specified method? In addition, what motivates the 1.1× solvability threshold in §3.3, and are the conclusions robust to other reasonable thresholds? These results would directly affect whether I can interpret the lower scores on LHI as evidence that LHI is intrinsically more difficult than KFC.

  2. In Table 1, GPT-5.6 Sol is evaluated with the Codex scaffold, whereas the other seven models use Claude Code. Please clarify whether this choice is constrained by model–scaffold compatibility and provide any available evidence of comparability, such as results for the same model under both scaffolds. If such evidence is unavailable, please state the scope and limitations of the cross-model rankings. The paper also claims “low run-to-run variability”; please report the number of repeated runs and the corresponding means, variances, or confidence intervals. Finally, Figure 4 shows that some category–format cells contain only one or two tasks. Please therefore quantify the uncertainty of the category-level results in Table 1, for example by providing task-level results, bootstrap intervals, leave-one-task-out analyses, or another analysis that reflects sensitivity to task sampling.

  3. The paper uses network isolation, rule-based monitoring, and a proctor agent to prevent external solution retrieval or manipulation of the evaluation during task execution, but these mechanisms cannot rule out parametric memorization. Some tasks are reconstructed from PRs, issues, and code in public repositories, so the corresponding implementations may have appeared in the models’ pretraining data. Did the authors apply a temporal split, select tasks according to model knowledge cutoffs, rewrite the tasks, or evaluate on held-out variants to assess potential contamination? If not, how should the best model’s score of 36.53% be interpreted, and to what extent does it reflect the ability to solve previously unseen infrastructure-engineering problems?

Reviewer_UCGNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Φ-Bench to evaluate Large Language Model (LLM) performance on code-related infrastructure tasks. It contains 85 tasks: 55 Kernel Function Completion (KFC), 20 Long-Horizon Implementation (LHI), and 10 End-to-End Optimization (E2EO). The tasks originate from real repositories, PRs, code-infilling, and expert design, covering nine infrastructure domains. The authors evaluated eight models and analyzed behavioral differences during the iterative optimization process.

Critical strengths
  1. Practical value of the problem: extending evaluations to repository understanding, implementation, debugging, and performance optimization aligns closely with real-world applications of coding agents.

  2. Real engineering origin: the combination of reference implementations, correctness checks, and performance measurement serves as a reasonable foundation for building a usable benchmark.

  3. Insightful process analysis: observations regarding lightweight validation, controlling experimental variables, and eliminating compilation cache interference help explain why agents succeed or fail.

Critical weaknesses
  1. Scoring metrics are difficult to interpret: reaching the reference implementation's performance still yields a score of zero, and blending this with binary functionality scores makes the total score unreflective of actual task completion ability.

  2. Lack of sufficient basis for task categorization: some "end-to-end" tasks only evaluate a single operator call; allowing modifications across the entire repository does not prove the need for system-level bottleneck discovery or cross-module optimization.

  3. Unclear incremental value over existing work: there is a lack of thorough comparison with closely related repository-level, application-level, and open-ended optimization benchmarks (such as CUDA Kernels), and task-by-task evidence for coverage breadth is lacking.

  4. Insufficient experimental control and reproducibility: models used different agent frameworks, and submission budgets varied across task types. Repetitions, resource budgets, and confidence intervals were not fully reported, limiting comparisons of results.

  5. Inadequate verification of correctness and anti-cheating mechanisms: the division between public and hidden tests and the rules for penalty adjudication are unclear. There is also a lack of manual audits or documented cheating cases to verify the reliability of detection.

Questions for authors

Please refer to the weakness.

Reviewer_bmoEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for LLM infrastructure engineering. Starting from 2,260 systems papers and 1,852 production repositories, the authors build a taxonomy of 9 domains and 62 topics, and construct 85 executable tasks in three forms: 55 single-file kernels (KFC), 20 multi-file long-horizon tasks (LHI), and 10 repository-level end-to-end optimizations (E2EO). Implementation tasks are scored binary; performance tasks require passing correctness, edit-scope, and anti-cheat checks, then receive R = min(1, ln(s/s_ref)/ln(s_ref)), which is zero when the candidate does not beat the reference implementation. Construction checks require the reference to pass, the starter to fail, and the reference to be at least 1.1× faster. Eight frontier models are evaluated on a single H20; the best score is 36.53.

After rebuttal:Thanks for the detailed response. My main concerns are now addressed: the human-expert baseline, the adversarial test of the isolation mechanism, and the zero-score attribution supply the evidence I was missing, and the per-task breakdown makes clear what the score actually measures. I agree that the evaluation target is meaningful and far from a closed problem — models still trail human experts, and even the highest expert score leaves substantial headroom. The construction procedure is sound, and within its stated scope the benchmark does measure the intended capability. I will revise my assessment accordingly.

Critical strengths
  1. Well-chosen target. Treating "LLMs modifying their own infrastructure" as a distinct capability is more valuable than another kernel-level benchmark, and existing benchmarks indeed do not measure repository-level, open-ended engineering.
  2. Construction discipline is solid: tasks come from real repositories, with bidirectional validation (reference passes, starter fails, reference is measurably faster).
  3. Taxonomy coverage is analyzed rather than asserted, which supports the claim that tasks are not a random sample.
  4. The trajectory analysis yields an actionable qualitative insight: stronger models treat optimization as a controlled experiment (cheap proxy first, then one variable at a time, then careful attribution).
Critical weaknesses
  1. The metric does not measure what the headline claims. Because the performance reward is zero at parity with the reference implementation, a correct submission that is faster than the starter but slower than the reference scores zero. The reported 36.53 is therefore a mixture of task completion and reference-beating optimization, not a success rate; the abstract's "only 36.53%" cannot be read as the fraction of engineering tasks the models can do. For a benchmark paper, measurement validity is the primary contribution, so this is a central rather than cosmetic issue.
  2. No construct-validity evidence is provided: no human expert baseline or upper bound, no correlation with real engineering outcomes, no test-retest or task-level variance. With only 10 E2EO tasks, a single main conclusion rests on a very small sample.
  3. Task-form differences are confounded with budget and reward type. KFC allows one submission while LHI/E2EO allow up to 16 attempts with best-of selection, and reward types are mixed within forms. Consequently "long-horizon is harder" cannot be attributed to horizon length — which is also in tension with E2EO (62.94) scoring far above LHI (21.60).
  4. Isolation and anti-cheat are not validated, and are described inconsistently. §3.3 describes a network proxy pointing to an unreachable address, while §5.3 refers to soft isolation blocking pip and related URLs; telling the model not to cheat is listed as a safeguard. Since the benchmark's credibility depends on isolation strength, at least one adversarial probe (direct clone, mirror access, source recovery from installed packages) would be needed, along with detector false-positive rates and zero-score attribution.
  5. Additional limitations: sparse per-cell coverage (some cells contain a single task), non-uniform scaffolds (Codex for GPT models, Claude Code for others), no limitations section, and the promised release is not currently available.
Questions for authors

Can you report separately, for performance tasks: correctness pass rate, fraction correct but below the reference, and fraction above the reference? This would make the headline number interpretable.

What is the human expert score and variance on the 85 tasks? How large is the gap between 36.53 and 28.12 relative to task-level noise?

Can you report results at a fixed submission/time budget across the three forms, so that form differences are not confounded by 1 vs 16 attempts?

Can you document an adversarial test of the isolation mechanism, and report detector false positives and the distribution of zero scores by cause?

What is the release status of the task set and runtime, and which conclusions do you consider unsupported without it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.