- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLMs can engineer and optimize the infrastructure that powers them. The authors argue that existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore cannot evaluate the long-horizon and open-ended nature of real-world LLM infrastructure engineering. To address this gap, they build Φ-Bench, which contains 85 tasks in three formats: 55 Kernel Function Completion (KFC) tasks, 20 Long-Horizon Implementation (LHI) tasks, and 10 repository-wide End-to-End Optimization (E2EO) tasks. They analyze 2,260 papers and 1,852 repositories to construct a three-level taxonomy consisting of 9 domains, 62 topics, and 410 tags. The benchmark covers all 9 domains and 52 of the 62 topics. Guided by this taxonomy, the tasks are constructed through PR/issue rollback, curated code masking, and expert curation. Each task has either a functional implementation objective or a performance optimization objective. All task materials, including test cases, are visible to the evaluated models except the evaluation harness. During task validation, a performance task is retained only when its reference solution achieves at least 1.1 times the baseline and the starter or no-op solution does not outperform the baseline. For a functional implementation task, the reference solution must pass all tests, while the starter solution must fail at least one. Functional tasks receive a binary score of 0 or 1, whereas performance tasks receive a log-normalized score between 0 and 1. Both metrics are gated by checks for buildability, functional correctness, edit constraints, and reward hacking. The authors evaluate eight frontier models on Φ-Bench. Claude Opus 5 performs best but achieves an overall score of only 36.53%, and every evaluated model scores lower on LHI than on KFC. The trajectory analysis, which the authors acknowledge is limited to one E2EO task, suggests that stronger models use inexpensive local validation, control variables and measurement noise, and check possible confounding factors more carefully.
Critical strengths
-
The first critical strength is that Φ-Bench extends evaluation from local kernels and pre-specified optimization targets to repository-level infrastructure engineering. Unlike ISO-Bench and CUDAHercules, which typically identify the target component or performance bottleneck in advance, Φ-Bench leaves bottleneck identification and the choice of optimization strategy to the agent, particularly in E2EO tasks. This extension is supported by more than the task formats themselves. The taxonomy is built from 2,260 papers and 1,852 repositories; the benchmark covers 52 of the 62 topics; and the tasks are constructed through PR/issue rollback, curated masking of repository code, and expert curation of research problems. Together, these provide checkable evidence that Φ-Bench expands the scope and open-endedness of evaluation rather than merely increasing the number of tasks. This supports acceptance because the benchmark targets realistic infrastructure-engineering abilities that are largely excluded by evaluations centered on local code completion or predefined bottlenecks.
-
The second critical strength is the concrete and executable evaluation protocol. As discussed under Informativeness, it combines visible task materials and test cases with a hidden evaluation harness, objective-specific task-validation criteria, a single formal submission for KFC, and both rule-based and agent-based anti-hacking checks. The rule-based monitor flagged three requests, while the proctor agent found no confirmed hacking. These safeguards matter because infrastructure-performance evaluations can be distorted by incorrect implementations, measurement loopholes, or reward hacking. Without them, a higher score may not reflect stronger engineering ability. Although the protocol cannot address every validity concern, it ensures that positive rewards are gated by checks for buildability, functional correctness, edit constraints, and reward hacking, and provides a reusable basis for future benchmarks.
Critical weaknesses
-
The first critical weakness is that score differences across task formats cannot be clearly interpreted as differences in the intrinsic difficulty of the formats. As discussed under Soundness, the reward type follows the task objective rather than the format, and the paper does not report the objective composition within each format. Although both rewards are bounded in , a binary pass/fail outcome and a continuous reward that increases after exceeding the reference solution have different interpretations. Therefore, while Table 2 shows that every model scores lower on LHI than on KFC, this difference cannot be directly attributed to the intrinsic difficulty of LHI because it may also be affected by objective composition and reward type. This issue is consequential because both the abstract and conclusion emphasize that models perform worse on long-horizon, repository-level implementation than on kernel-level tasks. The authors could address it by reporting the format-by-objective task composition, providing results stratified by objective, and conducting sensitivity analyses under unified or alternative scoring schemes.
-
The second critical weakness is that the cross-model rankings and behavioral analysis lack sufficient comparability controls, uncertainty quantification, and breadth of evidence. First, GPT-5.6 Sol is evaluated with the Codex scaffold, whereas the other seven models use Claude Code. The paper does not explain whether this choice is constrained by model–scaffold compatibility or assess how scaffold differences may affect the rankings in Table 1. Second, the introduction claims “low run-to-run variability,” but the paper does not report the number of repeated runs, repeated-run means, variances, or confidence intervals. Finally, the trajectory conclusions about inexpensive local validation, variable control, and cautious attribution are based on only three models performing a single E2EO task, and the analysis is mainly qualitative. These limitations do not invalidate the scores observed in Table 1, but they mean that the cross-model rankings and the behavioral conclusion that experimental discipline distinguishes stronger from weaker models should be treated as preliminary. The scaffold and variability concerns could be mitigated by explaining the scaffold-selection constraints, comparing the same model under different scaffolds, and reporting repeated-run statistics. The trajectory conclusions require validation across more tasks and models.
Questions for authors
-
Sections 3.1 and 3.5 state that the reward type is determined by the task objective rather than by whether the task is KFC, LHI, or E2EO. Please report the numbers of functional implementation and performance optimization tasks within each format, together with model results stratified by objective. Given that binary and continuous rewards have different interpretations, to what extent do the format-level averages in Table 2 reflect differences in the intrinsic difficulty of the task formats? Does the observed LHI–KFC performance gap remain when results are stratified by objective or aggregated using another clearly specified method? In addition, what motivates the 1.1× solvability threshold in §3.3, and are the conclusions robust to other reasonable thresholds? These results would directly affect whether I can interpret the lower scores on LHI as evidence that LHI is intrinsically more difficult than KFC.
-
In Table 1, GPT-5.6 Sol is evaluated with the Codex scaffold, whereas the other seven models use Claude Code. Please clarify whether this choice is constrained by model–scaffold compatibility and provide any available evidence of comparability, such as results for the same model under both scaffolds. If such evidence is unavailable, please state the scope and limitations of the cross-model rankings. The paper also claims “low run-to-run variability”; please report the number of repeated runs and the corresponding means, variances, or confidence intervals. Finally, Figure 4 shows that some category–format cells contain only one or two tasks. Please therefore quantify the uncertainty of the category-level results in Table 1, for example by providing task-level results, bootstrap intervals, leave-one-task-out analyses, or another analysis that reflects sensitivity to task sampling.
-
The paper uses network isolation, rule-based monitoring, and a proctor agent to prevent external solution retrieval or manipulation of the evaluation during task execution, but these mechanisms cannot rule out parametric memorization. Some tasks are reconstructed from PRs, issues, and code in public repositories, so the corresponding implementations may have appeared in the models’ pretraining data. Did the authors apply a temporal split, select tasks according to model knowledge cutoffs, rewrite the tasks, or evaluate on held-out variants to assess potential contamination? If not, how should the best model’s score of 36.53% be interpreted, and to what extent does it reflect the ability to solve previously unseen infrastructure-engineering problems?