acceptodds
Under review as a conference paper at ICLR 2027

PerfDiag: An Execution-Free Benchmark for Diagnosing Performance Regressions in AI Infrastructure

Abstract

A performance regression is a commit that changes no output and makes the system worse to run: throughput down, memory up, a tail latency that doubled. In a serving or training stack it has no failing test, and reproducing it needs hardware most researchers cannot rent. The fix is small; finding it is the work. So performance benchmarks ask a tractable question instead: make this code faster, and we will time it. We ask the diagnostic one: given the commit that introduced a regression, name the mechanism from a fixed 32-label set and cite the code site that has to change; the human fix commit is the answer key, and nothing is executed. We release PerfDiag: 430 (regression-inducing, fixing) commit pairs from four production AI-infrastructure repositories, with a read-only harness, decoy negatives for bounding false positives, and a measured zero-call baseline on cause, mechanism and fix site. The task is hard. The best of six models names the cause family on about 58% of items, and the two remedies the task invites do not separate: no prompt scaffold among twenty separates from a single flat call on the one model we swept, and a model-authored three-layer diagnostic decision graph walked one layer at a time is worse than one on three of the five models we could run it on and on neither of the two weakest. §4 reports the resolution of that sweep: 10.2 to 17.9 accuracy points. The mechanism keys on 94% of the scored items were written by Claude Opus 5, which ranks first, so we measure how reproducible they are and not that they are correct. A key it had no part in writing leaves the ranking unchanged. Every number is on train and dev; no run has scored the test split, though an author saw the gold label of 11 of its items during the label audit (Appendix I).

Primary area:
datasets and benchmarks
Keywords:
benchmark, performance regression, root-cause diagnosis, AI infrastructure, execution-free evaluation, commit-level diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bBpKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces PerfDiag, a benchmark for diagnosing performance regressions in AI infrastructure. PerfDiag contains 430 commit pairs collected from four open-source projects (vLLM, Megatron-LM, DeepSpeed, and TransformerEngine). Given only the inducing commit diff, a model must choose one of 32 regression mechanisms (grouped into 11 cause families) and cite the code site that needs to be fixed. The answers are scored against labels derived from the later human fix, and no code is executed. The authors evaluate six LLMs, 20 prompt variants on Claude Sonnet 4.5, and a model-authored three-layer decision graph. They report that the best model (Claude Opus 5) identifies the cause family on about 58% of items, that no prompt scaffold significantly improves on a single flat call, and that the decision graph is limited by routing errors.

Critical strengths
  • The paper targets a practical problem. Performance regressions in AI infrastructure are costly and hard to reproduce because reproducing them often requires multi-GPU nodes and specific workloads.
  • The benchmark is built from real commit histories of four widely used open-source AI infrastructure projects rather than from synthetic bugs, providing a realistic benchmark for an important task.
Critical weaknesses
  • Circular labels and self-preference bias. Claude Opus 5 wrote about 94% of the scored mechanism keys (324 of 344 items) and produced the 32-label taxonomy. The same model then ranks first in experiments, which may be a result of self-preference bias. Human verification covers only 19 items checked by a single author in a non-blind way. Since the main contribution of the paper is a benchmark, the validity of its labels is decisive: if the ground truth mainly reflects one model's view, the leaderboard and the reported accuracies cannot be interpreted as real diagnostic ability.

  • Flawed experimental setup that undermines the main conclusions. 50 of the 136 evaluation cells fall back to a 4,096-token output cap, which truncates 50% to 52% of the calls of the two strongest models. In addition, the no significant prompt improvement claim relies on one model and 67 items, and whether the mechanism is identifiable from the inducing diff is never measured. Together, these issues mean that both the model ranking and the main findings may be artifacts of configuration and sample size rather than properties of the task.

Questions for authors
  1. How well do independent human experts agree with the labels written by Claude Opus 5? If a blind, multi-expert annotation of a subset is available (or can be done), does the model ranking change when models are scored against it?
  2. How are items assigned to tier A and tier B in practice, and who inferred the 69 tier-B links "by reading both diffs", following what guideline?
  3. Do the model ranking and the main findings remain the same when all cells are re-run under the same output cap?
  4. What fraction of items can actually be diagnosed from the inducing diff alone (e.g., by human experts), especially among the 24% of items whose diffs are truncated?
  5. Suggestions: Please build or validate the taxonomy with domain experts, release labels from multiple blind annotators, and provide inter-annotator agreement (e.g., Cohen's kappa).
  6. Suggestions: Extend the prompt study to more models before drawing conclusions.
Reviewer_NCLaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces PerfDiag, a benchmark that contains 430 regression inducing and fixing commit pairs from four AI-infrastructure repositories. This benchmark is aimed for assessing whether models can identify the regression mechanism. The result shows that the best model reaches approximately 58% diagnostic accuracy. The authors also conducted prompt scaffolding and a model-authored, three-layer decision graph, which provide no consistent improvement. The paper also analyses of benchmark validity, including memorization and label circularity.

Critical strengths
  • The paper points to a practical problem regarding performance regressions in AI infrastructure, where diagnosing it without requiring expensive execution environments is important.

  • The benchmark uses real commit pairs from different repositories and addresses various aspects of diagnosis.

Critical weaknesses
  • There is no clear research questions and a coherent structure in the paper. Benchmark construction, task definitions, experimental results, and limitations are mixed together. The introduction does not clearly connect the prompt, routing, and validity experiments, and it ends without a discussion or conclusion that should summarize its findings.

  • A risk might be that the evaluation may favor Claude Opus 5 because it created labels and was then evaluated against those same ones. It also ranks first, so stronger diagnosis or better agreement with its own labels cannot be differentiated.

  • Several other problems: All reported scores use train/dev but no test set. Some test items repeat inputs from the development set, and some runs accidentally used shorter output limits.

Questions for authors
  • Can you provide results using labels that were not created by any evaluated model?

  • Why are model results reported only on train/dev, and how will the issues with the test set be addressed?

Reviewer_mAZTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an execution-free benchmark that evaluates whether a model can, given only the commit that introduced a performance regression, diagnose the regression's mechanism and locate the code that needs to change. The data consist of several hundred commit pairs from four AI-infrastructure repositories such as vLLM and Megatron-LM; the taxonomy and most of the answer keys were generated by Claude Opus 5. Evaluation of six models shows generally low diagnostic accuracy; the authors also test a series of prompt and reasoning-style variants and voluntarily disclose several limitations of the labels and protocol.

Critical strengths
  1. The topic is real and important. "Diagnosing performance regressions from a commit without executing any code" is a practically needed setting that execution-based benchmarks cannot cover.
  2. Unusually thorough disclosure. The label provenance and various protocol limitations are documented (although it is precisely these disclosures that expose the design problems below).
Critical weaknesses
  1. The benchmark's validity foundation does not hold: the answer keys were written by the model that ranks first, with no independent verification. The taxonomy, the diagnostic decision tree, and roughly 94% of the scored labels all come from Claude Opus 5, which happens to rank first. The only human correctness check is one author non-blindly reviewing 19 items, with no deliberately wrong labels planted to test whether such checking could detect problems. The authors themselves acknowledge that a score "certifies agreement with a machine-written naming convention" rather than diagnostic correctness. More fundamentally: all scores are measured on the train+dev data from which the taxonomy was induced; the test split was never run, and the strongest model's exact-mechanism accuracy already drops from 0.552 to 0.417 on the held-out portion; for 62% of items, the "regression-inducing commit" is the authors' inference, sanity-checked only by LLMs. A benchmark whose answer correctness cannot be independently confirmed provides no basis for its leaderboard or any downstream conclusions.

  2. The overall presentation resembles the product of an automated research pipeline: the narrative is disjointed and the claims are decoupled from the evidence. The main text claims "no significant prompt improvement," yet that experiment uses a single model and 67 items, and its statistical sensitivity floor alone could miss true differences of more than 10 points; the main text and the appendix also give different answers to how many variants are significantly worse. The only significant result for "layer-by-layer decision is worse" disappears when rerun at a different output budget. The "about 2x improvement when given the correct family" comparison simultaneously changes the number of options, which the authors themselves acknowledge makes it an upper bound rather than a measurement. The task design is equally awkward: the confirm/refute field, listed as one third of the task, is never scored; gold and predicted locations use two different coordinate systems; and for 19/295 items the gold family is unreachable from the labeled symptom on the authors' own decision tree. Throughout the paper, why the questions matter and what the findings support is never coherently narrated—it feels like the research log was submitted as-is rather than organized into a paper.

Questions for authors

Clarifications needed

  1. Can you have domain experts outside the project blindly label a subset (about 100 items) under the same inputs as the models, with known wrong labels planted to calibrate checking sensitivity? Is the model ranking on that subset consistent with the main table?
  2. Please report results on the test split (or a re-split held-out set) to show whether the taxonomy overfits the data it was induced from.

Suggestions

  • Rewrite the main text: keep a single clear line of claims, move the revision history to release notes, and state explicitly what C2–C5 each claim and what evidence supports each.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.