- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces PerfDiag, a benchmark for diagnosing performance regressions in AI infrastructure. PerfDiag contains 430 commit pairs collected from four open-source projects (vLLM, Megatron-LM, DeepSpeed, and TransformerEngine). Given only the inducing commit diff, a model must choose one of 32 regression mechanisms (grouped into 11 cause families) and cite the code site that needs to be fixed. The answers are scored against labels derived from the later human fix, and no code is executed. The authors evaluate six LLMs, 20 prompt variants on Claude Sonnet 4.5, and a model-authored three-layer decision graph. They report that the best model (Claude Opus 5) identifies the cause family on about 58% of items, that no prompt scaffold significantly improves on a single flat call, and that the decision graph is limited by routing errors.
Critical strengths
- The paper targets a practical problem. Performance regressions in AI infrastructure are costly and hard to reproduce because reproducing them often requires multi-GPU nodes and specific workloads.
- The benchmark is built from real commit histories of four widely used open-source AI infrastructure projects rather than from synthetic bugs, providing a realistic benchmark for an important task.
Critical weaknesses
-
Circular labels and self-preference bias. Claude Opus 5 wrote about 94% of the scored mechanism keys (324 of 344 items) and produced the 32-label taxonomy. The same model then ranks first in experiments, which may be a result of self-preference bias. Human verification covers only 19 items checked by a single author in a non-blind way. Since the main contribution of the paper is a benchmark, the validity of its labels is decisive: if the ground truth mainly reflects one model's view, the leaderboard and the reported accuracies cannot be interpreted as real diagnostic ability.
-
Flawed experimental setup that undermines the main conclusions. 50 of the 136 evaluation cells fall back to a 4,096-token output cap, which truncates 50% to 52% of the calls of the two strongest models. In addition, the
no significant prompt improvementclaim relies on one model and 67 items, and whether the mechanism is identifiable from the inducing diff is never measured. Together, these issues mean that both the model ranking and the main findings may be artifacts of configuration and sample size rather than properties of the task.
Questions for authors
- How well do independent human experts agree with the labels written by Claude Opus 5? If a blind, multi-expert annotation of a subset is available (or can be done), does the model ranking change when models are scored against it?
- How are items assigned to tier A and tier B in practice, and who inferred the 69 tier-B links "by reading both diffs", following what guideline?
- Do the model ranking and the main findings remain the same when all cells are re-run under the same output cap?
- What fraction of items can actually be diagnosed from the inducing diff alone (e.g., by human experts), especially among the 24% of items whose diffs are truncated?
- Suggestions: Please build or validate the taxonomy with domain experts, release labels from multiple blind annotators, and provide inter-annotator agreement (e.g., Cohen's kappa).
- Suggestions: Extend the prompt study to more models before drawing conclusions.