- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SWE-Quality, a benchmark for evaluating the quality of code generated by software engineering agents beyond functional correctness. Built on DeepSWE v1.1, the benchmark covers 33 Python tasks and 3,402 patches generated by 26 models. The authors extract 72 static code metrics across three file-scope lenses, resulting in 216 features per patch, to characterize maintainability, extensibility, and understandability. To validate these metrics, the authors use an LLM judge to rank 290 functionally correct patches and train a learned scoring function, the SWE-Quality Aligned Index, using Extra Trees. The proposed index achieves approximately 80% agreement with the LLM judge on held-out tasks and a model-level ranking correlation of 0.95. The paper also includes comparisons with a second LLM judge, a small-scale human evaluation, and analyses of the relationship between code quality and functional correctness. The main contribution is a computationally efficient framework for evaluating code quality as an additional dimension of coding-agent performance.
Critical strengths
S1. An important evaluation dimension beyond functional correctness. The paper addresses an important limitation of existing coding-agent benchmarks, which primarily evaluate whether generated patches pass functional tests. Assessing maintainability, extensibility, and understandability is valuable, particularly as coding agents are increasingly used for repository-level software development. Separating code quality from correctness provides a potentially useful perspective on agent capabilities.
S2. A computationally efficient and reasonably systematic evaluation framework. The paper combines a broad collection of static code metrics with a learned scoring function and provides detailed descriptions of the metric definitions and evaluation protocol. The use of task-level cross-validation, multiple file-scope lenses, and comparisons with an independent LLM judge are useful methodological choices. The low computational cost also makes the approach potentially practical for large-scale coding-agent evaluation.
S3. interesting empirical insights into the relationship between code quality and correctness. The paper provides interesting empirical observations about the relationship between code quality and functional correctness across different coding agents. In particular, the results suggest that strong functional performance does not necessarily imply high code quality, and that reference implementations are not always preferred by the proposed quality evaluation. These findings provide useful insights into the limitations of correctness-based evaluation and further motivate the need for complementary code-quality metrics.
Critical weaknesses
W1. Limited validity of static code-quality metrics. The proposed benchmark relies on 72 static metrics to evaluate code quality, but it remains unclear whether these metrics can accurately capture maintainability, extensibility, and understandability in real-world software engineering. Metrics such as cognitive complexity, function length, and documentation coverage are only indirect proxies for code quality. The paper lacks sufficient evidence that improvements in these metrics correspond to better practical engineering outcomes.
W2. Insufficient reliability and representativeness of human evaluation. The human evaluation covers only eight tasks and involves four paper authors, with only one annotator per task. This setup does not allow for measuring inter-annotator agreement, and it is unclear whether the selected tasks and annotators are representative. Furthermore, the human annotators did not follow the same evaluation rubric as the LLM judge, raising concerns about the consistency and reliability of the evaluation.
W3. Insufficient consideration of repository-specific quality standards. Code quality is highly dependent on repository-specific characteristics, including architectural conventions, developer practices, and domain requirements. A patch with favorable static metrics may still violate the existing design principles or coding conventions of its repository. The proposed within-task normalization and file-scope lenses do not fully address these repository-specific differences, raising concerns about the fairness and generalizability of the resulting quality scores.
W4. Limited generalizability across programming languages and task types. The evaluation is restricted to 33 Python tasks, despite the broader scope of agentic software engineering. Different programming languages and software engineering domains may have substantially different quality requirements and coding conventions. It remains unclear whether the proposed metrics and learned scoring function can generalize beyond the evaluated Python repositories.
W5. Lack of validation against downstream maintenance outcomes. Although the paper motivates code quality through long-term maintainability and software evolution, its experiments focus on static code characteristics and agreement with LLM judgments. It does not examine whether patches receiving higher quality scores actually lead to easier subsequent modifications, fewer regressions, or lower maintenance costs. This limits the extent to which the proposed benchmark can be interpreted as measuring practical software maintainability.
Questions for authors
Q1. Human Annotation Reliability and Representativeness.
How were the eight tasks used for human evaluation selected, and how representative are they of the full benchmark in terms of repository characteristics, task difficulty, and implementation complexity? Why was each task evaluated by only one human rater? Without multiple independent annotations per task, how can the reliability of the human judgments be established?
Q2. Direct Validation of the Aligned Index.
The paper reports human-alignment results for individual static metrics and the unweighted Overall score, but not for the final Extra Trees-based Aligned Index. Could the authors directly evaluate the agreement between the learned index and human rankings, preferably using a task-held-out protocol to avoid potential calibration overlap?
Q3. Repository-Specific Quality Criteria.
How does the proposed index account for repository-specific development conventions, architectural constraints, and coding practices that may not be captured by static metrics? Could the authors investigate whether its effectiveness varies across repositories with different characteristics, or validate it on previously unseen repositories?
Q4. Sensitivity to Calibration Data and Model Selection.
How stable is the learned index when the selected passing completions or the models used for LLM-judge calibration change? In particular, could the authors evaluate whether the index systematically favors coding styles represented by the nine calibration models?
Q5. Cross-Language Generalizability.
Given that the main evaluation covers only Python tasks, what evidence supports the applicability of the proposed metrics and learned scoring function to other programming languages, where code structure, type systems, and development conventions may differ substantially?
3 comments
Final remarks
Thank you again for your review. We hope that our response satisfies your questions and concerns. While we concede that we cannot directly attribute our metric to human-perceived quality (absent a sufficiently large human study), we demonstrate that it is a good proxy for powerful LLM judges, which are widely used in many benchmarks (due to feasibility), at a fraction of the cost. Furthermore, we acknowledge that this work cannot answer questions regarding long-term maintainability and software evolution. However, we believe we introduce generalizable tools that can be used for this purpose given a sufficiently large, open source, contamination free dataset with truly long horizon model completions, which does not currently exist and is extremely resource-intensive to create.
Thank you for the additional experiments and clarifications. However, my core concern remains unresolved. After stepping back from human alignment and downstream maintainability validation, the benchmark relies entirely on LLM-as-a-judge rubric scores as its ground truth. Yet rubric-based LLM scoring for code quality is itself noisy and biased toward surface coding styles (as seen in the 65.9% three-judge agreement). Fitting static metrics to an unvalidated LLM rubric proxy does not establish genuine software engineering quality. Therefore, I will maintain my original score.
Thank you for a careful review. We have updated our draft to address your concerns.
W2 / Q1. Human evaluation
The human study is a pilot. We have further adjusted wording to reflect this fact, and no claim rests on it. A comprehensive human study would require a lot of resources and lead time, which we do not currently have, as each comparison requires significant effort from experienced developers. Looking at a very few samples, we can also estimate that human raters have much higher variance than the LLM judges, requiring many raters per task for a good estimate — again a massive undertaking.
Q2. The Aligned Index against human rankings
Due to the relatively high standard errors in the pilot human study (due to limited samples) we did not think it would be appropriate to fit a combined index, which requires additional training and validation splits. Furthermore, we no longer draw any conclusions from the pilot human study (see above).
W2. Judges from one family
We have since run two further complete rounds with judges from different families, each blind to authorship, each scoring all 290 passing completions on all 33 tasks.
Agreement over the 1,204 within-task ordered pairs. All three judges order 65.9% of pairs alike.
Cross-family agreement is lower than the same-family figure. In our updated draft we use the median of the three judges' within-task robust z scores as the alignment target to mitigate any self-preference biases. The new index fitted to the median reaches concordance 0.778 ± 0.016 against 0.797 ± 0.018 for the single judge in the paper.
W3 / Q3. Repository-specific quality standards
Every feature is standardised within task, so each comparison is between patches to the same repository at the same commit, and a patch is never scored against a different codebase. That removes between-repository differences in absolute metric level, which is why we do not think the scores are unfair across repositories.
We agree with the real issue you bring up. A violation of a repository's architectural or stylistic conventions leaves no static trace. No feature in our set can see that, and we have updated our draft to state this limitation.
Q4. Sensitivity to the calibration pool
We held out each of the nine calibration models in turn, refit the index on the remaining eight, and scored the held-out model's own patches on held-out tasks. This needs no new judging, only a restriction on which rows the index may fit. The protocol is the paper's own — the same 80/20 task splits, 300 repeats, task-clustered standard errors — with the held-out model's rows dropped from the training fold, so the index has seen neither that style nor those tasks. Every figure below is paired against the full index on identical splits.
The index does not in general depend on having seen a model. Overall concordance moves by −0.002 on average, and the model ranking is close to fixed. The ordering of the remaining eight models has Spearman ρ 1.00 in eight of the nine refits and 0.95 in the ninth, where one adjacent pair swaps. For eight of the nine models the concordance of the pairs involving the held-out model is indistinguishable from the full index, between −0.008 and +0.005 with every z below 1.3 in magnitude.
One model is an exception. Gemini 3.7 Flash is the lowest-rated candidate by the judge. Holding it out costs 0.089 ± 0.021 concordance on its own pairs (z −4.3) and 0.017 ± 0.006 overall (z −2.7), and the aligned index becomes more generous to it, predicting −0.21 in place of −0.56 (with it in the calibration pool) against a true −1.35 (judge rating).
We purposefully chose strong models for the calibration pool, as we believe these are the models most useful for practitioners, and this demonstrates that the index is robust to the calibration pool when considering the top models.
W4 / Q5. Cross-language generalisation
The evaluation is 33 Python tasks and we have no cross-language evidence. We have adjusted wording and expressed this more explicitly in our draft.