- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper creates a methodology and broader framework for evaluating the maintainability of agent-generated code in multiple facets: Analysability, Reusability, Testability, and Static Modularity (a baseline). Several producer models reimplement programs from a reference; this stage often contains errors which are expected. These implementations are frozen, and subsequently maintainer models complete each of the three downstream. Three models are chosen as maintainers, and none of the maintainers work on their own code. Findings suggest that depending on maintainability criterion, rankings for producers would vary. State cohesion (LCOM) also has little correlation with the downstream metrics. The correlations between source features and task outcomes are also probed.
Critical strengths
S1. Maintainability is evaluated in a multifaceted way through three demands. Each demand corresponds to an executable task and is reported separately. The low rank correlations across tasks suggests that depending on the criteria of maintainability, producer models can perform differently.
S2. The general framework can be reused to derive maintainability tests from existing codebases with test cases. The test cases are adapted from ProgramBench and the methodology augments them into three downstream tasks for evaluating maintainability.
Critical weaknesses
W1. Lack of a clear central finding and low statistical power. While the protocol is sound, the results and figures center around differences and variability rather than a cohesive unifying finding. The finding on tasks and how it affects producer rankings could be strengthened by repeat runs and bootstrap intervals on tau to establish the reliability of these differences. The stability of Reusability rankings is also based upon a partially statistically insignificant p-value (0.068) in the best-case scenario. Figure 3 further shows weak correlations between state cohesion and Analyzability and the three other outcomes. It would be helpful for the authors to clarify the substantiality of the findings despite their heterogeneity and limited statistical significance and to provide higher-power analyses if possible.
Questions for authors
Q1. Is the disagreement across tasks and maintainers statistically significant, considering the modest differences? It would be helpful to show confidence intervals for tau values (for instance, bootstrapping).
Q2. Considering the potential risk of biases in either case of including or excluding the same model as a maintainer of itself, why was it chosen to exclude? Could you provide a justification or comparison results if the self model was not excluded?
Q3. Were the downstream tasks for Reusability, Analysability, and Testability written by humans or AI? Was there any human validation in the process of selecting or validating these tasks?
Q4: How should human practitioners use the benchmark or methodology to make critical decisions about which producer to use?
6 comments
Responses to Questions 2–4
Question 4: How should human practitioners use the benchmark or methodology to make critical decisions about which producer to use?
Thank you for your thoughtful question. Section 5 explains how to select producers for the intended maintenance demand and maintainer profile.
For source understanding and behavioral analysis, practitioners can prioritize Analysability score. For building new consumers, source reuse share and construction tokens capture reuse extent and cost; for test construction, practitioners can compare Testability construction tokens. Multi-agent software engineering illustrates these demands: MAGIS assigns code localization to a Repository Custodian (Tao et al., 2024), while MetaGPT and AgentCoder assign test construction to specialized agents (Hong et al., 2024; Huang et al., 2024).
Practitioners should use the panel matching their downstream maintainer and examine cross-maintainer agreement to assess whether producer rankings transfer. Lower agreement motivates reevaluating producer choices when the maintainer changes, particularly for Analysability and Testability.
Practitioners can also apply the same protocol in their deployment environments, using their actual maintainer profiles, tasks, and budgets to inform producer selection.
Question 2: Considering the potential risk of biases in either case of including or excluding the same model as a maintainer of itself, why was it chosen to exclude? Could you provide a justification or comparison results if the self model was not excluded?
Thank you for your comments. CarryBench evaluates maintainability through the downstream performance of fixed maintainer models. Different producers may generate code with distinct naming conventions, source organization, and implementation patterns. A model’s own outputs may align more closely with these patterns, introducing an additional distributional alignment factor. We therefore exclude producer–maintainer identity pairs to reduce its influence on maintainability comparisons. Cross-maintainer comparisons retain the seven producers evaluated by both profiles, ensuring a consistent candidate set.
As requested, we also report the excluded self-maintenance results:
| Producer–maintainer pair | Analysability score (%) ↑ | Source reuse share (%) ↑ | Reusability construction tokens (× profile median) ↓ | Testability construction tokens (× profile median) ↓ |
|---|---|---|---|---|
| DeepSeek V4 Flash | 92.15 (2/9) | 75.04 (9/9) | 0.92 (4/9) | 0.83 (2/9) |
| HY3 | 88.62 (5/9) | 77.82 (6/9) | 1.47 (8/9) | 0.84 (3/9) |
| Kimi K2.7 Code | 94.46 (1/9) | 82.77 (2/9) | 1.72 (9/9) | 1.01 (5/9) |
Parentheses indicate ranks among all nine producers under the corresponding maintainer, with 1 being best. Construction tokens use the same normalization denominators as Figure 2: the median producer-level cost among the eight non-self producers in each maintainer profile.
Question 3: Were the downstream tasks for Reusability, Analysability, and Testability written by humans or AI? Was there any human validation in the process of selecting or validating these tasks?
Thank you for your questions. As disclosed in our AI use statement, AI assisted with benchmark workflow development and downstream task preparation, and all AI-assisted work was reviewed by the authors.
For Analysability, questions were automatically derived from common-pass generation tests using predefined sampling rules, rather than generated by AI. As described in Section 3.3, the original behavioral oracle provides the expected outputs and grading criteria.
For Reusability and Testability, the authors defined the task objectives, behavioral scenarios, and evaluation requirements. AI assisted in preparing the task prompts and supporting materials, which were subsequently reviewed and corrected by the authors before evaluation.
References
Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution. In Advances in Neural Information Processing Systems, 2024.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework. In International Conference on Learning Representations, 2024.
Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation. arXiv preprint arXiv:2312.13010, 2024.
I would like to thank the authors for taking the time to respond to the weaknesses and questions I raised, and for clarifying a central finding. My concerns about Finding 1 are sufficiently addressed. I still have qualms about Finding 2 due to overlapping confidence intervals and the strong comparative claim about Reusability rankings. Q1 has been addressed by the paired bootstrap confidence intervals. The justification for excluding self-maintenance tasks is reasonable. I appreciate the clarification on Q3 and nuanced response to Q4. Upon further thought, I think the heterogeneity of the findings might not be a weakness, as maintainability should be evaluated in multiple dimensions and could depend on what models one uses to evaluate it. I would like to raise my overall rating to a 3.
Responses to Weaknesses 1 and Question 1
Dear Reviewer ATVR,
Thank you for your comments and questions.
Thank you for your sincere comments. Our central finding is that maintainability should be evaluated separately for each maintenance demand under a specified maintainer profile. Finding 1 supports separating demands because producers rank differently for understanding, reusing, and testing code. Finding 2 supports accounting for the maintainer because these rankings do not transfer equally well across maintainers. Finding 4 further motivates demand-specific optimization research, as the source characteristics associated with performance vary across demands and maintainers. We discuss the practical implications under Question 4 and will clarify this central finding in the revision.
We repeated Stage II evaluations twice using DeepSeek V4 Flash across all cases and demands, yielding three measurements per unit. Stage I artifacts, tasks, prompts, tools, and budgets were fixed.
We report single-measurement absolute-agreement ICC and between-run Kendall's τ for producer rankings:
The high ICC values (0.9799–0.9977) and between-run ranking τ values (0.750–1.000) indicate that producer rankings are relatively stable within each outcome, despite the low cross-demand agreement reported in Finding 1. We further examine uncertainty in the original cross-demand and cross-maintainer comparisons through paired case bootstrap under Question 1.
Regarding Finding 2, all three cross-maintainer τ values for source reuse share are positive, although one comparison has p = 0.068. Our paired case bootstrap further yields positive confidence intervals for the mean cross-maintainer agreement in both source reuse share (0.46, 95% CI [0.12, 0.71]) and Reusability construction tokens (0.49, 95% CI [0.17, 0.71]). These results support relatively stronger cross-maintainer agreement for Reusability without implying statistical significance for every maintainer pair.
Figure 3 further reinforces the importance of downstream evaluation. Its weak, mixed-direction associations between static cohesion and maintenance outcomes suggest that this measure alone is insufficient to characterize maintainability across different demands.
Thank you for your question. We bootstrapped original cases (2,000 paired resamples), applying common case weights across producers and the outcomes or maintainer profiles being compared, recomputing producer means and Kendall’s τa (tokens ranked low to high). We report percentile 95% CIs.
For Finding 1, each resample averages the five cross-demand outcome-pair τa values within a maintainer:
Cross-demand mean τa is near zero, and CI upper bounds remain at most 0.23; together with high repeatability, this supports demand-specific producer selection.
For Finding 2, each resample averages τa for the same outcome across the three maintainer pairs (Table A3):
Both Reusability outcomes have 95% CIs entirely above zero.