acceptodds
Under review as a conference paper at ICLR 2027

CarryBench: A Multi-Role, Multi-Maintainer Benchmark for the Maintainability of Agent-Generated Code

Abstract

Agent-generated code is often the starting point for later work. Functional correctness tests show whether the current task is done. They do not show how difficult the next task will be. Existing maintainability benchmarks usually measure one downstream activity, and most let an agent work on its own output, so the score mixes the artifact with the worker. CarryBench separates the two. Nine producer models write repository-scale programs for 13 cases in Python, Go, Rust, and C++, and frozen outputs are evaluated on three maintenance demands. Each demand is a targeted downstream task under three fixed maintainer profiles. An observer predicts behavior from source for Analysability, a consumer builder constructs a new use for Reusability, and a tester writes a black-box test suite for Testability. Static Modularity is measured on the same source as a traditional proxy. Rankings agree most across maintainers for Reusability, and less for Analysability and Testability. Different downstream outcomes produce different producer orderings. Systematic within-case analysis shows that higher state cohesion need not coincide with easier downstream work. Ablations of prespecified source-feature blocks find conditional associations with lexical organization, observable entry-to-result paths, and behavior localization. Those associations differ across maintenance demands and maintainer profiles. CarryBench treats maintainability as the difficulty a fixed downstream worker experiences on a specified next task over a frozen artifact. It reports that difficulty for each maintenance demand and each maintainer profile, from the executable tasks and the source-level associations, without combining those results into a universal score.

Primary area:
datasets and benchmarks
Keywords:
coding, agent, maintainability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ATVROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper creates a methodology and broader framework for evaluating the maintainability of agent-generated code in multiple facets: Analysability, Reusability, Testability, and Static Modularity (a baseline). Several producer models reimplement programs from a reference; this stage often contains errors which are expected. These implementations are frozen, and subsequently maintainer models complete each of the three downstream. Three models are chosen as maintainers, and none of the maintainers work on their own code. Findings suggest that depending on maintainability criterion, rankings for producers would vary. State cohesion (LCOM) also has little correlation with the downstream metrics. The correlations between source features and task outcomes are also probed.

Critical strengths

S1. Maintainability is evaluated in a multifaceted way through three demands. Each demand corresponds to an executable task and is reported separately. The low rank correlations across tasks suggests that depending on the criteria of maintainability, producer models can perform differently.

S2. The general framework can be reused to derive maintainability tests from existing codebases with test cases. The test cases are adapted from ProgramBench and the methodology augments them into three downstream tasks for evaluating maintainability.

Critical weaknesses

W1. Lack of a clear central finding and low statistical power. While the protocol is sound, the results and figures center around differences and variability rather than a cohesive unifying finding. The finding on tasks and how it affects producer rankings could be strengthened by repeat runs and bootstrap intervals on tau to establish the reliability of these differences. The stability of Reusability rankings is also based upon a partially statistically insignificant p-value (0.068) in the best-case scenario. Figure 3 further shows weak correlations between state cohesion and Analyzability and the three other outcomes. It would be helpful for the authors to clarify the substantiality of the findings despite their heterogeneity and limited statistical significance and to provide higher-power analyses if possible.

Questions for authors

Q1. Is the disagreement across tasks and maintainers statistically significant, considering the modest differences? It would be helpful to show confidence intervals for tau values (for instance, bootstrapping).

Q2. Considering the potential risk of biases in either case of including or excluding the same model as a maintainer of itself, why was it chosen to exclude? Could you provide a justification or comparison results if the self model was not excluded?

Q3. Were the downstream tasks for Reusability, Analysability, and Testability written by humans or AI? Was there any human validation in the process of selecting or validating these tasks?

Q4: How should human practitioners use the benchmark or methodology to make critical decisions about which producer to use?

6 comments
AuthorsOpenReview ↗

Responses to Weaknesses 1 and Question 1

Dear Reviewer ATVR,

Thank you for your comments and questions.

Weakness 1: Lack of a clear central finding and low statistical power. While the protocol is sound, the results and figures center around differences and variability rather than a cohesive unifying finding. The finding on tasks and how it affects producer rankings could be strengthened by repeat runs and bootstrap intervals on tau to establish the reliability of these differences. The stability of Reusability rankings is also based upon a partially statistically insignificant p-value (0.068) in the best-case scenario. Figure 3 further shows weak correlations between state cohesion and Analysability and the three other outcomes. It would be helpful for the authors to clarify the substantiality of the findings despite their heterogeneity and limited statistical significance and to provide higher-power analyses if possible.

Thank you for your sincere comments. Our central finding is that maintainability should be evaluated separately for each maintenance demand under a specified maintainer profile. Finding 1 supports separating demands because producers rank differently for understanding, reusing, and testing code. Finding 2 supports accounting for the maintainer because these rankings do not transfer equally well across maintainers. Finding 4 further motivates demand-specific optimization research, as the source characteristics associated with performance vary across demands and maintainers. We discuss the practical implications under Question 4 and will clarify this central finding in the revision.

We repeated Stage II evaluations twice using DeepSeek V4 Flash across all cases and demands, yielding three measurements per unit. Stage I artifacts, tasks, prompts, tools, and budgets were fixed.

We report single-measurement absolute-agreement ICC and between-run Kendall's τ for producer rankings:

OutcomeICC(A,1)Original vs. repeat 1 τOriginal vs. repeat 2 τRepeat 1 vs. repeat 2 τ
Analysability score0.98820.8210.7500.786
Source reuse share0.97990.8570.7860.929
Reusability construction tokens0.99771.0001.0001.000
Testability construction tokens0.99680.9290.9290.857

The high ICC values (0.9799–0.9977) and between-run ranking τ values (0.750–1.000) indicate that producer rankings are relatively stable within each outcome, despite the low cross-demand agreement reported in Finding 1. We further examine uncertainty in the original cross-demand and cross-maintainer comparisons through paired case bootstrap under Question 1.

Regarding Finding 2, all three cross-maintainer τ values for source reuse share are positive, although one comparison has p = 0.068. Our paired case bootstrap further yields positive confidence intervals for the mean cross-maintainer agreement in both source reuse share (0.46, 95% CI [0.12, 0.71]) and Reusability construction tokens (0.49, 95% CI [0.17, 0.71]). These results support relatively stronger cross-maintainer agreement for Reusability without implying statistical significance for every maintainer pair.

Figure 3 further reinforces the importance of downstream evaluation. Its weak, mixed-direction associations between static cohesion and maintenance outcomes suggest that this measure alone is insufficient to characterize maintainability across different demands.

Question 1: Is the disagreement across tasks and maintainers statistically significant, considering the modest differences? It would be helpful to show confidence intervals for tau values (for instance, bootstrapping).

Thank you for your question. We bootstrapped original cases (2,000 paired resamples), applying common case weights across producers and the outcomes or maintainer profiles being compared, recomputing producer means and Kendall’s τa (tokens ranked low to high). We report percentile 95% CIs.

For Finding 1, each resample averages the five cross-demand outcome-pair τa values within a maintainer:

Maintainer profileMean τa95% confidence interval
DeepSeek V4 Flash+0.04[−0.11, +0.23]
HY3−0.02[−0.14, +0.15]
Kimi K2.7 Code−0.07[−0.24, +0.04]

Cross-demand mean τa is near zero, and CI upper bounds remain at most 0.23; together with high repeatability, this supports demand-specific producer selection.

For Finding 2, each resample averages τa for the same outcome across the three maintainer pairs (Table A3):

OutcomeCross-maintainer mean τa95% confidence interval
Analysability score+0.16[−0.17, +0.41]
Source reuse share+0.46[+0.12, +0.71]
Reusability construction tokens+0.49[+0.17, +0.71]
Testability construction tokens−0.02[−0.21, +0.21]

Both Reusability outcomes have 95% CIs entirely above zero.

AuthorsOpenReview ↗

Responses to Questions 2–4

Question 4: How should human practitioners use the benchmark or methodology to make critical decisions about which producer to use?

Thank you for your thoughtful question. Section 5 explains how to select producers for the intended maintenance demand and maintainer profile.

For source understanding and behavioral analysis, practitioners can prioritize Analysability score. For building new consumers, source reuse share and construction tokens capture reuse extent and cost; for test construction, practitioners can compare Testability construction tokens. Multi-agent software engineering illustrates these demands: MAGIS assigns code localization to a Repository Custodian (Tao et al., 2024), while MetaGPT and AgentCoder assign test construction to specialized agents (Hong et al., 2024; Huang et al., 2024).

Practitioners should use the panel matching their downstream maintainer and examine cross-maintainer agreement to assess whether producer rankings transfer. Lower agreement motivates reevaluating producer choices when the maintainer changes, particularly for Analysability and Testability.

Practitioners can also apply the same protocol in their deployment environments, using their actual maintainer profiles, tasks, and budgets to inform producer selection.

Question 2: Considering the potential risk of biases in either case of including or excluding the same model as a maintainer of itself, why was it chosen to exclude? Could you provide a justification or comparison results if the self model was not excluded?

Thank you for your comments. CarryBench evaluates maintainability through the downstream performance of fixed maintainer models. Different producers may generate code with distinct naming conventions, source organization, and implementation patterns. A model’s own outputs may align more closely with these patterns, introducing an additional distributional alignment factor. We therefore exclude producer–maintainer identity pairs to reduce its influence on maintainability comparisons. Cross-maintainer comparisons retain the seven producers evaluated by both profiles, ensuring a consistent candidate set.

As requested, we also report the excluded self-maintenance results:

Producer–maintainer pairAnalysability score (%) ↑Source reuse share (%) ↑Reusability construction tokens (× profile median) ↓Testability construction tokens (× profile median) ↓
DeepSeek V4 Flash92.15 (2/9)75.04 (9/9)0.92 (4/9)0.83 (2/9)
HY388.62 (5/9)77.82 (6/9)1.47 (8/9)0.84 (3/9)
Kimi K2.7 Code94.46 (1/9)82.77 (2/9)1.72 (9/9)1.01 (5/9)

Parentheses indicate ranks among all nine producers under the corresponding maintainer, with 1 being best. Construction tokens use the same normalization denominators as Figure 2: the median producer-level cost among the eight non-self producers in each maintainer profile.

Question 3: Were the downstream tasks for Reusability, Analysability, and Testability written by humans or AI? Was there any human validation in the process of selecting or validating these tasks?

Thank you for your questions. As disclosed in our AI use statement, AI assisted with benchmark workflow development and downstream task preparation, and all AI-assisted work was reviewed by the authors.

For Analysability, questions were automatically derived from common-pass generation tests using predefined sampling rules, rather than generated by AI. As described in Section 3.3, the original behavioral oracle provides the expected outputs and grading criteria.

For Reusability and Testability, the authors defined the task objectives, behavioral scenarios, and evaluation requirements. AI assisted in preparing the task prompts and supporting materials, which were subsequently reviewed and corrected by the authors before evaluation.

References

Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution. In Advances in Neural Information Processing Systems, 2024.

Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework. In International Conference on Learning Representations, 2024.

Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation. arXiv preprint arXiv:2312.13010, 2024.

I would like to thank the authors for taking the time to respond to the weaknesses and questions I raised, and for clarifying a central finding. My concerns about Finding 1 are sufficiently addressed. I still have qualms about Finding 2 due to overlapping confidence intervals and the strong comparative claim about Reusability rankings. Q1 has been addressed by the paired bootstrap confidence intervals. The justification for excluding self-maintenance tasks is reasonable. I appreciate the clarification on Q3 and nuanced response to Q4. Upon further thought, I think the heterogeneity of the findings might not be a weakness, as maintainability should be evaluated in multiple dimensions and could depend on what models one uses to evaluate it. I would like to raise my overall rating to a 3.

Reviewer_vGFmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CarryBench, a benchmark for evaluating the maintainability of agent-generated code. It separates code generation from downstream maintenance and evaluates three aspects: analysability, reusability, and testability. The experiments cover nine producer models, 13 programming tasks, and three fixed maintainer models.

Critical strengths
  1. Interesting evaluation perspective. Measuring how easily other agents can understand, reuse, and test generated code is a meaningful step beyond functional correctness.

  2. Thoughtful experimental design. Separating producers from maintainers helps reduce confounding factors. Using multiple maintainer models also makes the results more

Critical weaknesses
  1. Testability covering just seven. Most tasks involve command-line programs, so it is unclear whether the findings generalize to larger, more complex software projects.

  2. Some metrics may not reflect real maintainability. For example, the reuse score counts original code executed through the program interface, which does not necessarily mean the code is easy to reuse or extend. Similarly, Testability focuses on reproducing expected behavior rather than detecting actual bugs.

Questions for authors
  1. How much of the reported reuse score comes from simply invoking the original program? Would the conclusions hold if these lines were excluded?

  2. Could you evaluate Testability using bug detection or mutation testing, rather than only checking whether generated tests reproduce program behavior?

2 comments

Dear Reviewer vGFm,

Thank you for your comments and questions.

Weakness 1: Testability covering just seven. Most tasks involve command-line programs, so it is unclear whether the findings generalize to larger, more complex software projects.

Question 2: Could you evaluate Testability using bug detection or mutation testing, rather than only checking whether generated tests reproduce program behavior?

Thank you for your thoughtful questions. CarryBench evaluates agent-generated code maintainability through downstream tasks beyond functional correctness. For Testability, a fixed tester constructs an executable black-box test suite, with construction tokens measuring the effort required to meet prescribed testing requirements. A key challenge is obtaining reliable ground truth across artifacts that differ in implementation language, internal structure, and functional correctness.

We define behavioral scenarios and observation schemas per case, then obtain trusted execution observations through a consistent CLI interface as artifact-specific ground truth. Mutation testing would additionally require valid mutation operators and comparable detection criteria across heterogeneous implementations. Observations produced by the submitted test suite are compared against these trusted observations, without requiring manually specified expected outputs for each implementation. This allows us to assess whether the tester accurately captures each artifact's behavior, regardless of its functional correctness. The same protocol supports new producers without redesigning tasks, while new cases can define their own scenarios and observation schemas to extend coverage.

Tokei (Appendix A7.3) illustrates substantive CLI-based testing: fixture construction, ignore and filtering rules, symbolic-link handling, two output surfaces, and relations between runs. Bug detection likewise requires constructing inputs, executing programs, and reliably observing their behavior before deviations from expected behavior can be identified. Our evaluation measures the cost of these test-construction activities.

Weakness 2: Some metrics may not reflect real maintainability. For example, the reuse score counts original code executed through the program interface, which does not necessarily mean the code is easy to reuse or extend. Similarly, Testability focuses on reproducing expected behavior rather than detecting actual bugs.

Thank you for your comments. In Section 3.4, source reuse share tracks executed task-relevant code provenance, including interface-based invocation, while construction tokens measure the interaction cost through the first accepted submission. Together, these metrics distinguish reuse extent from integration effort, as a high source reuse share does not necessarily imply easier reuse. We will clarify their complementary roles in the revision.

For Testability, constructing inputs, executing programs, and reliably capturing outputs and execution effects are common activities in software testing. These activities also support bug detection, where observed behavior is compared against expected behavior to identify potential faults. Our Testability task evaluates the effort required to construct executable black-box tests that accurately capture an artifact's behavior under prescribed scenarios.

Question 1: How much of the reported reuse score comes from simply invoking the original program? Would the conclusions hold if these lines were excluded?

Thank you for your question. Our analysis shows that artifact invocation accounts for 99.90%, 99.90%, and 99.95% of source reuse share under DeepSeek V4 Flash, HY3, and Kimi K2.7 Code, respectively. When measured against all executed task-relevant code, the corresponding percentages are 79.03%, 78.33%, and 77.74%. These results are based on artifact-level averages for each producer.

As explained under Weakness 1 and Question 2, the CLI provides a consistent interaction contract. Reusing functionality through library functions, APIs, or program interfaces is common software-engineering practice: consumers invoke existing implementations and add integration logic rather than copying or reimplementing functionality. Our CLI-based Reusability task therefore predominantly uses artifact invocation. For example, the Rhit consumer (Appendix A7.2) invokes existing queries while adding normalization, merging, deduplication, sorting, and policy checks to construct a new case-specific workflow.

Excluding invocation lines would leave only source copying or adaptation, which is uncommon in our tasks. Most scores would become zero, leaving insufficient variation for meaningful producer rankings or cross-maintainer agreement analysis. We therefore retain invocation lines because using existing functionality through program interfaces is itself a form of reuse.

AuthorsOpenReview ↗

A Brief Summary and Kind Reminder

Dear Reviewer vGFm,

Thank you again for your thoughtful comments and questions. As the discussion period is approaching its end, we would be grateful if you could take a moment to review our responses.

For your convenience, we briefly summarize our main clarifications:

  • Testability evaluation (Weakness 1 & Q2): We clarified that our evaluation measures the effort required to construct executable black-box tests, using trusted execution observations to enable consistent evaluation across heterogeneous implementations. We also explained the challenges of mutation-based evaluation and illustrated the substantive testing activities involved through the Tokei case in Appendix A7.3.
  • Maintainability metrics (Weakness 2): We clarified the complementary roles of source reuse share and construction tokens in measuring reuse extent and integration effort. For Testability, we explained how constructing inputs, executing programs, and reliably observing behavior are essential activities in software testing and also support bug detection.
  • Interface-based reuse (Q1): We provided additional statistics quantifying the contribution of artifact invocation to source reuse share. We explained why interface-based invocation is a meaningful form of code reuse, illustrated by the RHit case, and why excluding it would leave insufficient variation for meaningful comparative analysis.

We sincerely hope these clarifications have addressed your concerns. If any questions remain, we would be happy to provide further clarification before the discussion closes.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_CedUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CarryBench, where 9 models generate code for 13 cases and 3 fixed maintainer models then perform Analysability, Reusability, and Testability tasks on the frozen output to measure maintainability. Producer rankings shift across tasks and maintainers, and static LCOM cohesion is only weakly related to downstream difficulty.

Critical strengths
  1. The experiments are well organized. For example, the authors evaluate the code-writing and code-maintaining models separately and exclude cases where a model maintains its own output. The anti-cheating checks are also carefully executed.
  2. The raw data are fully provided in Table A2, and the conclusions are restrained, acknowledging correlation rather than causation.
Critical weaknesses
  1. Most of the main findings are about rankings that don't agree, and the mean cross-task τa is close to 0. But the authors never rerun the same maintainer on the same artifact, and Reusability and Testability end up comparable only through token counts, so I can't tell whether this disagreement is a real difference or just noise.
  2. Analysability gives the observer only 5 code inspections for 50 questions. Per Tables A2 and A11, Claude averages 10 source files and scores lowest, while GPT averages only 1.5 and scores highest. I suspect this mostly measures whether the code can be read within 5 calls, not how readable it is.
Questions for authors
  1. Are there repeated runs of the same maintainer on the same artifact? What is the test-retest τ or ICC? If there are none, how do you distinguish the disagreement in Findings 1 and 2 from noise?
  2. In Table A4, blocks with modest partial R² still get very small q values (e.g., the Analysability block has 13.80% with q = 0.0031, despite having 10 components and only 104 samples). Could you describe the permutation test and the effective sample size?
  3. How was the 5-call inspection limit for Analysability chosen? With a larger budget, do the producer rankings and the Lexical organization association still hold?
6 comments
AuthorsOpenReview ↗

Responses to Weaknesses 1–2 and Question 1

Dear Reviewer CedU,

Thank you for your comments and questions.

Weakness 1: Most of the main findings are about rankings that don't agree, and the mean cross-task τ is close to 0. But the authors never rerun the same maintainer on the same artifact, and Reusability and Testability end up comparable only through token counts, so I can't tell whether this disagreement is a real difference or just noise.

Question 1: Are there repeated runs of the same maintainer on the same artifact? What's the test-retest τ or ICC? If there are none, how do you distinguish the disagreement in Findings 1 and 2 from noise?

Thank you for your thoughtful comments. Our original evaluation prioritized coverage across producers, maintainers, and maintenance demands. To assess stability, we added two independent Stage II runs using DeepSeek V4 Flash per evaluation unit, retaining all cases and demands and fixing artifacts, tasks, prompts, tools, and budgets. Including the original, each unit has three measurements.

Three-run producer means ± sample SD are shown below (scores/shares in %, SD in percentage points; tokens in thousands).

Producer modelAnalysability score (%)Source reuse share (%)Reusability construction tokens (k)Testability construction tokens (k)
DeepSeek V4 Pro86.56 ± 0.7976.56 ± 0.90795.53 ± 12.89268.32 ± 3.46
GLM 5.186.92 ± 0.4678.88 ± 0.74753.13 ± 5.91314.13 ± 2.15
GLM 5.289.49 ± 0.4478.57 ± 0.941279.26 ± 12.29347.34 ± 5.75
HY391.74 ± 1.0878.28 ± 1.04774.76 ± 8.20326.35 ± 3.21
Kimi K2.7 Code89.23 ± 0.5580.10 ± 1.43926.90 ± 9.02358.10 ± 3.47
MiniMax M389.08 ± 0.8681.91 ± 1.011036.15 ± 8.56266.32 ± 2.81
GPT-5.6 Sol96.10 ± 0.4978.06 ± 0.61584.91 ± 3.98265.41 ± 1.70
Claude Opus 586.92 ± 1.4180.27 ± 1.191323.04 ± 3.07224.79 ± 2.28

Three-run means remain close to the original results, and the maximum relative SD across metrics is 1.78%. Repeatability is assessed using absolute-agreement ICC(A,1) and Kendall’s τ for producer rankings.

OutcomeICC(A,1)Original vs. repeat 1 τOriginal vs. repeat 2 τRepeat 1 vs. repeat 2 τ
Analysability score0.98820.8210.7500.786
Source reuse share0.97990.8570.7860.929
Reusability construction tokens0.99771.0001.0001.000
Testability construction tokens0.99680.9290.9290.857

ICC is computed at the artifact or task-unit level (tokens transformed by log(1 + tokens)); τ compares rankings based on the eight producers’ mean outcomes. ICC = 0.9799–0.9977 and τ = 0.750–1.000 indicate high repeatability.

Weakness 2: Analysability gives the observer only 5 code inspections for 50 questions. Per Tables A2 and A11, Claude averages 10 source files and scores lowest, while GPT averages only 1.5 and scores highest. I suspect this mostly measures whether the code can be read within 5 calls, not how readable it is.

Thank you for your sincere comments. The five-call budget provides a common inspection limit, and the observer can search multiple files by one call and reuse findings across all 50 questions. Table A11 shows GLM 5.2 achieving higher Analysability than GLM 5.1 despite having more source files. More importantly, across 104 artifacts per maintainer, controlling for case, producer, and language as in Section 4.4, adding log(1 + source file count) yields partial R² = 2.56%, 1.41%, and 3.48% for DeepSeek V4 Flash, HY3, and Kimi K2.7 Code. Source file count is one factor associated with Analysability, but its additional explanatory contribution is relatively small. Our lexical organization analysis further examines the associations of identifier sharing, cross-file spread, import density, duplication, and local control structure with Analysability.

AuthorsOpenReview ↗

Responses to Question 2 and Question 3

Question 3: How was the 5-call inspection limit for Analysability chosen? With a larger budget, do the producer rankings and the Lexical organization association still hold?

Thank you for your question. An early DeepSeek V4 Flash comparison found producer ordering and lexical organization associations largely stable as the budget increased. We chose five calls for efficiency and applied that setting to the other maintainers.

DeepSeek V4 Flash scores at 5/10/20 calls (parentheses: change from five calls, percentage points):

Producer model5 calls10 calls (change from 5)20 calls (change from 5)
DeepSeek V4 Pro86.8%86.6% (−0.2)87.5% (+0.7)
GLM 5.186.9%88.6% (+1.7)88.3% (+1.4)
GLM 5.289.2%90.9% (+1.7)92.2% (+3.0)
HY391.9%93.5% (+1.6)94.3% (+2.4)
Kimi K2.7 Code89.7%91.7% (+2.0)92.8% (+3.1)
MiniMax M389.2%90.8% (+1.6)92.3% (+3.1)
GPT-5.6 Sol96.3%97.1% (+0.8)96.9% (+0.6)
Claude Opus 587.2%88.3% (+1.1)88.0% (+0.8)

Scores generally rise with budget; Kendall’s τa is 0.893, 0.893, and 0.929 for 5–10, 5–20, and 10–20 calls.

The lexical organization association under each budget is:

Inspection budgetPartial R²Change from 5 calls (percentage points)q
513.80%—0.0031
1012.40%−1.400.0075
2010.30%−3.500.0180

Question 2: In Table A4, blocks with modest partial R² still get very small q values (e.g., the Analysability block has 13.80% with q = 0.0031, despite having 10 components and only 104 samples). Could you describe the permutation test and the effective sample size?

Thank you for pointing out this issue. Upon rechecking our analysis, we found an error in the description in Appendix A4.2. The 10 components of the lexical organization block were transformed, standardized, and aggregated into a single block score, which was then added to the baseline model as one predictor, consistent with Testability. We apologize for the confusion and will correct the description in the revised manuscript.

Each artifact produces one Analysability score from 50 questions. After excluding self-maintenance, each maintainer panel contains 13 cases × 8 producers = 104 observations. Producers build independently from empty workspaces, and we use case controls and within-case permutations to account for shared task structure.

Specifically, we first fit the baseline model, then shuffle its residuals among the eight artifacts within each case and add them back to the fitted values. We refit both the baseline and full models (including the block score) and recompute partial R², while keeping predictors and case, producer, and language labels fixed. Across 13 cases, the joint permutation space is (8!)13(8!)^{13}.

We repeat this procedure for 20,000 permutations and compute p=(X+1)/(20,001)p=(X+1)/(20{,}001), where X counts permutation statistics at least as large as observed. Finally, Benjamini–Hochberg correction across candidate blocks within each outcome–maintainer panel yields q.

AuthorsOpenReview ↗

A Brief Summary and Kind Reminder

Dear Reviewer CedU,

Thank you again for your thoughtful comments and questions. As the discussion period is approaching its end, we would be grateful if you could take a moment to review our responses.

For your convenience, we briefly summarize our main clarifications and additional analyses:

  • Evaluation stability (Weakness 1 & Q1): We conducted two additional independent Stage II runs, providing three measurements per evaluation unit. The results show high repeatability, with ICC values of 0.9799–0.9977 and Kendall's τ values of 0.750–1.000 across producer rankings.
  • File count and inspection budget (Weakness 2 & Q3): We provided controlled analyses showing that file count contributes relatively little additional explanatory power to Analysability (partial R² of 1.41%–3.48%). We also evaluated larger inspection budgets of 10 and 20 calls, finding that producer rankings and lexical organization associations remain largely stable.
  • Statistical methodology (Q2): We clarified the construction of the lexical organization block, the effective sample size, and the case-restricted permutation procedure used to obtain the reported q-values.

We sincerely hope these additional experiments and clarifications have addressed your concerns. If any questions remain, we would be happy to provide further clarification before the discussion closes.

Thank you for your time and consideration.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Summary

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for the time and effort devoted to evaluating our work. Your constructive feedback has helped us further strengthen the paper. We are pleased that our responses have helped address several of the reviewers' concerns, and we especially appreciate Reviewer ATVR's careful consideration of our clarifications, continued engagement throughout the discussion, and reconsideration of their assessment.

We are also grateful for the reviewers' recognition of the strengths of our work, including the meaningful evaluation perspective beyond functional correctness and thoughtful separation of producers and maintainers (vGFm), the multifaceted evaluation of maintainability through executable downstream tasks and the reusability of the framework (ATVR), and the well-organized experiments, carefully designed anti-cheating checks, restrained interpretation of findings, and transparent data reporting (CedU).

Throughout the discussion, we provided additional experiments and clarifications addressing the reviewers' concerns:

  • Ranking reliability and statistical uncertainty (CedU, ATVR): We conducted independent repeated evaluations, paired bootstrap analyses, and leave-one-case-out (LOCO) analysis. The repeated evaluations showed high repeatability (ICC: 0.9799–0.9977), while LOCO supported ranking robustness to individual case deletions. Together, these analyses provide evidence for the stability and reliability of the results at the current evaluation scale. Furthermore, 20,000 paired bootstrap resamples yielded one-sided 95% upper bounds of at most 0.200 for mean cross-demand agreement, providing direct statistical evidence for Finding 1.
  • Analysability and sensitivity analyses (CedU): We controlled for file-count confounding (partial R²: 1.41%–3.48%) and examined larger inspection budgets, finding that producer rankings and lexical-organization associations remained largely stable. We also clarified the effective sample size, block construction, and case-restricted permutation procedure.
  • Evaluation protocols and metric interpretation (vGFm, ATVR): We clarified the complementary roles of source reuse share and construction tokens, explained interface-based code reuse, and detailed the Testability protocol using artifact-specific ground truth. We also justified the exclusion of self-maintenance, provided the corresponding comparison results, and clarified downstream task preparation and validation.
  • Benchmark contribution and practical applications (CedU, ATVR): We clarified that CarryBench's contribution extends beyond individual findings by providing a reusable protocol for evaluating agent-generated code through downstream maintenance work. Findings 1 and 3 provide empirical support for demand-specific evaluation and the importance of direct maintenance-task evaluation alongside static measurements. We further illustrated how the framework can inform producer selection across maintenance demands and maintainer profiles.

We appreciate the reviewers' remaining suggestions and recognize the value of extending the evaluation to more independent repository cases and complex software projects. In the latest revised manuscript, we have corrected the description of the Analysability block construction in Appendix A4.2 (CedU), clarified the distinction between reuse extent and integration effort (vGFm), and refined the presentation of Finding 2 to make the experimental results and their interpretation clearer (ATVR).

Thank you again for your careful consideration. We hope these additional experiments, clarifications, and discussion outcomes will be taken into account in the final assessment.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.