W1. There is a discrepancy between the reported success rates and the stated statistical protocol; furthermore, the actual evaluation configuration is insufficient to reconstruct or verify the results.
Section 4.1 and Table 1 of the paper state that each model was run three times across 120 Operation tasks, with binary success rates reported. Following this protocol, the success rate should be the number of successful attempts (an integer) divided by 360. However, the 40.4% success rate reported for Opus 4.8 cannot be derived from this denominator, even when rounded to one decimal place. Specifically, 145 successes would yield approximately 40.3%, while 146 successes would yield approximately 40.6%. While this might simply be a typographical error in the table, it could also indicate undisclosed exclusions, re-runs, or weighting schemes; therefore, the authors should provide the actual numerators and denominators used.
This issue is directly related to how failure states are handled. Appendix A.3 categorizes missing answers, unexecuted answers, timeouts, environment errors, and functional failures, but it does not specify how these states factor into the final denominator used in Table 1, nor does it report the frequency of these states for each model. Calculating a functional success rate based only on successfully executed samples carries a different meaning than calculating a completion rate based on all task attempts.
Furthermore, although the paper repeatedly mentions that the budget and execution framework remained consistent, it fails to specify the actual limits, such as time, token counts, or tool-call budgets, nor does it provide the specific version of the framework or the complete prompts used during execution. While Appendix A.2 lists the fields that should be recorded in the manifest, these definitions are not a substitute for the full experimental configuration. For a benchmark whose primary contribution is the measurement of reliability, these omissions significantly hinder the interpretation and reproducibility of the results.
W2. While the stability of the scoring in the Exploration phase has been partially verified, there is insufficient evidence to confirm that these scores accurately measure the system's underlying knowledge.
In Section 5.4, the authors assess stability through repeated scoring and by substituting the scoring model which comparing results from a GPT-5.5 scorer versus a GPT-5.4 scorer. I believe this validation provides reasonable support for the experimental results. However, the consistency between two scoring models does not, in itself, prove that they are correctly evaluating the accuracy of source-based facts, the validity of paraphrased representations, or the severity of omitted information. The human review described in Appendix C focuses on the recoverability of ground-truth facts; it does not constitute an independent human evaluation of the agents' actual responses. The paper does not report human calibration results at the level of the responses themselves.
Table 4 shows that replacing the scoring model leads to variations in both the direction and magnitude of the scores. The average scores for GPT-5.5 and Doubao both increased by 2.2 points, whereas the score for DeepSeek dropped from 58.0 to 51.9, and those for MiniMax and Qwen decreased by 3.5 and 3.7 points, respectively. While this does not in itself prove the existence of bias, it suggests that a high overall correlation can still mask scoring discrepancies linked to response style or the specific model being evaluated.
Furthermore, the paper treats Exploration as an observable proxy for system knowledge, using it to discuss differences in model capabilities and fact types. The current evidence primarily indicates that models achieve different weighted rubric scores; it is not yet sufficient to confirm that these differences consistently reflect the completeness of the models' knowledge. Independent calibration, covering the factual accuracy of the agents' responses, the acceptability of alternative phrasings, and the penalties applied for hallucinations, is required. Additionally, it is necessary to verify whether the paper’s primary conclusions are sensitive to the specific weights and scoring models employed.
W3. The analysis of the three task categories and execution trajectories supports the existence of performance variations across different dimensions; however, it fails to clearly identify the knowledge bottlenecks or the specific stages of failure emphasized by the paper.
Section 4.1 explicitly acknowledges that Exploration and Operation utilize different sets of tasks. Consequently, while shifts in model rankings indicate that these two sets of tasks measure different performance aspects, they do not directly demonstrate a capability gap, a transition from knowing to operating, within a single maintenance task. The claim on page 2, that a unified hierarchical evaluation would misrank maintenance capabilities, lacks a reference to an independent, overarching standard for maintenance proficiency. Similarly, the lower success rate in the Evolution category cannot be attributed solely to the difficulty of knowledge acquisition, as the task content, output requirements, and the criteria for achieving a pass across all tests also differ.
The statistics presented in Section 5.3, concerning failures that occur even after the correct location has been identified, also require more precise measurement. In Appendices I.1 and I.2, file access is tracked exclusively through successful, dedicated Read calls. A hit on a file does not guarantee that the model has retrieved the specific code snippet corresponding to a given fact; conversely, source code information obtained via other tools is not fully captured by this metric. Consequently, the fact that 41.93% of the relevant items in Table 9 did not receive a perfect score indicates that file access is not equivalent to providing a complete answer. However, one cannot simply attribute all these instances to a failure in understanding following a successful localization. While the specific traces provided in the appendix offer insight into individual cases, they cannot substitute for an aggregate statistical breakdown of the stages at which failures occur.
Similarly, in Figure 4, the gap between legacy behavior invariants and modification paths ranges from 2.5 to 6.1 percentage points; however, the study does not report the uncertainty of these differences, nor does it account for the varying difficulty levels associated with different types of facts. While the examples from NetBeans and Flowable demonstrate that regressions in legacy behavior can lead to the rejection of a patch, the study does not quantify how frequently this mechanism contributes to the overall failure rate in the Evolution category. The absence of a comprehensive breakdown, distinguishing between failures in Fix-to-Pass tasks, Pass-to-Pass tasks, and both, makes it difficult to quantify the prevalence and actual impact of blind spots regarding legacy behavior.
W4. Task selection and prompt pruning based on model performance limit the scope of conclusions regarding repository dependency.
Appendix E notes that if at least two models successfully complete an Operation task without access to the repository, the task is modified to require more repository-specific information. Appendix D further states that the selection criteria for the Evolution category aim to ensure that the performance of the strongest model falls somewhere between the minimum and maximum observed values. While these choices help construct a benchmark with high discriminative power, they also mean that the final distribution of tasks is, a function of model performance. The paper does not explicitly clarify whether the models, computational budgets, or experimental runs used for this filtering process were independent of the final evaluation, nor does it report how the task distribution shifted as a result of this filtering.
Therefore, while the performance decline observed in the no-repository condition confirms that the filtered task set is indeed dependent on repository information, the magnitude of this decline cannot be directly extrapolated to represent the general degree to which real-world maintenance tasks rely on source code access. Furthermore, a drop in performance upon the removal of source code does not rule out the possibility that models are relying on memorized knowledge of public projects; source code access simultaneously alters both the cues available for retrieval and the conditions under which an answer can be verified. In fact, Appendix E acknowledges that eight tasks remain solvable by at least two models even in the absence of repository access. Furthermore, the MongoDB examples remove terminology related to mechanisms such as rollback and oplog, rather than merely stripping away final patch identifiers. The paper needs to explain why such information constitutes a leaked clue that ought to be removed, and whether the remaining context is still sufficient to address the actual requirements of the task. Appendix D mentions stratifying data by training cutoff dates and implementing controls, specifically restricted context and greenfield-hollow settings, yet it fails to provide the corresponding results. Consequently, it is difficult to disentangle the respective impacts of inherent maintenance difficulty, the reduction of prompt information, and the selection of tasks in the current experiments.