- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates what helps a language model after a failed program repair. This paper proposes a controlled framework for post-failure measurement that. In two prospective DeepSeek populations, BLIND best-of-3 recovers 52.4% and 49.0% of failures, STRUCTURE does not improve on it, and STRONG reaches 83.5% and 82.8%.
Critical strengths
-
This paper shows clear definition of what post-failure recovery measures. For example, Eq. 1 conditions on the screening failure. Excluding the screening candidate from the BLIND samples prevents BLIND from reusing the event that selected the task, and resetting the buggy state keeps the conditions independent.
-
Two independently screened prospective populations agree on the qualitative pattern. In both P1 and P2supp, BLIND recovers about half of the failures, and TRUCTURE does not exceed BLIND.
Critical weaknesses
- The historical and prospective runs differ in decoding temperature and screening configuration as well as in the failure population, so the paper's selection-effect claim is not isolated.
The abstract states that the historical experiments, with "a much lower blind baseline and an apparent structure effect", demonstrate "that failure-selected populations are themselves part of the measurement". Section 7 repeats this as "Failure populations are experimental objects". However, the historical DeepSeek and Xing interventions use temperature 0.0, and the prospective runs use 0.3 (Table 8, B.3). Also, the model configuration of the 919-cell historical screen "is not established by the supplied record" (section 3.2). Therefore, the two populations can also come from different screening models.
The historical GLM-5.3 Flash runs use temperature 0.3 and also show BLIND at 19.0%, which suggests that selection matters as well. However, DeepSeek, Xing and GLM-5.3 Flash all recover exactly 30/158 under BLIND (Table 14). The paper should explain this before the result can be read as evidence.
The selection effect is the paper's main methodological message. However, its main evidence, DeepSeek historical vs. prospective, changes temperature, screening configuration, and population at once.
- The evidence cannot be checked from the paper, and one reported test result cannot be reproduced from the reported counts.
By the paper's own account, the prompts for the five conditions, the task manifests, screening outputs, generated candidates, evaluator traces, and statistical scripts are "not available during editing" (Appendix E). Section 8 also states that "Exact prospective per-condition numerators and confidence intervals were not supplied". Without the STRUCTURE prompt, a reader cannot tell whether it is a non-diagnostic control or contains misleading content. This matters because STRUCTURE falls below BLIND in both prospective populations (−7.8 and −8.5).
Then, for P2supp, BLIND recovers 145/296 and STRUCTURE 120/296 (A.2). However, across every possible pairing of tasks, an exact McNemar test on these best-of-3 outcomes gives at most p≈0.14. An unpaired Fisher test gives p≈0.047. So the reported p=0.302 (Tables 4 and 11) matches neither, and the paper does not name the test.
Also, the results are inconsistent. The P2supp rates are "recomputed with the stated cohort denominator of 296 rather than the inconsistent denominator of 300 implied by the originally supplied percentages" (Table 10). But the "about 81-82%" in section 1 and Figure 2 matches the denominator of 300.
With the prompts for the five conditions missing, a key test result not reproducible, and the originally supplied percentages known to be inconsistent, a reader cannot verify the central "STRUCTURE does not help" result or the size of any effect.
Given that this paper's main contributions are empirical, current results are unable to support the paper's main claims and contributions.
Questions for authors
-
Could you show the results at the matched configuration, so the comparison of historical and prospective populations changes only the failure population?
-
Is BLIND run separately for each model in the historical five-condition study? How do DeepSeek, Xing and GLM-5.3 Flash, at two different temperatures, each recover exactly 30/158?
-
Which test produces p=0.344 and p=0.302 for STRUCTURE vs. BLIND? Can you report per-condition counts with confidence intervals for both prospective populations to validate your reported scores, which are currently unreproducible from results in the paper?
-
Which model and decoding configuration produce the 919-cell historical screen? How do the P2supp counts relate to the denominator of 300?
-
How do the results relate to Olausson et al. [1], who compare self-repair with resampling under matched budgets, and to EvalPlus [2], which shows that HumanEval's tests miss many incorrect solutions?
[1] Is Self-Repair a Silver Bullet for Code Generation? ICLR 2024.
[2] Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. NeurIPS 2023.
Dear Reviewers,
Please review the authors' rebuttal and participate in the discussion to address any remaining questions before the period closes on Nov 18 AoE.
Thank you!
AC