- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates code localization from an interesting perspective of repair-context completeness. It proposes CFRR to analyze why contexts are incomplete and proposes TraceFill to supplement existing localization context with code regions derived during execution. Experiments show that using locations identified by TraceFill can help to increase the effectiveness of repair approaches.
Critical strengths
The proposed research problem is interesting and worth studying. Identifying a relevant file or suspicious line does not necessarily give a repair model enough context to fix a bug. Thus, looking at localization through the lens of repair completeness is useful and novel to the best of my knowledge
Critical weaknesses
First, the presentation is quite difficult for readers to follow. Specifically, I found it difficult to follow how the six exploratory studies lead to CFRR and then to TraceFill. Many details and justifications are missing, so I cannot properly evaluate the soundness of these experiments. For example, what are the overarching objectives of the six exploratory studies? What are the implications of these exploratory results and how do they motivate CFRR and TraceFill? For CFRR, why are these four criteria chosen, and what distinct insight does each provide? For TraceFill, why should exception sites receive higher priority than call/return endpoints and other executed lines? These are just a few questions that I repeatedly ask myself as a reader and cannot find sufficient answers in the current submission.
Second, the contribution beyond existing work is insufficiently explained. CFRR separates failures in 4 sub-criteria: exposure, reachability, capacity and realization. Among these criteria: exposure and reachability are not new and widely-used by prior works despite the new name provided by the papers. Capacity and realization are somewhat new. However, Table 2 reports no capacity failures, so I am not convinced that these criteria are really important, especially given the long-context capabilities of frontier models. Meanwhile, the need for realization is also insufficiently motivated. My understanding is that it measures whether repair tools really select correct context from candidates provided by code localization approaches. Thus, its outcome depends on the repair tool effectiveness, so using it to evaluate code localization will introduce additional confounding factors. Besides, the novelty of TraceFill also remains unclear to me. Many prior works such as AutoCodeRover[1] have used execution information to improve code localization and demonstrate its effectiveness. TraceFill differs in preserving the initial context and greedily adding source windows under a residual budget. However, the paper does not sufficiently explain why these particular choices constitute a substantive advance or compare them with closely related alternatives under the same budget.
Finally, many experimental designs are questionable and yield threats to the validity of the findings. Specifically, CFRR is evaluated on a set of 117 single-file bugs whose reference patches lie on the crash path. Unfortunately, for single-file bugs, some criteria of CFRR is not fully evaluated, e.g., exposure reduces to checking whether one target file is retrieved, rather than whether all required files are available. Besides, the repair improvements may largely come from problems in the Native baseline. Native provides no source code for 798 of 1,004 bugs, often due to parsing failures, and all net repair gains for DeepSeek and GPT occur in these empty-context cases (Section 5.3 and Table 5b). The authors should fix the parsing and configuration problems and evaluate whether the improvements remain. The comparisons also do not establish the effectiveness of TraceFill’s selection strategy. TraceFill adds up to 414 source lines, but the controls restrict candidate selection, while the ablations provide limited support for its design choices. Comparisons with expanded Native windows, enclosing functions, and standard fault-localization rankings under the same budget would better establish its effectiveness.
[1] Zhang, Yuntong, et al. "Autocoderover: Autonomous program improvement." Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2024.
Questions for authors
Q1. What is the overarching objective of the six exploratory studies, and how do their findings motivate CFRR’s four criteria and TraceFill’s design?
Q2. What is the novelty of CFRR criteria provided beyond existing localization metrics?
Q3. What is TraceFill’s main methodological contribution beyond existing execution-based approaches such as AutoCodeRover? Why should preserving the initial context and greedily adding execution-derived windows provide a substantive advantage?
Q4. Why is CFRR evaluated only on 117 single-file bugs whose reference patches lie on the crash path? Would including multi-file and off-path bugs change the findings about the four criteria?
Q5. Do TraceFill’s improvements remain after correcting Native’s parsing and configuration problems, and how does it compare with expanded Native windows, enclosing functions, and standard fault-localization rankings under the same context budget?